A multi-modal time sequence quality real-time identification system and method for resistance spot welding

Abstract

The present application relates to the field of resistance spot welding quality monitoring, in particular to a multi-modal time sequence quality identification system and method for electronic spot welding, synchronously collecting three types of multi-modal signals of electric-thermal, acoustics and vision, realizing hardware level time alignment by FPGA, extracting local mutation features by one-dimensional convolutional neural network 1D-CNN, capturing long time sequence dependence by time convolution network TCN, introducing cross-modal attention mechanism, taking electric-thermal features as the leading dynamic fusion auxiliary modal information, constructing electric-thermal-acoustic-vision four-channel parallel collection architecture through edge deployment optimization, solving the problem that the current modal signal information is one-sided and difficult to comprehensively represent the welding physical process; the deep learning model inference delay is high, which cannot meet the real-time requirement of <20ms of the production line, the model decision is "black box", and the defect root cause is not explainable.

Description

Technical Field

[0001] This invention relates to the field of resistance spot welding quality monitoring, and specifically to a multimodal time-series quality real-time identification system and method for resistance spot welding. Background Technology

[0002] Resistance spot welding is a core joining process in the manufacturing of automotive body-in-white, rail transit vehicles, and aerospace structural components. Its quality directly determines the structural strength, fatigue life, and safety performance of the product. In industrial production, if welding defects (such as incomplete welds, over-welding, and spatter) are not identified in a timely manner, they will lead to batch rework, safety hazards, and even major quality accidents. Therefore, achieving 100% online, real-time, and accurate identification of welding quality is an urgent engineering requirement.

[0003] Currently, the mainstream quality monitoring methods in industrial sites still mainly rely on traditional single-mode threshold discrimination: judging poor solder joints by monitoring the rate of decrease of the dynamic resistance curve, or based on the current integral value ( Assessing melt nugget size using methods heavily relies on manual experience to set thresholds, making it susceptible to interference from material surface oxidation, electrode wear, and power grid fluctuations, resulting in poor robustness. Furthermore, a single electrical signal cannot fully characterize complex physical processes (such as transient acoustic emissions from spatter and visual characteristics during melt nugget formation), leading to a missed detection rate of over 30% for hidden defects such as microcracks and internal porosity. While some companies have introduced destructive sampling inspections (metallographic testing, tensile shear testing) as a supplement, these methods are time-consuming and destructive, failing to achieve closed-loop process control, and their sampling ratio is limited, making it difficult to meet the requirements of high-end manufacturing for end-to-end quality traceability.

[0004] In recent years, artificial intelligence technology has provided new insights into welding quality identification. Some studies have attempted to use convolutional neural networks (CNNs) or long short-term memory networks (LSTMs) to process single-mode time-series signals such as current and voltage, achieving some success in laboratory settings. However, existing solutions still have the following limitations:

[0005] 1. The information dimension is singular, and it does not systematically integrate multi-source heterogeneous signals such as acoustic (acoustic emission), visual (indentation morphology, spark characteristics), and thermal (melting temperature), making it difficult to capture the multi-physics field coupling characteristics of defects;

[0006] 2. Insufficient real-time performance: The BiLSTM and other recurrent neural network structures used have serial computation and high inference latency (usually >50 milliseconds), which cannot meet the production line cycle time requirements (industrial sites generally require single weld point analysis latency to be less than 20 milliseconds). In addition, the model has a large number of parameters, making it difficult to deploy on edge computing devices.

[0007] 3. Lack of interpretability: The model decision-making process is a "black box," making it impossible to pinpoint the time of defect occurrence or correlate multimodal signal characteristics. This makes it difficult for process engineers to understand the judgment criteria, hindering technology implementation and process optimization.

[0008] 4. The deployment lacks theoretical support. Existing edge optimization relies heavily on empirical parameter tuning and has not established a quantitative relationship between inference latency and model structure and hardware parameters, resulting in unstable deployment effects and difficulty in scaling up.

[0009] Existing solutions mostly employ simple feature splicing or fixed weighting, failing to establish dynamic semantic relationships between modes (such as the causal correspondence between "current abnormality moment" and "acoustic emission pulse"), and do not design a dedicated architecture for the temporal causality and high noise characteristics of welding signals. In addition, the scarcity of labeled samples and the high noise of signals in industrial fields further restrict the practicality of complex models.

[0010] In summary, existing technologies have significant shortcomings in three aspects: deep collaboration of multimodal information, real-time reasoning assurance at the edge, and interpretable analysis of defect root causes. There is an urgent need for an intelligent identification technology for resistance spot welding quality that combines high precision, strong real-time performance, high reliability, and adaptability to industrial edge environments to support the core requirement of intelligent manufacturing for the "perception-diagnosis-optimization" closed loop of welding processes. Summary of the Invention

[0011] The purpose of this invention is to provide a multimodal time-series quality real-time identification system and method for resistance spot welding. It aims to construct a four-in-one technical system integrating "signal acquisition → feature extraction → dynamic fusion → edge reasoning," overcoming three core defects in existing resistance spot welding quality monitoring technologies: 1. Single-modal signal information is one-sided and cannot comprehensively characterize the welding physical process; 2. Deep learning model inference latency is high and cannot meet the production line's <20ms real-time requirement; 3. Model decision-making is "black box," lacking interpretability of defect root causes.

[0012] The resistance spot welding identification method based on multimodal temporal fusion and temporal convolutional network aims to construct a four-in-one technical system of "signal acquisition-feature extraction-dynamic fusion-edge reasoning", including a multimodal collaborative perception module, a 1D-CNN and TCN cascaded encoding module, a cross-modal attention dynamic fusion module, a theory-driven edge optimization module, and an interpretable closed-loop output module.

[0013] Multimodal collaborative sensing module: Simultaneously acquires electro-thermal main modes (current, voltage, pressure, weld nugget temperature), acoustic auxiliary modes (acoustic emission signal), and visual auxiliary modes (indentation area, spark brightness, electrode displacement). It achieves sub-millisecond time alignment using the current rising edge as the hardware trigger reference. For the electro-thermal main modes, a Hall current sensor and a voltage divider resistor network are used to acquire the welding current and electrode voltage respectively. A piezoelectric pressure sensor is used to monitor dynamic changes in electrode pressure. Simultaneously, an infrared thermal imager and a thermocouple array fusion temperature measurement strategy are employed to capture the weld nugget temperature field distribution. The acoustic-assisted modality uses a broadband acoustic emission sensor attached to the lower electrode arm to capture the characteristic signals of metal plastic deformation and microcrack propagation during the weld nugget formation stage. The visual-assisted modality consists of an industrial high-speed camera and a coaxial light source. Image preprocessing extracts the geometric contour of the indentation, the integral value of the spatter brightness, and the electrode displacement curve. To cope with the strong electromagnetic interference environment in the industrial field, the electro-thermal signal channel adopts differential transmission and a π-type filter network, the acoustic signal channel is configured with bandpass filtering and an adaptive noise cancellation algorithm, and the visual signal channel implements light intensity normalization and motion blur compensation. All modal data are marked with a unified timestamp and divided into fixed-length time segments according to the welding cycle to form a structured multimodal tensor input, laying the data foundation for subsequent deep feature extraction.

[0014] 1D-CNN and TCN cascaded encoding module: Hardware-level time alignment is achieved using FPGA. A three-level one-dimensional convolutional neural network 1D-CNN is used to extract local mutation features (such as splash pulses and current drops). Then, a TCN module composed of six layers of dilated causal convolutional residual blocks is used to capture long-term temporal dependencies, taking into account both local sensitivity and global context awareness.

[0015] Cross-modal attention dynamic fusion module: Introduces a cross-modal attention mechanism, dynamically fuses auxiliary modal information with electro-thermal features as the main driver, calculates its attention weights with acoustic and visual modalities, realizes the causal correlation enhancement of "electrical anomaly moment - acoustic emission response", and generates highly discriminative fusion features.

[0016] Specifically, the module first uses the electro-thermal feature vector encoded by 1D-CNN and TCN as the query sequence, and the acoustic features and visual features as key-value pairs inputs to the dual-branch attention structure. For the interaction between the electro-thermal query vector and the acoustic key vector, a scaling dot product attention method is used to calculate the similarity matrix. The dynamic weight distribution of the acoustic mode is obtained through normalization. This weight is significantly enhanced at abnormal moments such as sudden drops in current or voltage spikes in the electrical signal, accurately capturing the transient energy release of metal plastic deformation accompanying the acoustic emission signal. For the visual mode, a time-aware position encoding is introduced, enabling the attention mechanism to associate the temporal correspondence between changes in electrode indentation morphology and electro-thermal features. Especially in the frame interval where the brightness integral value of the spatter spark suddenly increases, the visual weight is adaptively increased to strengthen the representation of geometric defects. To avoid feature overwhelming caused by the dominance of a single mode, the module sets a modal confidence gating mechanism, which dynamically adjusts the fusion ratio based on the real-time estimated value of the signal-to-noise ratio of each mode. When electromagnetic interference in the industrial field causes a decrease in the quality of the electro-thermal signal, the fusion weight of the acoustic-visual mode is automatically increased to ensure the robustness of feature representation.

[0017] Theory-driven edge optimization module: This module optimizes edge deployment by constructing a four-channel parallel acquisition architecture (electric, thermal, acoustic, and visual), establishing a quantitative relationship model between inference latency and input length, model structure, and hardware resources, and implementing input pruning within the system. Five optimizations—INT8 quantization, operator fusion, memory pool reuse, and knowledge distillation—ensure a stable inference latency of less than 20 milliseconds for a single solder joint. Input pruning is based on the effective duration analysis of the electro-thermal signal, dynamically removing redundant sampling points in the pre-stressing and maintenance phases while preserving the complete features of the current rising edge, peak plateau, and decay tail. INT8 quantization employs a layer-by-layer sensitivity calibration scheme, implementing asymmetric quantization on the attention weight matrix and convolution kernel parameters. The optimal scaling factor is determined by minimizing KL divergence, resulting in a 75% reduction in model size while keeping Top-1 accuracy loss within 0.8%. Operator fusion targets… The Conv1D-BN-ReLU combination in the temporal convolutional network and the QKV linear transformation in multi-head attention are graph optimized to eliminate the memory read / write overhead of intermediate tensors, improving the execution efficiency of a single operator by 3.2 times. The memory pool reuse mechanism is designed with a circular buffer to manage cross-frame feature caching and avoid fragmentation latency caused by dynamic memory allocation. Knowledge distillation uses a full-precision model deployed in the cloud as the teacher network and a lightweight model at the edge as the student network, constructing a multi-task training framework that includes feature alignment loss, response distillation loss and relation distillation loss, especially strengthening the student's ability to learn the transient patterns of splash sparks and the acoustic emission features of microcracks.

[0018] Interpretable closed-loop output module: generates cross-modal attention heatmaps, accurately locates the time of defect occurrence and dominant mode, and provides physical root cause analysis while outputting process status judgment results.

[0019] Furthermore, the theory-driven edge optimization module also includes: a model constraint parsing module: used to obtain an initial welding quality model and parse the hierarchical structure parameters and inference constraint parameters of the model, wherein the inference constraint parameters include: the number of network layers L, the computational cost of each layer, etc. Number of feature channels Input timing length Input timing trimming module: Used to trim the multi-source timing sequence of the welding process without changing the physical time sequence of the welding process.

[0020] Joint optimization module for structure pruning and quantization: Used to perform joint optimization of structure pruning and parameter quantization on the initial model while maintaining the consistency of the model topology;

[0021] Operator fusion and memory reuse module: used to fuse continuous computation operators in the model inference process and perform static memory pool reuse on intermediate feature tensors;

[0022] Knowledge distillation constraint module: used to introduce teacher models to constrain student models during model lightweighting.

[0023] Edge Deployment and Implementation Inference Module: Used to deploy the optimized welding quality recognition model to edge computing devices for real-time inference.

[0024] Further specifying the steps, the details are as follows:

[0025] Step S1: Synchronously acquire multi-mode signals during the resistance spot welding process;

[0026] Step S2: Preprocess the multimodal signal;

[0027] Step S3: Based on the local feature extraction of a one-dimensional convolutional neural network, the preprocessed data is input into a three-level convolutional structure to extract local mutation features;

[0028] Step S4: Design a global temporal model based on a temporal convolutional network (TCN);

[0029] Step S5: Fusion and collaborative diagnosis based on cross-modal attention;

[0030] Step S6, Classification Decision and Interpretability Analysis;

[0031] Step S7: Perform edge inference optimization and deployment.

[0032] Further specifying, the specific steps in step S1 are as follows:

[0033] Step S1.1: Configure the sensor array to construct a monitoring network composed of multiple physical quantity sensors, including the electro-thermal mode: using a Hall current sensor and a voltage divider circuit to obtain the welding current. With voltage The pressure sensor obtains the electrode pressure. Infrared thermopile sensors monitor the temperature of the molten core region. ;

[0034] Acoustic modes: Piezoelectric acoustic emission sensors are used to capture elastic waves generated by welding spatter and lattice fracture;

[0035] Visual modality: Capturing electrode displacement using a high-speed industrial camera Spark brightness and indentation morphology characteristics;

[0036] The welding current / voltage sampling rate is 10kHz with an accuracy of 0.5%FS; the infrared thermopile response time is <0.1ms; the acoustic emission sensor has a frequency band of 50-400kHz; and the high-speed industrial camera operates at 1000fps.

[0037] Step S1.2: Design a sub-millisecond synchronization mechanism based on FPGA. Use an FPGA module to monitor the rising edge of the welding circuit current as a hardware trigger signal to synchronously activate all sensor channels, ensuring the time alignment error of the electrical, acoustic, and optical multimodal signals. .

[0038] Further specifying, the specific steps in step S2 are as follows:

[0039] Step S2.1, Time Alignment and Window Capture: Based on the trigger time, capture a single spot welding cycle as the key analysis window, with a duration of 200ms.

[0040] Step S2.2: Perform normalization processing, and independently perform Z-score normalization on each channel signal:

[0041]

[0042] in , Let represent the mean and standard deviation of the j-th channel, respectively.

[0043] Step S2.3, noise reduction preprocessing: The acoustic emission signal is decomposed into five layers using the Daubechies-4 wavelet basis. The high-frequency coefficients are reconstructed after soft thresholding. The soft thresholding function is used to shrink the high-frequency coefficients of each layer, retaining the main acoustic energy of the welding process and eliminating random noise, thereby improving the signal-to-noise ratio of subsequent feature extraction.

[0044] Further specifying, in step S3, the formula for extracting local mutation features from the preprocessed data using the three-level convolutional structure is: Conv1D + BatchNorm + ReLU. In the first level, Conv1D, the input channels = In the first stage of Conv1D, the input channels are 32, the output channels are 64, and the convolutional kernel is 5. In the second stage, the input channels are 64, the output channels are 128, and the convolutional kernel is 3. Corresponding to electro-thermal, acoustic, and visual modalities, respectively. Let the original number of channels for each modality be such that the output feature dimension is . , The number of convolutional kernels is gradually reduced and the number of channels is increased to determine the critical window time step. The final output contains a local feature map with 128 feature channels. Each convolutional layer is followed by a Batch Normalization layer to accelerate convergence, and the ReLU activation function is used to enhance the nonlinear expressive power.

[0045] Further specifying, the specific steps in step S4 are as follows:

[0046] Step S4.1, Dilated Causal Convolution Modeling: Six dilated causal convolution residual blocks are used to capture long-term temporal dependencies. Layer expansion factor is The residual block calculation logic is as follows:

[0047]

[0048] in, Indicates the first The input feature map of a single residual block is an intermediate feature representation of the welding process after multi-source temporal fusion.

[0049] Indicates the first The output feature maps of each dilated causal convolutional residual block are used as inputs for subsequent temporal modeling or quality identification modules;

[0050] Indicates the first In each residual block, the intermediate temporal features after the first layer of dilated causal convolution are calculated by dilated one-dimensional convolution, batch normalization, activation function and random deactivation.

[0051] Indicates the first In each residual block, the intermediate temporal features after the second layer of dilated causal convolution are used as the output of the residual branch;

[0052] Dropout o.2The Dropout operation, which represents a random inactivation rate of 0.2, is used to reduce the risk of model overfitting and improve the generalization ability under complex working conditions at the welding site.

[0053] This represents the linear mapping weight matrix in the residual connection. When the number of channels in the input features and the output features are inconsistent, channel alignment is achieved through 1×1 convolution.

[0054] The residual connection structure represents the input features With dilated convolution branch output Element-wise addition is performed to alleviate the gradient vanishing problem in deep temporal modeling.

[0055] It is a 1×1 convolution, enabled when the number of channels is mismatched.

[0056] Step S4.2, Sensing field coverage, setting the module's sensing field Step, in 10 At the sampling rate, corresponding to 12.7 The physical time is determined to ensure coverage of the timing context of critical welding stages, as shown in the formula:

[0057]

[0058] It can cover the key stages of a typical spot welding cycle.

[0059] Further specifying, the specific steps in step S5 are as follows:

[0060] Step S5.1, Projection and Attention Weight Calculation: Using the electro-thermal modal features as the query vector Acoustic and visual features are respectively and :

[0061]

[0062]

[0063]

[0064] Calculate attention weights to obtain

[0065]

[0066]

[0067] The fusion output is as follows:

[0068] Among them, H (e)This represents the electro-thermal modal characteristic sequence, used to characterize the timing information related to current, voltage, and heat input during the welding process; H (a) This represents the acoustic modal feature sequence, used to characterize the temporal characteristics of acoustic signals such as arc sound and molten pool vibration during welding; H (v) It represents the visual modal feature sequence, used to characterize the temporal features of visual information such as weld pool morphology and spatter status;

[0069] The temporal dimension T of the modal features represents the sequence length of each modal feature in the time dimension, corresponding to the number of time steps obtained by continuous sampling in the welding process;

[0070] W Q W K and W y The linear projection weight matrices represent the query, key, and value, respectively, used to map features from different modalities to a unified semantic space. The 64-dimensional semantic space represents the common feature dimension used in cross-modal attention computation, ensuring that features from different modalities are modeled for relevance at the same scale. This indicates that the electro-thermal modal features are used as the query vector to guide the response of other modes to welding quality-related information, K. (a) =H (a) W k V (a) =H (a) W y These represent the key vector and value vector generated from acoustic modal features, respectively, and are used to participate in cross-modal attention weight calculation and feature aggregation.

[0071] This represents the visual modal contextual feature output obtained through the same attention mechanism;

[0072] This indicates that, guided by the electro-thermal mode, key information from the acoustic and visual modes is fused using a residual method to form a unified cross-modal feature representation.

[0073] LayerNorm(.) represents the layer normalization operation, which is used to scale the fused features to improve stability under different welding conditions;

[0074] H fused This represents the final output feature of the cross-modal attention fusion module, which is used for subsequent welding quality identification or state determination.

[0075] H fused This represents the fused output features in tensor form with a time dimension of T and a feature dimension of 64.

[0076] A (a)This represents the attention weight matrix between the electro-thermal and acoustic modes, used to reflect the degree of correlation between features at different time steps.

[0077] Softmax(.) represents the normalization function used to map relevance scores to attention weights in the form of a probability distribution.

[0078] scaling factor This represents the scale normalization term in attention calculation, used to prevent the gradient from becoming unstable due to excessively large inner product values ​​of high-dimensional features.

[0079] This represents the contextual feature output after weighted aggregation of acoustic modal features based on attention weights.

[0080] Step S5.2: Determine the physical mechanism of electro-acoustic collaborative diagnosis. In the cross-modal attention fusion module, when an abnormal current fluctuation is detected in the electro-thermal mode at a certain time, the attention weight automatically enhances the high-frequency energy response of the corresponding time in the acoustic mode.

[0081] Further specifying, the specific steps in step S6 are as follows:

[0082] Step S6.1, the formula is: To eliminate the influence of differences in welding process duration on the state recognition results, a two-layer fully connected network (64→64→4) and Softmax are used to output four types of process state probabilities:

[0083]

[0084] Meanwhile, during training, a weighted cross-entropy loss is used, resulting in:

[0085]

[0086] The category weight is calculated as follows:

[0087]

[0088] Used to improve the discriminative weight of welding defect states with few samples during model training;

[0089] Among them, H fused H represents the multimodal fusion timing features of the welding process obtained after cross-modal attention fusion in step S5. fused This represents the feature vector corresponding to the fused feature at time step t, with a time length of [time value missing]. This indicates the effective time sequence length for participating in welding status determination, corresponding to the continuous sampling window during the welding process. This represents the global feature vector of the welding process obtained through global average pooling, which is used to comprehensively characterize the state features of the entire welding cycle.

[0090] The fully connected network (64→64→4) indicates that the welding state discrimination network consists of two fully connected layers, with an input feature dimension of 64 and an output of 4 types of welding process states;

[0091] This represents the unnormalized score (logit) output by the fully connected network for the c-th welding state.

[0092] The Softmax function is used to normalize the scores of each category into a probability distribution for the welding condition prediction results;

[0093] This represents the predicted probability that the welding process belongs to welding state c.

[0094] The weighted cross-entropy loss function is used for model optimization to constrain the classification performance of the welding state recognition model.

[0095] B represents the number of samples involved in the calculation during each model optimization process;

[0096] The model represents the first The true labels of each sample in the c-type welding state are represented by unique thermal coding.

[0097] The model represents the first The predicted probability that a sample belongs to the c-th welding state;

[0098] Data stability items Used to avoid numerical instability in logarithmic operations and ensure the robustness of the calculation process;

[0099] This represents the loss weight corresponding to the c-th welding state, used to alleviate the problem of uneven distribution of samples of different quality states in welding data;

[0100] This represents the total number of all welding samples in the training dataset;

[0101] This represents the number of welding state samples of class c in the training dataset;

[0102] Step S6.2: Extract the cross-modal attention matrix and Generate a time-modal attention heatmap; locate the peak weight moments in defect samples and perform time-frequency analysis by correlating them with the original signals.

[0103] Further specifying, the edge inference optimization and deployment in step S7 includes:

[0104] The model constraint resolution module constructs the theoretical expression for model inference time:

[0105]

[0106] in, For the effective memory bandwidth of edge computing devices, For the first Number of times the kernel function is started. This refers to the latency of a single kernel function execution.

[0107] The input timing trimming module trims the multi-source timing of the welding process to ensure that the input time step meets the requirements.

[0108] < And satisfy the monotonic constraint relationship between model inference time and time step length: >0, thereby limiting Directly reduce model inference latency;

[0109] The joint optimization module for structure pruning and quantization performs joint optimization of structure pruning and parameter quantization on the initial model. Structure pruning is used to reduce the computational cost of each layer. The parameter quantization uses a symmetric linear quantization method, mapping the floating-point weight W to... , and require to meet ;

[0110] The operator fusion and memory reuse module fuses consecutive computational operators during model inference and performs static memory pool reuse on intermediate feature tensors. This involves fusing consecutive operators, including at least convolution, normalization, and activation operators, into a single execution unit, enabling... By pre-allocating memory through a static memory pool, dynamic memory allocation is eliminated, thus... ,in, Determine the delay constant for a single dynamic memory allocation;

[0111] The knowledge distillation constraint module ensures that the student model parameter size meets the following requirements. And using the distillation loss function To constrain the performance degradation of the lightweight model in the welding quality identification task to not exceed a preset threshold;

[0112] Edge deployment and implementation of the inference module ensure that the total inference latency for single weld quality identification meets the requirements. ,in, The optimized model inference time

[0113] After optimization, industrial closed-loop and linkage control is implemented: the optimized model is deployed on the Jetson AGX Orin platform, and the measured single solder joint inference delay is 14.2±1.8ms. When an anomaly is detected, an alarm is triggered and parameter correction suggestions are generated. The process closed loop is realized by interfacing with PLC / MES through industrial Ethernet.

[0114] The advantages of this invention compared to the prior art are as follows:

[0115] 1. Multimodal deep collaboration significantly improves recognition accuracy. For the first time, it performs semantic-level fusion of four heterogeneous signals—electric, thermal, acoustic, and optical—through a cross-modal attention mechanism, breaking through the information limitations of single-modal threshold discrimination.

[0116] 2. The TCN architecture replaces BiLSTM, achieving an order-of-magnitude breakthrough in inference speed. It employs a Temporal Convolutional Network (TCN) instead of a traditional recurrent neural network, utilizing causal dilated convolutions and residual connections to achieve fully parallel computation. In actual testing on the Jetson AGX Orin edge platform, the single-solder-point inference latency was 14.2 milliseconds (including preprocessing), a 3.5-fold speedup compared to the BiLSTM solution, meeting the stringent real-time requirement of <20ms for automotive welding lines for the first time.

[0117] 3. Theoretical models guide edge optimization, significantly enhancing deployment reliability. An innovative quantitative relationship model for inference latency is constructed, establishing an explicit correlation between optimization measures (such as input pruning and quantization) and performance indicators, avoiding the blindness of empirical parameter tuning. The model size is compressed to 0.45MB (FP32→INT8), reducing memory access overhead by 76%, providing a reproducible and verifiable technical path for the deployment of industrial edge devices.

[0118] 4. Attention visualization empowers process optimization, creating a closed loop of "identification-diagnosis-improvement," and accurately pinpointing the moment of defect occurrence through cross-modal attention heatmaps (e.g., ...). (At the peak of acoustic modal weights), AI decisions are transformed into physical justifications that engineers can understand.

[0119] 5. Full-stack system integration, with strong engineering feasibility and industrial promotion value. It forms a complete technology chain from FPGA hardware synchronous data acquisition and lightweight algorithm design to TensorRT edge deployment, and has passed ISO 14327 welding quality standard verification. The system supports seamless integration with PLC / MES, enabling real-time uploading of quality data and closed-loop optimization of process parameters, making it suitable for high-end manufacturing scenarios such as automotive, rail transportation, and aerospace. Attached Figure Description

[0120] Figure 1 This is a schematic diagram illustrating the specific process of the present invention;

[0121] Figure 2 This is a detailed flowchart of step S1, multimodal signal acquisition, in this invention.

[0122] Figure 3 This is a detailed flowchart illustrating the signal preprocessing step S2 of the present invention.

[0123] Figure 4 This is a detailed flowchart illustrating step S7, edge inference optimization and deployment, of the present invention.

[0124] Figure 5 This is a module architecture diagram of the system in this invention. Detailed Implementation

[0125] To enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0126] Example:

[0127] like Figures 1-5 As shown, the resistance spot welding recognition method based on multimodal temporal fusion and temporal convolutional network is based on the construction of a four-in-one technical system of "signal acquisition → feature extraction → dynamic fusion → edge reasoning". It includes a multimodal collaborative perception module, a 1D-CNN and TCN cascaded encoding module, a cross-modal attention dynamic fusion module, a theory-driven edge optimization module, and an interpretable closed-loop output module. Among them, the multimodal collaborative perception module: synchronously acquires the electro-thermal main mode (current, voltage, pressure, melt temperature), acoustic auxiliary mode (acoustic emission signal), and visual auxiliary mode (indentation area, spark brightness, electrode displacement), and achieves sub-millisecond time alignment with the current rising edge as the hardware trigger reference.

[0128] 1D-CNN and TCN cascaded encoding module: Hardware-level time alignment is achieved using FPGA. A three-level one-dimensional convolutional neural network 1D-CNN is used to extract local mutation features (such as splash pulses and current drops). Then, a TCN module composed of six layers of dilated causal convolutional residual blocks is used to capture long temporal dependencies, taking into account both local sensitivity and global context awareness.

[0129] Cross-modal attention dynamic fusion module: Introduces a cross-modal attention mechanism, dynamically fuses auxiliary modal information with electro-thermal features as the main driver, calculates its attention weights with acoustic and visual modalities, realizes the enhanced causal correlation between "electrical anomaly moment - acoustic emission response", and generates highly discriminative fusion features;

[0130] Theory-driven edge optimization module: This module optimizes edge deployment by constructing a four-channel parallel acquisition architecture (electric, thermal, acoustic, and visual), establishing a quantitative relationship model between inference latency and input length, model structure, and hardware resources, and implementing input pruning within the system. Five optimizations, including INT8 quantization, operator fusion, memory pool reuse, and knowledge distillation, ensure that the single solder joint inference latency is consistently below 20 milliseconds.

[0131] Interpretable closed-loop output module: generates cross-modal attention heatmaps, accurately locates the time of defect occurrence and dominant mode, and provides physical root cause analysis while outputting process status judgment results.

[0132] The technical advantages of this identification method are reflected in three dimensions: At the perception level, it enables the four-source signals of electricity, heat, sound, and vision to have true temporal comparability, laying a data foundation for subsequent fusion analysis; At the representation level, the cascaded architecture of 1D-CNN and TCN achieves hierarchical decoupling of "local transient capture - long-range trend modeling". Compared with a single LSTM or GRU network, it expands the receptive field to the entire welding cycle while maintaining parallel computing efficiency, effectively identifying latent defects with a latency of hundreds of milliseconds; At the decision level, the cross-modal attention mechanism breaks through the simple weighted mode of early fusion and late fusion, and establishes a dynamic interaction mechanism of "dominant modality driven and auxiliary modality corroborated", enabling the model to adaptively respond to different process disturbance scenarios.

[0133] The theory-driven edge optimization module also includes:

[0134] Model constraint parsing module: Used to obtain the initial welding quality model and parse the hierarchical structure parameters and inference constraint parameters of the model. The inference constraint parameters include: the number of network layers L, the computational cost of each layer, etc. Number of feature channels Input timing length ;

[0135] Input timing trimming module: Used to trim the multi-source timing sequence of the welding process without changing the physical time sequence of the welding process.

[0136] Joint optimization module for structure pruning and quantization: Used to perform joint optimization of structure pruning and parameter quantization on the initial model while maintaining the consistency of the model topology;

[0137] Operator fusion and memory reuse module: used to fuse continuous computation operators in the model inference process and perform static memory pool reuse on intermediate feature tensors;

[0138] Knowledge distillation constraint module: used to introduce teacher models to constrain student models during model lightweighting.

[0139] Edge Deployment and Implementation Inference Module: Used to deploy the optimized welding quality recognition model to edge computing devices for real-time inference.

[0140] The specific steps are as follows:

[0141] Step S1: Synchronously acquire multi-mode signals during the resistance spot welding process;

[0142] Step S1.1: Configure the sensor array to construct a monitoring network composed of multiple physical quantity sensors.

[0143] Including electro-thermal modes: using Hall current sensors and voltage divider circuits to obtain welding current. With voltage The pressure sensor obtains the electrode pressure. Infrared thermopile sensors monitor the temperature of the molten core region. ;

[0144] Acoustic modes: Piezoelectric acoustic emission sensors are used to capture elastic waves generated by welding spatter and lattice fracture;

[0145] Visual modality: Capturing electrode displacement using a high-speed industrial camera Spark brightness and indentation morphology characteristics;

[0146] The welding current / voltage sampling rate is 10kHz with an accuracy of 0.5%FS; the infrared thermopile response time is <0.1ms; the acoustic emission sensor has a frequency band of 50-400kHz; and the high-speed industrial camera operates at 1000fps.

[0147] Step S1.2: Design a sub-millisecond synchronization mechanism based on FPGA. Use an FPGA module to monitor the rising edge of the welding circuit current as a hardware trigger signal to synchronously activate all sensor channels, ensuring the time alignment error of the electrical, acoustic, and optical multimodal signals. ;

[0148] Step S2: Preprocess the multimodal signal;

[0149] Step S2.1, Time Alignment and Window Capture: Based on the trigger time, capture a single spot welding cycle as the key analysis window, with a duration of 200ms.

[0150] Step S2.2: Perform normalization processing, and independently perform Z-score normalization on each channel signal:

[0151]

[0152] in , Let represent the mean and standard deviation of the j-th channel, respectively.

[0153] Step S2.3, noise reduction preprocessing: The acoustic emission signal is decomposed into five layers using the Daubechies-4 wavelet basis. The high-frequency coefficients are reconstructed after soft thresholding. The high-frequency coefficients of each layer are shrunk using the soft thresholding function to retain the main acoustic energy of the welding process and remove random noise, thereby improving the signal-to-noise ratio of subsequent feature extraction and performing data augmentation. To address the problem of scarce welding defect samples, a combined strategy of temporal perturbation and modal dropout is adopted: ±5% amplitude scaling and 10ms time shift are applied to the electro-thermal signal, spectral masking is performed on the acoustic signal (randomly blocking 20% ​​of the frequency band), and the visual modality is zeroed out with a probability of 0.3, effectively expanding the diversity of training samples and enhancing the generalization ability of the model.

[0154] Step S3: Based on local feature extraction using a one-dimensional convolutional neural network, the preprocessed data is input into a three-level convolutional structure to extract local mutation features (such as current pulse mutations caused by splashing). The formula is: Conv1D + BatchNorm + ReLU. In the first-level Conv1D, the input channel = In the first stage of Conv1D, the input channels are 32, the output channels are 64, and the convolutional kernel is 5. In the second stage, the input channels are 64, the output channels are 128, and the convolutional kernel is 3. Corresponding to electro-thermal, acoustic, and visual modalities, respectively. Let the original number of channels for each modality be such that the output feature dimension is . , The number of convolutional kernels is gradually reduced and the number of channels is increased to determine the critical window time step. The final output contains a local feature map with 128 feature channels. Each convolutional layer is followed by a Batch Normalization (BN) layer to accelerate convergence, and the ReLU activation function is used to enhance the nonlinear expressive power.

[0155] Step S4: Design a global temporal model based on a temporal convolutional network (TCN);

[0156] Step S4.1, Dilated Causal Convolution Modeling: Six dilated causal convolution residual blocks are used to capture long-term temporal dependencies. Layer expansion factor is The residual block calculation logic is as follows:

[0157]

[0158] in, Indicates the first The input feature map of a single residual block is an intermediate feature representation of the welding process after multi-source temporal fusion.

[0159] Indicates the first The output feature maps of each dilated causal convolutional residual block are used as inputs for subsequent temporal modeling or quality identification modules;

[0160] Indicates the first In each residual block, the intermediate temporal features after the first layer of dilated causal convolution are calculated by dilated one-dimensional convolution, batch normalization, activation function and random deactivation.

[0161] Indicates the first In each residual block, the intermediate temporal features after the second layer of dilated causal convolution are used as the output of the residual branch;

[0162] Dropout o.2 The Dropout operation, which represents a random inactivation rate of 0.2, is used to reduce the risk of model overfitting and improve the generalization ability under complex working conditions at the welding site.

[0163] This represents the linear mapping weight matrix in the residual connection. When the number of channels in the input features and the output features are inconsistent, channel alignment is achieved through 1×1 convolution.

[0164] The residual connection structure representation represents the input features With dilated convolution branch output Element-wise addition is performed to alleviate the gradient vanishing problem in deep temporal modeling.

[0165] It is a 1×1 convolution, enabled when the number of channels is mismatched;

[0166] Step S4.2, Sensing field coverage, setting the module's sensing field Step, in 10 At the sampling rate, it corresponds to approximately 12.7. The physical time can fully cover the temporal context dependencies of the critical moments in nucleus growth, as shown in the formula:

[0167]

[0168] It can cover the critical stages of a typical spot welding cycle (300–1500ms).

[0169] Step S5, fusion and collaborative diagnosis based on cross-modal attention;

[0170] Step S5.1, Projection and Attention Weight Calculation: Using the electro-thermal modal features as the query vector Acoustic and visual features are respectively and ,

[0171]

[0172]

[0173]

[0174] Calculate the attention weights and combine them with the context to obtain:

[0175]

[0176]

[0177] The fusion output is as follows:

[0178] Among them, H (e) This represents the electro-thermal modal characteristic sequence, used to characterize the timing information related to current, voltage, and heat input during the welding process; H (a) This represents the acoustic modal feature sequence, used to characterize the temporal characteristics of acoustic signals such as arc sound and molten pool vibration during welding; H (v) It represents the visual modal feature sequence, used to characterize the temporal features of visual information such as weld pool morphology and spatter status;

[0179] The temporal dimension T of the modal features represents the sequence length of each modal feature in the time dimension, corresponding to the number of time steps obtained by continuous sampling in the welding process;

[0180] W Q W K and W y The linear projection weight matrices represent the query, key, and value, respectively, used to map features from different modalities to a unified semantic space. The 64-dimensional semantic space represents the common feature dimension used in cross-modal attention computation, ensuring that features from different modalities are modeled for relevance at the same scale. This indicates that the electro-thermal modal features are used as the query vector to guide the response of other modes to welding quality-related information, K. (a) =H (a) W k V (a) =H (a) W y These represent the key vector and value vector generated from acoustic modal features, respectively, and are used to participate in cross-modal attention weight calculation and feature aggregation.

[0181] This represents the visual modal contextual feature output obtained through the same attention mechanism;

[0182] This indicates that, guided by the electro-thermal mode, key information from the acoustic and visual modes is fused using a residual method to form a unified cross-modal feature representation.

[0183] LayerNorm(.) represents the layer normalization operation, which is used to scale the fused features to improve stability under different welding conditions;

[0184] H fused This represents the final output feature of the cross-modal attention fusion module, which is used for subsequent welding quality identification or state determination.

[0185] H fused This represents the fused output features in tensor form with a time dimension of T and a feature dimension of 64.

[0186] A (a) This represents the attention weight matrix between the electro-thermal and acoustic modes, used to reflect the degree of correlation between features at different time steps.

[0187] Softmax(.) represents the normalization function used to map relevance scores to attention weights in the form of a probability distribution.

[0188] scaling factor This represents the scale normalization term in attention calculation, used to prevent the gradient from becoming unstable due to excessively large inner product values ​​of high-dimensional features.

[0189] This represents the contextual feature output after weighted aggregation of acoustic modal features based on attention weights;

[0190] Step S5.2: Determine the physical mechanism of electro-acoustic collaborative diagnosis. In the cross-modal attention fusion module, when an abnormal current fluctuation is detected in the electro-thermal mode at a certain time, the acoustic and visual features are weighted and fused based on the attention weights. The attention weights automatically enhance the high-frequency energy response in the acoustic mode at the corresponding time. The physical mechanism is that the transient acoustic emission signal generated by splashes or microcracks has a causal relationship with the current anomaly in time. Accurate identification is achieved by enhancing the evidence weight at this time position.

[0191] Step S6, Classification Decision and Interpretability Analysis;

[0192] Step S6.1, perform global average pooling on Hfused, with the following formula: To eliminate the influence of differences in welding process duration on the state recognition results, a two-layer fully connected network (64→64→4) and Softmax are used to output four types of process state probabilities.

[0193] The four process state probabilities are output through a two-layer fully connected network 64→64→4 and Softmax:

[0194]

[0195] Meanwhile, during training, a weighted cross-entropy loss is used, resulting in:

[0196]

[0197] The category weight is calculated as follows:

[0198]

[0199] Used to improve the discriminative weight of welding defect states with few samples during model training;

[0200] Among them, H fused H represents the multimodal fusion timing features of the welding process obtained after cross-modal attention fusion in step S5. fused This represents the feature vector corresponding to the fused feature at time step t, with a time length of [time value missing]. This indicates the effective time sequence length for participating in welding status determination, corresponding to the continuous sampling window during the welding process. This represents the global feature vector of the welding process obtained through global average pooling, which is used to comprehensively characterize the state features of the entire welding cycle.

[0201] The fully connected network (64→64→4) indicates that the welding state discrimination network consists of two fully connected layers, with an input feature dimension of 64 and an output of 4 types of welding process states;

[0202] This represents the unnormalized score (logit) output by the fully connected network for the c-th welding state.

[0203] The Softmax function is used to normalize the scores of each category into a probability distribution for the welding condition prediction results;

[0204] This represents the predicted probability that the welding process belongs to welding state c.

[0205] The weighted cross-entropy loss function is used for model optimization to constrain the classification performance of the welding state recognition model.

[0206] B represents the number of samples involved in the calculation during each model optimization process;

[0207] The model represents the first The true labels of each sample in the c-type welding state are represented by unique thermal coding.

[0208] The model represents the first The predicted probability that a sample belongs to the c-th welding state;

[0209] Data stability items Used to avoid numerical instability in logarithmic operations and ensure the robustness of the calculation process;

[0210] This represents the loss weight corresponding to the c-th welding state, used to alleviate the problem of uneven distribution of samples of different quality states in welding data;

[0211] This represents the total number of all welding samples in the training dataset;

[0212] This represents the number of welding state samples of class c in the training dataset;

[0213] Step S6.2: Extract the cross-modal attention matrix and Generate time-modal attention heatmaps; locate the peak weight moments in defect samples and perform time-frequency analysis by correlating them with the original signals;

[0214] Step S7, perform edge inference optimization and deployment, including:

[0215] The model constraint resolution module constructs the theoretical expression for model inference time:

[0216]

[0217] in, For the effective memory bandwidth of edge computing devices, For the first Number of times the kernel function is started. This refers to the latency of a single kernel function execution.

[0218] The input timing trimming module trims the multi-source timing of the welding process to ensure that the input time step meets the requirements.

[0219] < And satisfy the monotonic constraint relationship between model inference time and time step length: >0, thereby limiting Directly reduce model inference latency;

[0220] The joint optimization module for structure pruning and quantization performs joint optimization of structure pruning and parameter quantization on the initial model. Structure pruning is used to reduce the computational cost of each layer. The parameter quantization uses a symmetric linear quantization method, mapping the floating-point weight W to... ,

[0221] And require to meet ;

[0222] The operator fusion and memory reuse module fuses consecutive computational operators during model inference and performs static memory pool reuse on intermediate feature tensors. This involves fusing consecutive operators, including at least convolution, normalization, and activation operators, into a single execution unit, enabling... By pre-allocating memory through a static memory pool, dynamic memory allocation is eliminated, thus... ,in, Determine the delay constant for a single dynamic memory allocation;

[0223] The knowledge distillation constraint module ensures that the student model parameter size meets the following requirements. And using the distillation loss function To constrain the performance degradation of the lightweight model in the welding quality identification task to not exceed a preset threshold;

[0224] Edge deployment and implementation of the inference module ensure that the total inference latency for single weld quality identification meets the requirements. ,in, This represents the optimized model inference time.

[0225] After optimization, industrial closed-loop and linkage control is implemented: the optimized model is deployed on the Jetson AGX Orin platform, and the measured single solder joint inference delay is 14.2±1.8ms. When an anomaly is detected, an alarm is triggered and parameter correction suggestions are generated. The process closed loop is realized by interfacing with PLC / MES through industrial Ethernet.

[0226] This method demonstrates significant engineering adaptability and diagnostic reliability in real industrial environments. Notably, the cross-modal attention heatmap has successfully revealed the physical evolution of various complex defects in practical applications: when uneven wear occurs on the electrode cap end face, the electro-thermal mode shows a gradual decrease in current density, while the visual mode simultaneously captures an increase in indentation ellipticity, and the acoustic mode shows an abnormal high-frequency energy release during the solidification stage of the melt nucleus. The temporal correlation of these three factors provides a quantitative basis for the dynamic optimization of the electrode regrinding cycle.

[0227] At the system integration level, this method achieves seamless integration with mainstream welding controllers through a standardized API interface. The electro-thermal signal acquisition module uses an isolated Hall sensor and a 24-bit Σ-Δ ADC, achieving an effective bit accuracy of 0.05% at a 10kHz sampling rate. The FPGA synchronous trigger unit is built based on the Xilinx Zynq-7000 series, using programmable logic units to achieve nanosecond-level detection of current rising edges and multi-channel DMA transmission, with time alignment jitter controlled within 50ns. The acoustic signal processing link incorporates bandpass filtering and programmable gain amplification, performing anti-aliasing processing on acoustic emission signals in the 50-400kHz frequency band, resulting in a 18dB improvement in signal-to-noise ratio after wavelet denoising. The visual modality employs a global shutter CMOS sensor with active infrared illumination, maintaining a 1280×1024 resolution at a 1000fps frame rate. A GPU-accelerated sub-pixel edge detection algorithm extracts the electrode displacement curve, achieving a displacement measurement accuracy better than 5μm.

[0228] During the model training phase, a labeled dataset covering typical process windows was constructed, containing 470,000 samples across six states: normal weld points, spatter, lack of fusion, cracks, cold welds, and burn-through. These samples were collected from production conditions with varying plate thickness combinations (0.8+0.8mm to 2.0+2.5mm), coating types (galvanized, bare steel, aluminum-silicon coating), and electrode states (new electrode, moderate wear, severe wear). To address the class imbalance problem, in addition to temporal perturbations and modal dropout, Focal Loss was introduced to adaptively weight difficult-to-classify samples, improving the model's F1-score for defect categories with fewer samples from 0.71 to 0.89. The application of transfer learning further shortened the model adaptation cycle during production line switching. When the plate thickness combination changed, only 500-800 new samples needed for fine-tuning to restore the original accuracy level, representing an efficiency improvement of over 20 times compared to full retraining.

[0229] The optimization effect of edge deployment is particularly prominent in hardware resource-constrained scenarios. The Jetson AGX Orin platform, in 15W power mode, achieves a model inference throughput of 70 FPS after five optimizations, while simultaneously meeting the real-time requirements of parallel monitoring of multiple welding torches. The memory pool reuse mechanism reduces tensor allocation overhead from an average of 3.2ms to 0.15ms. The model size after INT8 quantization is compressed from 127MB of the original FP32 to 34MB, while the Top-1 accuracy only decreases by 1.3 percentage points, which is within an acceptable range for engineering applications. During knowledge distillation, a six-layer TCN teacher model guides a three-layer TCN student model. Attention transfer loss is used to force alignment of their cross-modal attention distributions, enabling the student model to maintain over 94% of the defect localization capability of the teacher model while reducing the number of parameters by 62%.

[0230] The interpretable output module plays a crucial role in the closed-loop process optimization. The attention heatmap generated by the system shows a high degree of consistency with the physical simulation results: in the case of splash defects, the peak time of attention weight deviates from the CFD simulation time of molten metal splashing from the melt nugget by less than 5ms; in the case of non-fusion defects, the changing trend of the attention weight ratio between the electro-thermal mode and the visual mode matches the ultrasonic detection results of the melt nugget diameter. This two-way verification mechanism of "data-driven - physical verification" significantly enhances the field engineers' trust in the intelligent diagnostic system and promotes the transformation of the process management model from "experience-based modeling" to "data-based modeling". The statistical characteristics of attention weight accumulated during system operation can also be used to build electrode life prediction models, realizing the upgrade of maintenance strategies from periodic replacement to predictive maintenance.

[0231] The above provides a detailed description of a multimodal timing quality real-time identification system and method for resistance spot welding provided by the present invention. The description of specific embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A multimodal timing quality real-time identification system for resistance spot welding, characterized in that: The system comprises a multimodal collaborative sensing module, a 1D-CNN and TCN concatenated encoding module, a cross-modal attention dynamic fusion module, a theory-driven edge optimization module, and an interpretable closed-loop output module. The multimodal collaborative sensing module simultaneously acquires three types of multimodal signals: electrical-thermal, acoustic, and visual. The 1D-CNN and TCN concatenated encoding module utilizes FPGA for hardware-level time alignment, extracts local mutation features through a one-dimensional convolutional neural network (1D-CNN), and captures long-term temporal dependencies using a temporal convolutional network (TCN). The cross-modal attention dynamic fusion module introduces a cross-modal attention mechanism, dynamically fusing auxiliary modal information with electrical-thermal features as the primary driver. The theory-driven edge optimization module optimizes the model through edge deployment, constructs a four-channel parallel acquisition architecture (electrical-thermal-acoustic-visual), and establishes a quantitative relationship model between inference latency and input length, model structure, and hardware resources. The interpretable closed-loop output module generates a cross-modal attention heatmap, accurately locating the time of defect occurrence and the dominant modality, outputting process status judgment results while providing physical root cause analysis.

2. The multimodal timing quality real-time identification system for resistance spot welding according to claim 1, characterized in that: The theory-driven edge optimization module further includes: a model constraint parsing module: used to obtain the initial welding quality model and parse the hierarchical structure parameters and inference constraint parameters of the model. The inference constraint parameters include: the number of network layers L, the computational cost of each layer, etc. Number of feature channels Input timing length Input timing trimming module: Used to trim the multi-source timing sequence of the welding process without changing the physical time sequence of the welding process. Joint optimization module for structure pruning and quantization: Used to perform joint optimization of structure pruning and parameter quantization on the initial model while maintaining the consistency of the model topology; Operator fusion and memory reuse module: used to fuse continuous computation operators in the model inference process and perform static memory pool reuse on intermediate feature tensors; Knowledge distillation constraint module: used to introduce teacher models to constrain student models during model lightweighting. Edge Deployment and Implementation Inference Module: This module is used to deploy the optimized welding quality recognition model to edge computing devices for real-time inference.

3. The method for a real-time identification system of multimodal timing quality for resistance spot welding according to claim 1, characterized in that: The specific steps are as follows: Step S1: Synchronously acquire multi-mode signals during the resistance spot welding process; Step S2: Preprocess the multimodal signal; Step S3: Based on the local feature extraction of a one-dimensional convolutional neural network, the preprocessed data is input into a three-level convolutional structure to extract local mutation features; Step S4: Design a global temporal model based on a temporal convolutional network (TCN); Step S5: Fusion and collaborative diagnosis based on cross-modal attention; Step S6, Classification Decision and Interpretability Analysis; Step S7: Perform edge inference optimization and deployment.

4. The method for a real-time identification system of multimodal timing quality for resistance spot welding according to claim 3, characterized in that: Step S1 involves constructing a four-channel parallel acquisition architecture (electric, thermal, acoustic, and visual), and the specific steps are as follows: Step S1.1: Configure the sensor array to construct a monitoring network composed of multiple physical quantity sensors. Including electro-thermal modes: using Hall current sensors and voltage divider circuits to obtain welding current. With voltage The pressure sensor obtains the electrode pressure. Infrared thermopile sensors monitor the temperature of the molten core region. ; Acoustic modes: Piezoelectric acoustic emission sensors are used to capture elastic waves generated by welding spatter and lattice fracture; Visual modality: Capturing electrode displacement using a high-speed industrial camera Spark brightness and indentation morphology characteristics; The welding current / voltage sampling rate is 10kHz with an accuracy of 0.5%FS; the infrared thermopile response time is <0.1ms; the acoustic emission sensor has a frequency band of 50-400kHz; and the high-speed industrial camera operates at 1000fps. Step S1.2: Design a sub-millisecond synchronization mechanism based on FPGA. Use an FPGA module to monitor the rising edge of the welding circuit current as a hardware trigger signal to synchronously activate all sensor channels, ensuring the time alignment error of the electrical, acoustic, and optical multimodal signals. .

5. The method for a real-time identification system of multimodal timing quality for resistance spot welding according to claim 3, characterized in that: The specific steps in step S2 are as follows: Step S2.1, Time Alignment and Window Capture: Based on the trigger time, capture a single spot welding cycle as the key analysis window, with a duration of 200ms. Step S2.2: Perform normalization processing, and independently perform Z-score normalization on each channel signal: ，， in , Let represent the mean and standard deviation of the j-th channel, respectively. Step S2.3, noise reduction preprocessing: The acoustic emission signal is decomposed into five layers using the Daubechies-4 wavelet basis. The high-frequency coefficients are reconstructed after soft thresholding. The soft thresholding function is used to shrink the high-frequency coefficients of each layer, retaining the main acoustic energy of the welding process and eliminating random noise, thereby improving the signal-to-noise ratio of subsequent feature extraction.

6. The method for a real-time identification system of multimodal timing quality for resistance spot welding according to claim 3, characterized in that... In step S3, the formula for extracting local mutation features from the preprocessed data by inputting it into a three-level convolutional structure is: Conv1D + BatchNorm + ReLU. In the first level, Conv1D, the input channels = In the first stage of Conv1D, the input channels are 32, the output channels are 64, and the convolutional kernel is 5. In the second stage, the input channels are 64, the output channels are 128, and the convolutional kernel is 3. Corresponding to electro-thermal, acoustic, and visual modalities, respectively. Let the original number of channels for each modality be such that the output feature dimension is . , The number of convolutional kernels is gradually reduced and the number of channels is increased to determine the critical window time step. The final output contains a local feature map with 128 feature channels. Each convolutional layer is followed by a BatchNormalization layer to accelerate convergence, and the ReLU activation function is used to enhance the nonlinear expressive power.

7. The method for a real-time identification system of multimodal timing quality for resistance spot welding according to claim 3, characterized in that: The specific steps in step S4 are as follows: Step S4.1, Dilated Causal Convolution Modeling: Six dilated causal convolution residual blocks are used to capture long-term temporal dependencies. Layer expansion factor is The residual block calculation logic is as follows: ， in, Indicates the first The input feature map of a single residual block is an intermediate feature representation of the welding process after multi-source temporal fusion. Indicates the first The output feature maps of each dilated causal convolutional residual block are used as inputs for subsequent temporal modeling or quality identification modules; Indicates the first In each residual block, the intermediate temporal features after the first layer of dilated causal convolution are calculated by dilated one-dimensional convolution, batch normalization, activation function and random deactivation. Indicates the first In each residual block, the intermediate temporal features after the second layer of dilated causal convolution are used as the output of the residual branch; Dropout o.2 The Dropout operation, which represents a random inactivation rate of 0.2, is used to reduce the risk of model overfitting and improve the generalization ability under complex working conditions at the welding site. This represents the linear mapping weight matrix in the residual connection. When the number of channels in the input features and the output features are inconsistent, channel alignment is achieved through 1×1 convolution. The residual connection structure represents the input features With dilated convolution branch output Element-wise addition is performed to alleviate the gradient vanishing problem in deep temporal modeling. It is a 1×1 convolution, which is enabled when the number of channels does not match; Step S4.2, Sensing field coverage, setting the module's sensing field Step, in 10 At the sampling rate, corresponding to 12.7 The physical time is determined to ensure coverage of the timing context of critical welding stages, as shown in the formula: ， It can cover the key stages of a typical spot welding cycle.

8. The method for a real-time identification system of multimodal timing quality for resistance spot welding according to claim 3, characterized in that: The specific steps in step S5 are as follows: Step S5.1, Projection and Attention Weight Calculation: Using the electro-thermal modal features as the query vector Acoustic and visual features are respectively and : ， ， ， The attention weights are calculated as follows: ， ， The fusion output is as follows: ， Among them, H (e) This represents the electro-thermal modal characteristic sequence, used to characterize the timing information related to current, voltage, and heat input during the welding process; H (a) This represents the acoustic modal feature sequence, used to characterize the temporal characteristics of acoustic signals such as arc sound and molten pool vibration during welding; H (v) It represents the visual modal feature sequence, used to characterize the temporal features of visual information such as weld pool morphology and spatter status; The temporal dimension T of the modal features represents the sequence length of each modal feature in the time dimension, corresponding to the number of time steps obtained by continuous sampling in the welding process; W Q W K and W y The linear projection weight matrices represent the query, key, and value, respectively, used to map features from different modalities to a unified semantic space. The 64-dimensional semantic space represents the common feature dimension used in cross-modal attention computation, ensuring that features from different modalities are modeled for relevance at the same scale. This indicates that the electro-thermal modal features are used as the query vector to guide the response of other modes to welding quality-related information, K. (a) =H (a) W k V (a) =H (a) W y These represent the key vector and value vector generated from acoustic modal features, respectively, used in cross-modal attention weight calculation and feature aggregation. This represents the visual modal contextual feature output obtained through the same attention mechanism; This indicates that, guided by the electro-thermal mode, key information from the acoustic and visual modes is fused using a residual method to form a unified cross-modal feature representation. LayerNorm(.) represents the layer normalization operation, which is used to scale the fused features to improve stability under different welding conditions; H fused This represents the final output feature of the cross-modal attention fusion module, which is used for subsequent welding quality identification or state determination. H fused This represents the fused output features in tensor form with a time dimension of T and a feature dimension of 64. A (a) This represents the attention weight matrix between the electro-thermal and acoustic modes, used to reflect the degree of correlation between features at different time steps. Softmax(.) represents the normalization function used to map relevance scores to attention weights in the form of a probability distribution. scaling factor This represents the scale normalization term in attention calculation, used to prevent the gradient from becoming unstable due to excessively large inner product values ​​of high-dimensional features. This represents the contextual feature output after weighted aggregation of acoustic modal features based on attention weights; Step S5.2: Determine the physical mechanism of electro-acoustic collaborative diagnosis. In the cross-modal attention fusion module, when an abnormal current fluctuation is detected in the electro-thermal mode at a certain time, the acoustic and visual features are weighted and fused based on the attention weight. The attention weight automatically enhances the high-frequency energy response in the acoustic mode at the corresponding time.

9. The method for a real-time identification system of multimodal timing quality for resistance spot welding according to claim 3, characterized in that: The specific steps in step S6 are as follows: Step S6.1, for H fused Perform global average pooling, the formula is: To eliminate the influence of differences in welding process duration on the state recognition results, a two-layer fully connected network (64→64→4) and Softmax are used to output four types of process state probabilities: ， Meanwhile, during training, a weighted cross-entropy loss is used, resulting in: ， The category weight is calculated as follows: ， Used to improve the discriminative weight of welding defect states with few samples during model training; Among them, H fused H represents the multimodal fusion timing features of the welding process obtained after cross-modal attention fusion in step S5. fused This represents the feature vector corresponding to the fused feature at time step t, with a time length of [time value missing]. This indicates the effective time sequence length for participating in welding status determination, corresponding to the continuous sampling window during the welding process. This represents the global feature vector of the welding process obtained through global average pooling, used to comprehensively characterize the state features of the entire welding cycle. The fully connected network (64→64→4) indicates that the welding state discrimination network consists of two fully connected layers, with an input feature dimension of 64 and an output of 4 types of welding process states; This represents the unnormalized score (logit) output by the fully connected network for the c-th welding state. The Softmax function is used to normalize the scores of each category into a probability distribution for the welding condition prediction results; This represents the predicted probability that the welding process belongs to welding state c. The weighted cross-entropy loss function is used for model optimization to constrain the classification performance of the welding state recognition model. B represents the number of samples involved in the calculation during each model optimization process; The model represents the first The true labels of each sample in the c-type welding condition are represented by unique thermal coding. The model represents the first The predicted probability that a sample belongs to the c-th welding state; Data stability items Used to avoid numerical instability in logarithmic operations and ensure the robustness of the calculation process; This represents the loss weight corresponding to the c-th welding state, used to alleviate the problem of uneven distribution of samples of different quality states in welding data; This represents the total number of all welding samples in the training dataset; This represents the number of welding state samples of class c in the training dataset; Step S6.2: Extract the cross-modal attention matrix and Generate a time-modal attention heatmap; locate the peak weight moments in defect samples and perform time-frequency analysis by correlating them with the original signals.

10. The method for a real-time identification system of multimodal timing quality for resistance spot welding according to claim 2, characterized in that: The edge inference optimization and deployment in step S7 includes: The model constraint resolution module constructs the theoretical expression for model inference time: ， in, For the effective memory bandwidth of edge computing devices, For the first Number of times the kernel function is started. This refers to the latency of a single kernel function execution. The input timing trimming module trims the multi-source timing of the welding process to ensure that the input time step meets the requirements. < And satisfy the monotonic constraint relationship between model inference time and time step length: >0, thereby limiting Directly reduce model inference latency; The joint optimization module for structure pruning and quantization performs joint optimization of structure pruning and parameter quantization on the initial model. Structure pruning is used to reduce the computational cost of each layer. The parameter quantization uses a symmetric linear quantization method, mapping the floating-point weight W to... , And require to meet ; The operator fusion and memory reuse module fuses consecutive computational operators during model inference and performs static memory pool reuse on intermediate feature tensors. This involves fusing consecutive operators, including at least convolution, normalization, and activation operators, into a single execution unit, enabling... By pre-allocating memory through a static memory pool, dynamic memory allocation is eliminated, thus... ,in, Determine the delay constant for a single dynamic memory allocation; The knowledge distillation constraint module ensures that the student model parameter size meets the following requirements. And using the distillation loss function To constrain the performance degradation of the lightweight model in the welding quality identification task to not exceed a preset threshold; Edge deployment and implementation of the inference module ensure that the total inference latency for single weld quality identification meets the requirements. ,in, The optimized model inference time After optimization, industrial closed-loop and linkage control is implemented: the optimized model is deployed on the Jetson AGX Orin platform, and the measured single solder joint inference delay is 14.2±1.8ms. When an anomaly is detected, an alarm is triggered and parameter correction suggestions are generated. The process closed loop is realized by interfacing with PLC / MES through industrial Ethernet.

Images