Resistance welding process multi-sensor data fusion method considering time sequence characteristics

Abstract

The invention discloses a resistance welding process multi-sensor data fusion method considering time sequence characteristics, which comprises the following steps of: identifying trend change of time sequence data of a resistance welding process by utilizing an extended autocorrelation function, and performing stage segmentation on a time sequence according to a trend change identification result, determining a demarcation point of each stage to obtain a stage set; processing each stage of self-attention fusion through a bilinear parallel relation to obtain a final feature embedding result of the sensor; performing multi-channel stage feature extraction on a plurality of sensors, converting a plurality of extracted features into nodes of a heterogeneous graph, converting a relationship among the features into edges, and measuring node correlation by adopting covariance; and defining an edge sampling function, updating a node state by using an attention mechanism, randomly generating the size of a new node expansion graph, and completing fusion of multi-sensor data. According to the method, a good effect can be achieved when a time sequence with more stages and larger fluctuation is processed, and the data fusion precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing in the resistance welding process, and particularly to a multi-sensor data fusion method for the resistance welding process considering temporal characteristics. Background Art

[0002] In industrial processes, there are usually different production stages, and the influence degrees of different stages on product quality are different. However, deep learning models usually perform poorly in processing industrial datasets. These models often ignore the importance of each stage in industrial time series data and are unable to capture the complex relationships between multiple series. For example, in a transformer-based model, more attention is easily obtained in the descending stage, while less attention is paid in the critical stage. The main reason for this excessive focus on unimportant stages is the insufficient embedding method, which makes it difficult for the model to learn the key stage features. Traditional models usually use positional encoding and time windows to embed time series data. However, using positional encoding to embed time series data causes the model to focus on stages with large changes and low oscillation frequencies and be insensitive to certain abnormal fluctuations. But if time window encoding is used, serious embedding redundancy will occur. Therefore, applying the above data processing methods to the resistance welding industrial process still has many drawbacks:

[0003] First, traditional temporal data processing methods have poor effects. For example, the dynamic time warping (DTW) algorithm and the autoregressive integrated moving average (ARIMA) model usually assume that time series data has a fixed and predictable pattern. However, the data in the resistance welding process is complex and variable, and this assumption makes these methods unable to effectively process welding data and difficult to accurately reflect the real situation of the welding process.

[0004] Second, deep time series data models have limitations. For example, the temporal convolutional network (TCN) and the cross convolutional model (XCM). Due to the limitations of the convolutional kernel, it is difficult to capture the relationship between industrial time series data in the time domain and the frequency domain stages. In resistance welding data processing, it is impossible to accurately grasp the internal connection of data in different stages of the welding process, thereby affecting the judgment and control of welding quality.

[0005] Third, the industrial time series embedding method is imperfect. When dealing with industrial time series data, positional encoding has certain advantages over traditional methods. However, due to the periodicity, redundancy, volatility, and mutation of industrial time series data, positional encoding is often difficult to fully embed. For example, in resistance welding, the fluctuations and mutations of welding current, voltage, etc. will make positional encoding unable to accurately represent data features and affect the model's recognition of welding states.

[0006] Fourth, existing models are difficult to process multi-sensor data. During the resistance welding process, multiple sensors collect data such as current, force, voltage, resistance, upper displacement, and lower displacement. There are complex relationships between different sensors. Simply splicing and fusing the features of these sensors makes it difficult to capture the complex relationships between them. For example, there are mathematical relationships between current, resistance, and voltage, and there is a linear relationship between upward and downward displacements. However, existing models cannot effectively utilize these relationships for data fusion and analysis. Summary of the Invention

[0007] To solve at least one of the above-mentioned technical problems, the present invention provides a multi-sensor data fusion method for resistance welding process considering temporal characteristics.

[0008] The present invention provides a multi-sensor data fusion method for resistance welding process considering temporal characteristics, and the method includes:

[0009] Obtain multiple sensor data during the resistance welding process, and use a multi-stage multi-sensor industrial time series network to perform data processing and data fusion on the multiple sensor data, including:

[0010] Use an extended autocorrelation function to identify the trend changes of the time series data of the resistance welding process, segment the time series according to the trend change identification results, and determine the demarcation points of each stage to obtain a stage set;

[0011] Based on the stage segmentation results, adopt a bilinear self-attention mechanism to capture the correlation relationships between different stages, and process self-attention through bilinear parallel relationships to fuse each stage, obtaining the final feature embedding results of the sensors;

[0012] Based on the final feature embedding results of the sensors, perform multi-channel stage feature extraction on multiple sensors, including extracting the trend features, periodic features, abnormal fluctuation features of local stages, and the relationship between local stage features and the global time series of the time series data;

[0013] Convert the extracted multiple features into nodes of a heterogeneous graph, convert the relationships between the features into edges, and use covariance to measure the node correlation; define an edge sampling function, use the attention mechanism to update the node state, and randomly generate new nodes to expand the size of the graph, completing the fusion of multi-sensor data.

[0014] Preferably, the use of an extended autocorrelation function to identify the trend changes of the time series data of the resistance welding process, segment the time series according to the trend change identification results, and determine the demarcation points of each stage to obtain a stage set includes:

[0015] Obtain the extended autocorrelation function:

[0016]

[0017] Wherein, ACF(x) is the extended autocorrelation function, x represents the value of the time series observed at time t, represents the mean value of the time series, n represents the total number of observed values, that is, the length of the time series; k represents the number of lag periods;

[0018] The autocorrelation function is used to divide the time series data of the resistance welding process into three stages, and the boundary point P of different stages is expressed as:

[0019] P = {i|y i -y i-1 <0, Y = ACF(X), y i ∈Y}

[0020] distance(i,j)<200

[0021] Wherein, i and j are index variables, that is, the indexes of the moments or stages in the time series; Y represents the result set obtained by calculating the extended autocorrelation function ACF for the time series x, and y i represents the element in the set Y, that is, the autocorrelation value;

[0022] Each stage is expressed as:

[0023]

[0024] Wherein, S1 represents any stage cut according to ACF, [j, k] is the cutting coordinate, and S2 represents the set of intermediate stages.

[0025] Preferably, the final feature embedding result of the sensor is:

[0026]

[0027] X i ′ =norm(X i ′ )

[0028] Wherein, X ′ represents the final feature embedding result of the fused sensor, K is a hyperparameter, i is an index variable, and X i ′ is a feature vector, and W Q 、W K 、W V are the weight matrices for projecting the feature vector into the query space, key space, and value space respectively; d k is the dimension parameter related to the key, softmax is the activation function, and norm is the normalization.

[0029] Preferably, the multi-channel stage feature extraction for multiple sensors includes:

[0030] Processing each stage of the time series data using a scaling encoding representation based on the redundancy characteristics of the time series data;

[0031] Utilizing the Savitzky-Golay filter to capture local stage trends and extract abnormal fluctuation features;

[0032] Using an improved TimesNet model to extract frequency and period features;

[0033] Analyzing the correlation between local stage features and the global time series using a residual network.

[0034] Preferably, the processing of each stage of the time series data using a scaling encoding representation based on the redundancy characteristics of the time series data includes:

[0035]

[0036] where is the result after being processed by the scaling encoding representation, i and j are index variables, w scale is the scaling weight parameter, m p is the scaling window size, representing the range of time series data points considered when calculating the scaling value; x i,k represents the data point at the k-th lag period position of the i-th index stage.

[0037] Preferably, the utilization of the Savitzky-Golay filter to capture local stage trends and extract abnormal fluctuation features includes:

[0038] Capturing the local trend of the data by fitting consecutive subsets of adjacent data, and using the least squares method to determine points with a low-order polynomial:

[0039]

[0040]

[0041] where is the smoothed feature point, is the set of smoothed filter signal values; m is the window size of the filter, c i is the weight coefficient, represents the trend feature, F Trend is a convolutional model for extracting the trend feature of the smoothed signal, W t is a learnable parameter matrix, R C×lIn the formula, R is the set of real numbers, C is the number of channels, and l is the feature length; i and j are index variables;

[0042] Extract abnormal fluctuation features:

[0043]

[0044] In the formula, is the abnormal fluctuation feature, is the result set after processing the scaled coding representation at the i-index stage, F flu is the model for extracting abnormal fluctuation features, W c are learnable parameters.

[0045] Preferably, the use of the improved TimesNet model to extract frequency and period features includes:

[0046]

[0047] In the formula, represents the eigenvalue of the embedding dimension 2z at the j-index position in the i-index stage, d model is the input embedding dimension of the model, is the frequency parameter, is the frequency-domain embedding feature, TimesNet is the TimesNet model, is the position embedding at the i-index stage, W r is the learnable parameter matrix.

[0048] Preferably, the residual network is:

[0049]

[0050] In the formula, represents the local feature at the i-index stage, X global 、X scale are the global time series feature and the scaled-coded feature respectively, W i are learnable parameters, is the value of X global at the j-index position, obtained by taking the sliding window average of m and w are the time window and the sliding window respectively, s is the step size of the sliding window, u is the summation index variable, L scale 、L global are the length of the scaled-coded feature and the length of the global time series respectively.

[0051] Preferably, the use of covariance to measure node correlation includes:

[0052]

[0053] In the formula, σ(n i ,n j ) represents the node correlation metric of n i , n j in the heterogeneous graph, c ij is the covariance, both represent the values of the variables corresponding to the nodes at time t, i and j are index variables and are used as identification nodes in the heterogeneous graph; is the mean value of the variables corresponding to the nodes, and T is the length of the time series.

[0054] Preferably, the edge sampling function is:

[0055]

[0056] P(c ij ) = softmax[(1 - |c ij |)W ij )

[0057] In the formula, P(c ij ) represents probability, c ij is the covariance processed by the edge sampling function, that is, c ij takes the value of c ij ·W ij with probability P(c ij ), and takes the value of 0 with probability 1 - P(c ij ), and W ij is the weight matrix.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] The present invention provides a multi-sensor data fusion method for the resistance welding process considering temporal features, and introduces a multi-stage multi-sensor industrial time series network, that is, the MMITNet model. This model focuses on the importance of different stages in industrial time series data and the fusion of multi-sensor data. The time series embedding adopts stage-based segmentation and bilinear self-attention embedding, so as to effectively focus on the key stages of time series data. This embedding method adopts a parallel structure and simultaneously considers the relationships and changes between different stages in the time series. Subsequently, the multi-sensor fusion network uses the graph attention mechanism to fuse multi-sensor data to capture the non-linear correlations between features. Compared with previous embedding methods, MMITNet can focus on specific key stages and enhance the model performance by using stage-based self-attention embedding. MMITNet uses a special attention mechanism to transmit information and update the state between different sensors. This method can effectively obtain and integrate the relationships between different sensors, generate new implicit features, and greatly improve the accuracy of the multi-sensor data fusion result in the resistance welding process.

[0060] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to illustrate the technical solutions in the embodiments of the present invention or the background art more clearly, the drawings required for use in the embodiments of the present invention or the background art will be described below.

[0062] Figure 1 A characteristic data of a current sensor varying with time is provided;

[0063] Figure 2 A flowchart of a multi-sensor data fusion method for a resistance welding process considering timing characteristics is provided;

[0064] Figure 3 A schematic structural diagram of an MMITNet network model is provided;

[0065] Figure 4 A schematic diagram of a current divided by time sequence with an ACF function is provided;

[0066] Figure 5 A graphical schematic diagram of phased data is provided;

[0067] Figure 6 A schematic structural diagram of a multi-level multi-channel sensor series encoder is provided;

[0068] Figure 7 A schematic structural diagram of abnormal fluctuation characteristics of a stage is provided;

[0069] Figure 8 A schematic diagram of a multi-level structure of TimesNet is provided;

[0070] Figure 9 A schematic diagram of a multi-feature attention-GNN structure is provided;

[0071] Figure 10 Experimental data comparison results of various parameters of the MMITNet network and other networks are provided;

[0072] Figure 11 Experimental data comparison results of ablation experiments are provided;

[0073] Figure 12 Experimental data comparison results of baseline experiments are provided;

[0074] Figure 13 A schematic diagram of steps for calculating dispersion entropy is provided;

[0075] Figure 14The relationship between dispersion entropy and model accuracy is provided. Detailed implementation manners

[0076] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0077] In industrial production processes such as resistance welding, there are obviously different production stages, and the influence degrees of different stages on product quality are different. Refer to Figure 1 , Figure 1 The time-varying data of the current sensor is provided. As Figure 1 shown in (a), the current data can be divided into three stages, which are respectively named: the rising period, the stable period, and the falling period. In the rising period, any small data change will have a significant impact on product quality; while in the falling period, the impact of data change on product quality is negligible. However, in the current model, the importance of these stages is usually ignored. As Figure 1 shown in (b), too much attention is paid to the falling stage, while the key stage lacks attention. There are reasons why these models overly focus on unimportant stages, because insufficient embedding methods make it difficult for the model to learn the features of the key stage. In recent years, the model often uses positional encoding and time windows to embed time series data. If positional encoding is used to embed time series, it will cause the model to focus on the stage with large changes and low oscillation frequencies, and be insensitive to some abnormal fluctuations. If time window encoding is used, serious embedding redundancy will occur.

[0078] To solve the problem of multi-stage time series encoding, the present invention aims to provide a multi-sensor data fusion method for the resistance welding process considering temporal characteristics, introducing the MMITNet model. This model proposes an embedding method that segments time series based on trend changes and combines a bilinear self-attention mechanism to obtain the importance of different stages. The stage segmentation module can effectively identify the trend change patterns of time series data. The bilinear self-attention mechanism captures the relationships between different stages and integrates these relationships to generate sensor features. In addition, there are complex relationships between different sensors in industrial production. There are mathematical relationships between current, resistance, and voltage, and there is a linear relationship between upward and downward displacements. Since simply splicing and fusing the features of these sensors makes it difficult to capture the complex relationships between them, MMIT converts its sensor features into nodes of a heterogeneous graph and converts the relationships between various sensor features into edges. For such a heterogeneous graph, MMITNet designs an edge sampling function and an attention GNN mechanism, which can capture the deep relationships between sensors by supervising the propagation of sensor features. This is of great significance for achieving time series data classification across multiple sensors and also provides support for constructing future interpretive models.

[0079] Please refer to Figure 2 , Figure 2 which is a schematic flow chart of a multi-sensor data fusion method for the resistance welding process considering temporal characteristics provided by an embodiment of the present invention. As Figure 2 shown, the method includes:

[0080] S10. Obtain multiple sensor data during the resistance welding process, and perform data processing and data fusion on the multiple sensor data using a multi-stage multi-sensor industrial time series network, including:

[0081] S101. Use the extended autocorrelation function to identify the trend changes of the time series data of the resistance welding process, segment the time series according to the trend change identification results, and determine the demarcation points of each stage to obtain a stage set;

[0082] S102. Based on the stage segmentation results, adopt a bilinear self-attention mechanism to capture the correlation relationships between different stages, and fuse each stage through bilinear parallel relationship processing of self-attention to obtain the final feature embedding result of the sensor;

[0083] S103. Based on the final feature embedding result of the sensor, perform multi-channel stage feature extraction on multiple sensors, including extracting the trend features, periodic features, abnormal fluctuation features of the local stage, and the relationship between the local stage features and the global time series;

[0084] S104. Convert the multiple extracted features into nodes of a heterogeneous graph, convert the relationships between the features into edges, and use covariance to measure node correlation; define an edge sampling function, update the node states using an attention mechanism, and randomly generate new nodes to expand the size of the graph, thus completing the fusion of multi-sensor data.

[0085] In this embodiment, a multi-stage multi-sensor industrial time series network (MMITNet) is mainly introduced. As Figure 3 shown, first, considering the stage characteristics of industrial time series data, MMITNet designs an embedding method based on stage segmentation and bilinear self-attention mechanism. In addition, in view of the redundancy, feature relationships, trends, and periodicity in industrial time series, MMITNet uses multi-channel feature extraction in each stage. Finally, due to the complex relationships between sensors, MMITNet designs an attention GNN to capture the deep relationships between different sensors. Through appropriate training and parameter adjustment, MMITNet can flexibly adapt to various types of time series data.

[0086] As Figure 1 shown, industrial time series have multiple stages, and the importance of different stages varies. First, MMITNet identifies different stages of the time series according to trend changes. To calculate the trend at any moment and divide the time series into different stages based on this trend, MMITNet extends the autocorrelation function (ACF). The autocorrelation function measures the degree of correlation between two events at different time intervals, can identify the missing fundamental frequencies in time series signals, and reveal the repetitive patterns in the data.

[0087] In one embodiment, the trend changes of the time series data of the resistance welding process are identified using the extended autocorrelation function, and the time series is segmented into stages according to the trend change identification result, and the demarcation points of each stage are determined to obtain a stage set, including:

[0088] Obtain the extended autocorrelation function:

[0089]

[0090] where ACF(x) is the extended autocorrelation function, x represents the value of the time series observed at time t, represents the mean of the time series, n represents the total number of observations, that is, the length of the time series; k represents the number of lag periods;

[0091] The time series data of the resistance welding process is divided into three stages using the autocorrelation function, and the demarcation points P of different stages are expressed as:

[0092] P = {i|y i -y i-1<0, Y = ACF(X), y i ∈Y} (2)

[0093] distance(i, j) < 200

[0094] where i and j are index variables, i.e., the indices of moments or stages in the time series; Y represents the set of results obtained by calculating the extended autocorrelation function ACF for the time series x, and y i represents an element in the set Y, i.e., the autocorrelation value;

[0095] Each stage is represented as:

[0096]

[0097] where S1 represents any stage cut according to ACF, [j, k] are the cutting coordinates, and S2 represents the set of intermediate stages.

[0098] See Figure 4 , Figure 4 provides a schematic diagram of the current time series with the ACF function. In the above embodiment, in the rising stage of the time series, the calculated ACF value is greater than 0 at each moment, while in the stable stage of the time series, the ACF value is less than 0, as shown in formula (1). In Figure 4 , the current data is automatically divided into three stages by the autocorrelation function ACF. Therefore, according to the calculated change trend, the boundary point P of different stages can be expressed as formula (2).

[0099] The multi-level bilinear self-attention network is used for multi-sensor industrial time series classification. Therefore, each stage in the time series can be mathematically represented as formula (3). Among them, S1 represents a set of stages cut according to ACFtrend, and the point set p can divide the time series into the stage set S1, but the change of the internal stage information of S1 is ignored. To focus on the change of the internal stage of S1, the model designs the intermediate stage set S2, which is expressed as formula (4).

[0100] To facilitate visually seeing each data stage, in one embodiment, a schematic diagram of the graphical relationship between S1 and S2 is provided, as Figure 5 shown, where each node represents a stage and is connected by a unidirectional edge. The upper and lower lists represent S1 and S2 respectively. For a node in S1 to the next node in S1, it needs to pass through a node (intermediate node) in S2, and vice versa. By reflecting this relationship in the list, the segmented sequence can be represented as X = [X 1 , X 2 , … X K .

[0101] In one embodiment, in step S102, based on the stage segmentation results, a bilinear self-attention mechanism is adopted to capture the correlation between different stages, and the self-attention is used to fuse each stage through a bilinear parallel relationship to obtain the final feature embedding result of the sensor. This model processes the self-attention fusion based on this bilinear parallel relationship. From the first common stage to the first intermediate stage, it is the query space, which is projected into the previous space through W Q From the first intermediate stage to the second common stage, it is the key space, which is projected into the space of the latter part through W K The model performs a cross product to find the similarity score from the previous space to the latter space. In this scoring, the model concatenates three paragraphs and multiplies them to obtain the final adjacent embedding. Therefore, the final feature embedding result of the sensor can be expressed as:

[0102]

[0103] X i ′ = norm(X i ′ )

[0104] wherein, X ′ represents the final feature embedding result of the sensor after fusion, K is a hyperparameter, i is an index variable, X i ′ is a feature vector, and W Q , W K , W V are weight matrices for projecting the feature vector into the query space, the key space, and the value space respectively; d k is a dimension parameter related to the key, softmax is an activation function, and norm is normalization.

[0105] In one embodiment, the multi-channel stage feature extraction for multiple sensors includes:

[0106] Processing each stage of the time series data using a scaling coding representation method based on the redundancy characteristics of the time series data;

[0107] Utilizing the Savitzky-Golay filter to capture the local stage trend and extract abnormal fluctuation features;

[0108] Using an improved TimesNet model to extract frequency and period features;

[0109] Utilizing the residual network to analyze the correlation between the local stage features and the global time series.

[0110] Time series embedding using only phase-based segmentation and bilinear self-attention fusion is insufficient. We need to extract their features. MMITNet encodes all phases with trend features, periodic features, abnormal fluctuation features, and their relationships with the global sequence, as Figure 6 shown. Additionally, the model needs to map sequence phases to maintain the consistency of input dimensions, especially when dealing with features in time series data. Obviously, simply scaling these phases will distort their proportional relationships, leading to inconsistent subsequent feature extraction results. To address this issue, MMIT Net has made some improvements. The model uses a scaling encoding representation based on the redundancy characteristics of time series data to process each phase of the time series data, specifically:

[0111] Using a scaling encoding representation based on the redundancy characteristics of time series data to process each phase of the time series data, including:

[0112]

[0113] In the formula, is the result after being processed by the scaling encoding representation, i and j are index variables, w scale is the scaling weight parameter, m p is the scaling window size, indicating the range of time series data points considered when calculating the scaling value; x i,k represents the data point at the k-th lag period position of the i-th index phase.

[0114] In one embodiment, using the Savitzky-Golay filter to capture local phase trends and extract abnormal fluctuation features includes:

[0115] Capturing the local trend of the data by fitting consecutive subsets of adjacent data, and using the least squares method to determine points with a low-order polynomial:

[0116]

[0117] In the formula, is the smoothed feature point, is the set of smoothed filter signal values; m is the window size of the filter, c i is the weight coefficient, represents the trend feature, F Trend is a convolutional model for extracting the trend feature of the smoothed signal, W t is a learnable parameter matrix, R C×l where R is the set of real numbers, C is the number of channels, l is the feature length; i and j are index variables;

[0118] Extracting abnormal fluctuation features:

[0119]

[0120] wherein, is the abnormal fluctuation feature, is the result set after processing the scaled coding representation at the i-index stage, F flu is the model for extracting the abnormal fluctuation feature, W c is the learnable parameter.

[0121] To capture the trend of each local stage, the model uses the Savitzky-Golay filter which not only smooths the time series but also captures the local trend of the data by fitting consecutive subsets of adjacent data, determines points with a low-order polynomial using the least squares method, helps store fluctuations, and at the same time ensures that the shape and width of the signal remain unchanged. The specific expression is shown in formula (7). Usually, the stage trend for the series can be expressed as formula (8), where F Trend can be replaced by any convolution-based model. For the simplicity of experiments, the Resnext34 model is preferably used.

[0122] It should be noted that data abnormal fluctuations will affect the quality of industrial production. Abnormal fluctuations may come from sensor acquisition errors, environmental changes, or other uncertainties during the data acquisition process and are manifested as data fluctuations in the sensor. By extracting the abnormal fluctuation features, the impact of abnormal fluctuations on data quality can be identified and reduced, thereby improving the accuracy and availability of the data. At the same time, the extraction of abnormal fluctuation features helps to enhance the robustness of the model, enabling it to better handle abnormal fluctuations and inconsistencies. The abnormal fluctuation features of the original data can also be captured similarly, as Figure 7 shown, and the mathematical representation is formula (9).

[0123] In one embodiment, the use of the improved TimesNet model to extract frequency and period features includes:

[0124]

[0125] wherein, represents the eigenvalue of the embedding dimension 2z at the j-index position at the i-index stage, d model is the input embedding dimension of the model, is the frequency parameter, is the frequency domain embedding feature, TimesNet is the TimesNet model, is the position embedding at the i-index stage, W r is the learnable parameter matrix.

[0126] MMITNet can not only capture the phase fluctuation characteristics of time series from sensor sequences, but also extract their frequency and period characteristics. It has obtained a lot of inspiration from the work of TimesNet, which is an advanced period feature extraction model. The core idea of TimesNet is to use the attention mechanism and temporal convolution to capture the temporal relationships in time series data. Therefore, by combining TimesNet and the phase feature segmentation structure discussed above, MMITNet can more accurately distinguish the period characteristics of different time phases. This improves the efficiency of the model in capturing the periodic information of time series data. Figure 8 A schematic diagram of the multi-level structure of TimesNet is provided, as Figure 8 shown, MMITNet extracts frequency features for different time periods. MMITNet makes use of the excellent performance of TimesNet in feature extraction. It inputs the segmented periodic data into the TimesBlock. On this basis, TimesNet further segments the phases according to the frequency features and converts them into two-dimensional images for convolution. Finally, it reconstructs the time series of the entire phase. For the time-domain data of each phase, MMITNet performs concatenation pooling and then inputs it into the relevant decoder. Equation (10) is the same as that of the transformer-based model, and the j index position ranges from 0 to the phase length minus 1. dmodel is the input embedding dimension of the model. The principle of this formula is to use the periodicity of sine and cosine functions to represent the relative position information between different positions. By adjusting the frequency parameter the periods of different dimensions can be controlled. Then MMITNet uses the SOTA model to obtain the frequency-domain embedding of the phase from a series to get the frequency-domain embedding result of Equation (11).

[0127] To further explore the correlation between local phase features and the global time series, the model introduces a residual network, and its mathematical representation is as follows:

[0128]

[0129] In the formula, represents the local feature under the i-indexed phase, X global , X scale are the global time series feature and the scaled and encoded feature respectively, W i is the learnable parameter, is the value of X global at the j-indexed position, obtained by taking the sliding window average of , where m and w are the time window and the sliding window respectively, s is the step size of the sliding window, u is the summation index variable, and L scale , L globalThey are the length of the scaled coding feature and the length of the global time series respectively.

[0130] Furthermore, the mathematical representation of the series data of the feature is:

[0131]

[0132] In the formula, X i ′ is the feature representation of the sequence data, which is obtained by multiplying the frequency domain feature trend feature abnormal fluctuation feature and local feature after splicing by the learnable parameter W, is the feature vector obtained by intermediate calculation, and the final feature representation is obtained after the normalization norm operation.

[0133] It can be understood that the acquisition of industrial production time series data requires multiple sensors, and there are complex relationships between different sensors. MMITNet needs to obtain input features from different sensor series and channels, capture the feature relationships between different sensors, and obtain in-depth understanding and analysis of the data by fusing the features of different sensors. The relationship features between sensors contain rich information, and the importance of the relationships between different sensors varies. Therefore, MMITNet has developed an nAttention-GNN method to fuse the information from the features of different sensors using a graph neural network, transforming the relationships between the original features into the form of graph neural network representation learning. Among them, the multi-feature attention-GNN structure is as Figure 9 shown.

[0134] Assume that each node feature X node = N = n 1 、n 2 ,,,n k is the time series data of the stage. Then the edge feature X edge between nodes represents the relationship between the features of different sensors, denoted as C. Therefore, for each input, the model can construct a complete graph G = N, C in real time. For the relationships between the nodes in the graph, MMITNet uses covariance as a measure of the correlation between variables. When the absolute value of the covariance is large, it indicates that there is a strong association between variables; while when the absolute value of the covariance is small, it indicates that there is a weak correlation between variables. Therefore, the relationship between n i and n j nodes is expressed as follows:

[0135]

[0136] In the formula, σ(n i ,n j) represents the node correlation metric for n i and n j in the heterogeneous graph, where c ij is the covariance, and are the values of the variables corresponding to the nodes at time t, i and j are index variables and serve as identifying nodes in the heterogeneous graph;

[0137] Based on the covariance, MMITNet represents different node relationships as edges on the graph, so each set of data can be represented as a graph. For graph classification tasks, a complete graph is usually not the optimal solution. This is because, if a fully connected graph has too many connections, it may lead to overfitting of the model and ignore the local structural information between nodes, which may cause the model to learn noise rather than the true data patterns.

[0138] To solve this problem, MMITNet designs an edge sampling function. Different from the traditional methods used in GNN models, in this embodiment, it is considered that in industrial production, the feature similarity captured between different sensor series is relatively high. Over-transmitting these redundant features during the information propagation process makes it difficult for nodes to update to new states and extract hidden information. Secondly, the number of sensors in the industrial dataset is limited, resulting in a limited number of nodes in the graph. The initial structure of the graph has a relatively small impact on the model. Therefore, MMITNet focuses on relationships with low correlation.

[0139] Preferably, the edge sampling function is:

[0140]

[0141] P(c ij ) = softmax[(1 - |c ij |)W ij )

[0142] In the formula, P(c ij ) represents the probability, c ij is the covariance after being processed by the edge sampling function, that is, c ij takes the value of c ij ·W ij with probability P(c ij ), and takes the value of 0 with probability 1 - P(c ij ), where W ij is the weight matrix.

[0143] Further, MMITNet uses an attention mechanism to update the node state, as Figure 9 shown. The formula for updating the node state is given here:

[0144]

[0145] c ix = c ij , c jx = c ji , c ij = c xi = c xj = 0(18)

[0146] Wherein, n i i is the updated node state, [n i |n j represents concatenating n i , n j nodes, W Q , W K , W V are the weight matrices for projecting the feature vectors into the query space, key space, and value space respectively; d k is the dimension parameter related to the key, softmax is the activation function, c ix , c jx , c xi , c xj are all relationship metric values between different nodes. To avoid circular embedding, their values are set to 0, that is, the relationship connections between the original nodes are eliminated.

[0147] For ease of understanding, in one embodiment, specific experimental data is given, including readings from six sensors: current, force, voltage resistance, upper displacement, and lower displacement. The data length of each sensor is approximately 3650 ms, but for experimental convenience, it is uniformly truncated to 3600 ms. In the experiment, the ratio of the training set to the test set in the dataset is set to 6:4. 4 Tesla K80 GPUs are used for training, and hyperparameters such as the learning rate are automatically adjusted for different models. The training process is repeated 50 times, and the best-performing result is selected as the classification result of the model. Due to the high sampling density of the actual production data and the significant differences between the data of different sensors, there are variable data in different time periods. This conflicts with the design principles of some models, making it difficult for them to learn important classification features from the data. Therefore, these models are difficult to obtain satisfactory results. In many cases, after 50 times of training, the classification results of these models are either all 1 or all 0, indicating no discrimination ability. Therefore, this part of the experimental data needs to be excluded.

[0148] Since the industrial production environment pays more attention to identifying negative cases, we use specificity and F1 score as indicators to measure the accuracy of negative class recognition. See Figure 10 , Figure 10 provides the experimental comparison results of various parameters of the MMITNet network and other networks. According toFigure 10 It can be seen that MMITNet performs excellently on this real-world dataset, with all metrics exceeding 0.8, demonstrating its superiority, especially in identifying negative-class data, which is better than other models. Although the autoencoder and information encoder show advantages in some metrics, they may perform poorly in other aspects, so their performance needs to be considered comprehensively. Both DLinear and TimesNet belong to pattern recognition models, which focus more on learning the specific feature linear trend of this series to learn the trend of this series, while TimesNet is more inclined to learn the overall-local periodicity. Therefore, they have certain advantages in processing long and challenging data.

[0149] To verify the effectiveness of each hierarchical module in the model, a series of ablation experiments were conducted, such as Figure 11 shown, which shows the effectiveness of our various model structures in the design of industrial spot welding models. Obviously, without the ACF segmentation structure, the accuracy of MMITNet will decrease. This indicates that the segmented design method of the model is correct and effective. In industry, not every production stage is equally important. The model needs to pay more attention to certain stages. When the TimesNet structure is removed, the model test accuracy drops significantly to 0.706, which indicates that frequency domain feature extraction is crucial. At the same time, when the smoothing structure and abnormal fluctuation structure are removed, the accuracy of the results also decreases. This indicates that the feature extraction based on the smoothing and abnormal fluctuation structures is effective. The influence of the trends and noises in each stage on the experimental results is obvious, indicating that trends and abnormal fluctuations have a certain impact on the quality of industrial spot welding.

[0150] In one embodiment, the performance of MMITNet on the baseline dataset was studied and further compared with other models. In the experiment, all baseline datasets were divided into training and test parts in a ratio of 6:4. Each model was trained 50 times. The average of the best three accuracies and the weighted F1 of the best accuracy were taken as the representative results. The experimental results show that MMITNet performs well in most cases, such as Figure 12As shown. It performs best on datasets with long sequences, high volatility, high periodicity, and high phase, such as the self-regulating SCP dataset, with an accuracy of 0.672 on the test set; in short-term sequence and volatility datasets, such as the face detection dataset, the MMITNet model shows better performance, with an accuracy of 0.714; in the case of short-term sequences and small sequence fluctuations, such as the handwriting dataset, its test set accuracy reaches 0.593. Although it is not as good as the classical methods Rocket and LSTM, it is still slightly better than other methods because most of these methods are applicable to volatile sequences; in the case of extremely large sequence fluctuations and very few datasets, such as the atrial fibrillation dataset, the accuracy of the model is close to other methods; in smooth sequences, such as the ethanol concentration dataset, the classical methods XGBoost and Rocket respectively achieve SOTA.

[0151] To measure the impact of data fluctuations on the model classification results, discrete entropy is used to measure the complexity of different time series. For continuous sequences, the greater the degree of fluctuation, the greater the difference between a certain moment and the adjacent moment. Therefore, the higher the discrete entropy, the more obvious the fluctuation of the time series. Algorithms such as Figure 13 As shown, so the discreteness of each dataset can be calculated, as Figure 14 As shown. Obviously, the lower the complexity of the time series, the better results the classical methods tend to achieve, such as the ethanol concentration dataset. The data in this dataset is smoother. The fluctuations are not obvious either, so XGBoost achieves the optimal result. This situation also applies to the PEMS-SF dataset and the handwriting dataset. However, as the complexity of the time series increases, classical methods often have difficulty modeling complex fluctuating data. Therefore, deep learning methods have better classification effects when dealing with these time series; for the Kepler light curve dataset, the data fluctuations are extremely complex and the time series length is extremely long, which makes it closer to SOTA in classification.

[0152] In summary, applying the MMITNet model to the classification task of industrial production data, this model integrates features from multiple sensors and achieves satisfactory classification results. Through the MMITNet model, complex industrial production data can be effectively classified, providing strong support for process monitoring and optimization.

[0153] The method provided by the present invention introduces a multi-stage multi-sensor industrial time series network MMITNet, which focuses on the importance of different stages in industrial time series data and the fusion of multi-sensor data. The time series embedding adopts stage-based segmentation and bilinear self-attention embedding, thus effectively focusing on the key stages of the time series data. This embedding method uses a parallel structure and simultaneously considers the relationships and changes between different stages in the time series. Subsequently, the multi-sensor fusion network uses the graph attention mechanism to fuse multi-sensor data to capture the non-linear correlations between features. Compared with previous embedding methods, MMITNet can focus on specific key stages and enhance the model performance by using stage-based self-attention embedding. MMITNet uses a special attention mechanism to transmit information and update the state between different sensors. This method can effectively obtain and integrate the relationships between different sensors, generate new implicit features, and greatly improve the accuracy of the multi-sensor data fusion results in the resistance welding process.

[0154] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

Claims

1. A multi-sensor data fusion method for resistance welding process considering timing characteristics, characterized in that: The method comprises: Acquire multiple sensor data during the resistance welding process, and use a multi-stage multi-sensor industrial time series network to process and fuse multiple sensor data, including: The extended autocorrelation function is used to identify the trend change of the time series data of the resistance welding process, and the time series is segmented into stages according to the trend change identification results, and the dividing points of each stage are determined to obtain the stage set; Based on the stage segmentation results, a bilinear self-attention mechanism is used to capture the correlation between different stages. The self-attention fusion stages are processed through bilinear parallel relations to obtain the final feature embedding result of the sensor. Based on the final feature embedding results of the sensors, multi-channel stage feature extraction is performed on multiple sensors, including extracting trend features, periodic features, abnormal fluctuation features of local stages, and the relationship between local stage features and global time series data; The extracted multiple features are converted into nodes of a heterogeneous graph, the relationships between the features are converted into edges, and the covariance is used to measure the node correlation; the edge sampling function is defined, the node status is updated using the attention mechanism, the size of the new node expansion graph is randomly generated, and the fusion of multi-sensor data is completed.

2. The multi-sensor data fusion method for resistance welding process considering timing characteristics according to claim 1 is characterized in that: The method of using the extended autocorrelation function to identify the trend change of the time series data of the resistance welding process, segmenting the time series into stages according to the trend change identification result, and determining the dividing points of each stage to obtain a stage set includes: Get the expanded autocorrelation function: In the formula, ACF(x) is the extended autocorrelation function, x represents the value of the time series observed at time t, represents the mean of the time series, n represents the total number of observations, i.e. the length of the time series; k represents the number of lags; The time series data of the resistance welding process is divided into three stages using the autocorrelation function, and the dividing point P of different stages is expressed as: P={i|y i -and i-1 <0,Y=ACF(X),y i ∈Y} distance(i,j)<200 In the formula, i and j are index variables, i.e., the index of the moment or stage in the time series; Y represents the result set obtained by calculating the extended autocorrelation function ACF of the time series x, and y i represents the elements in the set Y, i.e., the autocorrelation value; The various stages are represented as: j=0,when k=n,when k-j>200 Where S1 represents any stage of ACF cutting, [j, k] is the cutting coordinate, and S2 represents the set of intermediate stages.

3. The multi-sensor data fusion method for resistance welding process considering timing characteristics according to claim 1, characterized in that: The final feature embedding result of the sensor is: X′ i =norm(X′ i ) In the formula, X′ represents the final feature embedding result of the fused sensor, K is a hyperparameter, i is an index variable, and X′ i is the feature vector, W Q , W K , W V are the weight matrices that project the feature vector into the query space, key space, and value space respectively; d k is the dimension parameter associated with the key, softmax is the activation function, and norm is the normalization.

4. The multi-sensor data fusion method for resistance welding process considering timing characteristics according to claim 1, characterized in that: The multi-channel stage feature extraction of the multiple sensors includes: Use the scaling coding representation method based on the redundant characteristics of time series data to process the various stages of time series data; Savitzky-Golay filter is used to capture local stage trends and extract abnormal fluctuation characteristics; Use the improved TimesNet model to extract frequency and period features; The residual network is used to analyze the correlation between local stage characteristics and global time series.

5. The multi-sensor data fusion method for resistance welding process considering timing characteristics according to claim 4, characterized in that: The various stages of processing time series data using a scaling coding representation method based on the redundant characteristics of time series data include: In the formula, is the result after scaling coding representation, i,j are index variables, w scale is the scaling weight parameter, m p is the scaling window size, which indicates the range of time series data points considered when calculating the scaling value; x i,k represents the data point at the kth lag number position in the ith index stage.

6. The multi-sensor data fusion method for resistance welding process considering timing characteristics according to claim 5, characterized in that: The Savitzky-Golay filter is used to capture the local stage trend and extract the abnormal fluctuation characteristics, including: Capture local trends in the data by fitting contiguous subsets of adjacent data, using the least squares method to determine the points with a low-order polynomial: In the formula, Is smooth The feature points after is the set of smoothed filtered signal values; m is the filter window size, c i is the weight coefficient, Indicates trend characteristics, F Trend is a convolution model, which is used to extract the trend characteristics of the smoothed signal. t is the learnable parameter matrix, R C×l Where R is a real number set, C is the number of channels, l is the feature length; i, j are index variables; Extract abnormal fluctuation features: In the formula, It is an abnormal fluctuation characteristic. is the result set after the scaling coding representation is processed at the i-index stage, F flu To extract the abnormal fluctuation characteristics of the model, W c are learnable parameters.

7. The multi-sensor data fusion method for resistance welding process considering timing characteristics according to claim 4, characterized in that: The improved TimesNet model is used to extract frequency and period features, including: In the formula, represents the eigenvalue of embedding dimension 2z at index position j at index stage i, d model is the input embedding dimension of the model, is the frequency parameter, is the frequency domain embedding feature, TimesNet is the TimesNet model, is the position embedding at the i-index stage, W r is the learnable parameter matrix.

8. The multi-sensor data fusion method for resistance welding process considering timing characteristics according to claim 4, characterized in that: The residual network is: In the formula, represents the local features at the i index stage, X global , X scale are the global time series features and the scaled encoded features, W i is a learnable parameter, For X global The value at index position j is obtained by The sliding window is averaged, where m and w are the time window and sliding window respectively, s is the step size of the sliding window, u is the summation index variable, and L scale , L global They are the length of the scaled encoding feature and the length of the global time series, respectively.

9. The multi-sensor data fusion method for resistance welding process considering timing characteristics according to claim 1, characterized in that: The method of using covariance to measure node correlation includes: In the formula, σ(n i ,n j ) represents the heterogeneous graph n i 、n j Node correlation measure, c ij is the covariance, are the values ​​of the variables corresponding to the nodes at time t, i and j are index variables, which serve as identification nodes in the heterogeneous graph; is the mean of the variable corresponding to the node, and T is the length of the time series.

10. The multi-sensor data fusion method for resistance welding process considering timing characteristics according to claim 9, characterized in that: The edge sampling function is: P(c ij )=softmax[(1-|c ij |)W ij ] In the formula, P(c ij ) represents the probability, c ij is the covariance after edge sampling function processing, that is, c ij With probability P(c ij ) takes the value of c ij ·W ij , with probability 1-P(c ij ) takes the value of 0, W ij is the weight matrix.

Images