A time series transformer-based industrial profile sequence reconstruction and filtering method and system

Abstract

This invention discloses a method and system for industrial contour sequence reconstruction and filtering based on temporal Transformer, belonging to the fields of industrial intelligent detection and deep learning. The method first downsamples and divides the industrial contour data into blocks, then constructs a data validity mask and uses positional encoding to generate sequence position information. Furthermore, an improved Transformer encoder and decoder structure is used to achieve global feature extraction and sequence reconstruction of the industrial contour sequence. Simultaneously, a layer normalization method based on the data mask is proposed to avoid the influence of invalid values ​​on statistical calculations. This method can effectively improve the accuracy and model stability of industrial contour sequence reconstruction and is suitable for industrial scenarios such as automatic wire laying, laser contour detection, and complex surface detection.

Description

Technical Field

[0001] This invention belongs to the fields of industrial automation, time-series data processing, and deep learning technology, specifically relating to a method for industrial contour sequence reconstruction and filtering based on a time-series Transformer. It is applicable to scenarios such as automated fiber placement (AFP), laser contour measurement, surface defect detection, and continuous time-series signal reconstruction. Background Technology

[0002] In industrial applications such as automated fiber placement (AFP), laser profile scanning, and complex surface inspection, line laser profile sensors (LPS) continuously output large amounts of high-frequency temporal profile data. Due to the complex industrial environment, variations in workpiece surface reflectivity, occlusion, and sensor noise, the acquired profile sequences typically suffer from the following problems: 1. Local data loss; 2. Numerous invalid points; 3. Severe sequence noise; 4. Difficulty in modeling long-term temporal dependencies.

[0003] Traditional convolutional methods struggle to simultaneously preserve local details and global relationships. While traditional recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) can handle temporal problems, their sequential structure leads to low computational efficiency for long sequences and suffers from long-term dependency decay. Although Transformer networks possess strong global modeling capabilities, traditional Transformers are primarily designed for natural language processing tasks, and their standard structure is not suitable for continuous industrial contour sequences. Particularly in industrial contour data, a large number of invalid values ​​severely impact the stability of attention computation.

[0004] Therefore, a novel Transformer architecture suitable for industrial contour sequences is needed to achieve high-precision and robust temporal reconstruction and filtering. Since its introduction by Vaswani et al. in the 2017 paper "Attention is All You Need," the Transformer model has become the mainstream architecture in Natural Language Processing (NLP), Computer Vision, and multimodal tasks due to its parallel computing capabilities based entirely on the attention mechanism and its excellent long-distance dependency modeling performance. However, existing Transformer models are mainly designed for discrete symbol sequences (such as text words), and have many technical shortcomings when processing high-dimensional continuous numerical vector time series, making them difficult to directly apply to practical engineering data. Specifically, existing technologies mainly suffer from the following deficiencies:

[0005] 1. Insufficient handling of invalid sentinel values: In fields such as industrial measurement, scientific experiments, and physical simulation, collected or generated data is often stored in binary file format. Due to sensor failures, measurement interruptions, data truncation, etc., a large number of invalid sentinel values ​​(such as extreme values ​​like -1000000000) are commonly present in the data. Traditional Transformer layernorms, attention calculations, and loss functions all assume that all input data is valid. When encountering such invalid values, it can easily lead to statistical bias, gradient explosion, or unstable model training. Although existing technologies include simple masking, these are mostly padding masks used in NLP, which are difficult to adapt to the complex situation where some feature dimensions are valid while others are invalid in high-dimensional vectors.

[0006] 2. Inefficient processing of high-dimensional, long sequences: Time series data in engineering often have extremely high feature dimensions (hundreds to thousands of dimensions per time step) and extremely long sequence lengths (thousands to tens of thousands of time steps). Directly inputting these sequences into a standard Transformer leads to memory overload and computational inefficiency. Existing chunking or slicing methods are mostly simple divisions that fail to effectively maintain the temporal continuity and feature integrity within the sequence, and they also lack dedicated chunking mechanisms for vector data, making it difficult to train models on large datasets under real-world hardware conditions.

[0007] 3. Lack of temporal decay control in attention mechanisms: Time series data exhibits significant temporal characteristics, with historical information from further back in time generally having lower importance. Standard self-attention mechanisms treat all positions in the sequence equally, easily introducing excessive noise and affecting the model's focus on recent key features. While existing improvements (such as relative position encoding and local attention) have shown some effectiveness, most are designed for discrete tokens and are difficult to apply directly to continuous high-dimensional vector scenarios. Furthermore, they fail to provide a flexibly configurable decay intensity control mechanism.

[0008] 4. Embedding and normalization methods are not suitable for continuous vectors: Traditional Transformers rely on word embedding layers to map discrete tokens to low-dimensional dense vectors. However, high-dimensional vector sequences already carry rich feature information, and forcibly using embedding may result in the loss of original information. At the same time, the standard LayerNorm introduces significant errors when there are a large number of invalid values, making it difficult for the model to converge.

[0009] 5. Mismatch between loss function and training strategy: Existing sequence models often employ cross-entropy loss and other designs tailored for classification tasks, while vector sequence reconstruction is essentially a continuous value regression problem. Directly applying these methods leads to insensitivity to key engineering features such as peaks and abrupt changes, and fails to effectively shield against the interference of invalid values ​​in loss calculation. Currently, although there are some Transformer variants for time series (such as Informer, Autoformer, FEDformer, etc.), these works mainly focus on low- or medium-dimensional time series prediction and rarely consider the large number of invalid values ​​and high-dimensional vector characteristics present in real-world engineering data. A systematic and complete technical solution for high-dimensional vector sequences + invalid value masking + attention decay + efficient memory loading has yet to emerge.

[0010] Therefore, in order to address the shortcomings of the existing technologies, developing a Transformer system specifically designed for high-dimensional engineering vector time series, possessing robustness against invalid data, supporting efficient processing of long sequences, and introducing a temporal attention decay mechanism, has significant theoretical implications and broad engineering application value. Summary of the Invention

[0011] The technical problem solved by this invention is to provide a method and system for industrial contour sequence reconstruction and filtering based on temporal Transformer. By constructing a data encoding method with a data masking mechanism, an improved layer normalization structure, and an encoder-decoder structure suitable for industrial continuous sequences, high-precision reconstruction and filtering of industrial contour time series data can be achieved.

[0012] Technical Solution: To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0013] A temporal Transformer-based method for industrial contour sequence reconstruction and filtering is proposed. This method first downsamples and divides the industrial contour data into blocks, then constructs a data validity mask and uses positional encoding to generate sequence position information. Furthermore, an improved Transformer encoder and decoder structure is used to achieve global feature extraction and sequence reconstruction of the industrial contour sequence. A layer normalization method based on the data mask is also proposed to avoid the influence of invalid values ​​on statistical calculations. This method effectively improves the accuracy and model stability of industrial contour sequence reconstruction and is applicable to industrial scenarios such as automated wire laying, laser contour detection, and complex surface detection.

[0014] Specifically, the steps include: (1) loading high-dimensional binary sequence data, performing downsampling and block processing, and generating a validity mask; (2) inputting the block-wise sequence into a Transformer model containing mask-aware embedding and layer normalization, and combining it with an attention correction mechanism for feature encoding and decoding; (3) training the model using a mask loss function that calculates the error only for valid positions; and (4) reconstructing the input sequence in one go using a non-autoregressive method to obtain a high-dimensional vector sequence consistent with the input dimension. When calculating the multi-head attention score, a linear temporal distance bias is introduced to achieve attention decay.

[0015] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0016] 1. The method of the present invention aims to efficiently process high-dimensional, long-sequence structured vector data containing invalid label values.

[0017] 2. This invention solves the technical problems faced by traditional Transformer models when processing the above-mentioned high-dimensional engineering vector sequences, such as invalid value interference, memory explosion, and difficulty in controlling temporal dependency decay, by introducing a mask-aware mechanism, custom layer normalization, attention decay correction, and memory-efficient data loading method. It belongs to a sequence-to-sequence vector reconstruction and prediction technology designed specifically for actual engineering data.

[0018] 3. The layer normalization method and attention calculation method based on data masking proposed in this invention can effectively suppress the impact of invalid values ​​on model training. Compared with traditional convolutional networks, it has stronger global modeling capabilities; this model is suitable for contour reconstruction and filtering of sensors such as laser sensors in complex industrial environments.

[0019] 4. This invention is applicable to, but not limited to, the following application scenarios: two-dimensional / three-dimensional profile reconstruction in industrial inspection (such as laser scanning, ultrasonic testing, X-ray imaging, etc.); it also includes sequence analysis of medical images or biological signals (such as EEG, ECG). Attached Figure Description

[0020] Figure 1 This is the overall network structure diagram of the present invention;

[0021] Figure 2 This is a structural diagram of the encoder of the present invention;

[0022] Figure 3 This is a structural diagram of the decoder of the present invention;

[0023] Figure 4 This is a normalized structure diagram of the mask layer of the present invention;

[0024] Figure 5 This is the reconstructed industrial outline sequence in this embodiment;

[0025] Figure 6 This is the binarized image data resulting from the processing in this embodiment. Detailed Implementation

[0027] The present invention will be further illustrated below with reference to specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0028] Example 1

[0029] This embodiment provides a method for industrial contour sequence reconstruction and filtering based on temporal Transformer, including the following steps:

[0030] S1: Industrial profile data acquisition;

[0031] S2: Sequence downsampling and block division;

[0032] S3: Construct a data validity mask;

[0033] S4: Location code;

[0034] S5: Encoder feature extraction;

[0035] S6: Mask layer normalization;

[0036] S7: Decoder reconstruction;

[0037] S8: Loss function construction.

[0038] In this embodiment, the above steps are implemented as follows:

[0039] S1: Acquire industrial contour data and extract height values.

[0040] The input data processed by this invention is typically stored in binary file format and contains a large number of continuous numerical sequences (such as profile data collected by scientific instruments, engineering measurement data, physical simulation results, multidimensional sensor time series, etc.). These data often have the following typical characteristics:

[0041] 1. High dimensionality: The feature dimension of each time step can reach hundreds to thousands (far higher than the word embedding dimension commonly used in natural language processing).

[0042] 2. Contains invalid values: Due to measurement errors, sensor failures, data acquisition interruptions, etc., there are a large number of invalid marker values ​​(Sentinel values, such as -1000000000, etc.) in the data.

[0043] 3. Continuous numerical type: The data is a vector of floating-point numbers, rather than discrete symbols or tokens.

[0044] This embodiment uses a line laser contour sensor to acquire workpiece surface contour data. The raw data is stored in the following format:

[0045] ;

[0046] in, It is a sequence of three-dimensional points on a cross-section. The horizontal axis; The vertical axis; For height values, extract the height sequence from the original 3D points as the input sequence: .

[0047] S2: Sequence downsampling and block division

[0048] To reduce the computational complexity of long sequences, the model downsamples the original sequence after input:

[0049] ;

[0050] in: Let be the new sequence obtained after downsampling, and s be the length of the downsampled sequence; where:

[0051]

[0052] Where m is the sampling step size, which is usually related to the sensor resolution. If the sensor has n=1600, a sampling accuracy of 0.05, and a maximum detection accuracy of 0.2 mm, then m can be a maximum of 4. Subsequently, the input sequence is divided into fixed-length sequence blocks:

[0053] ;

[0054] in, For a fixed-length sequence block, Let be the length of each sequence after the split. At this point, each input is constructed as a set of sequences of the same length. The number of sequences is:

[0055] ;

[0056] Since the original data is a continuous geometric cross-section, its arrangement order must be maintained after segmentation; otherwise, the true physical meaning reflected by the geometric cross-section will be destroyed. This sequence can be regarded as a special time series.

[0057] S3: Construct a data validity mask

[0058] To address the large number of invalid values ​​in cross-sectional data in industrial measurements, a data validity mask is constructed:

[0059] ;

[0060] Where M is a binary validity mask matrix, where a value of 1 indicates that the data is valid and can participate in the calculation, and a value of 0 indicates that the data is invalid. Mark invalid values, M i For the corresponding z i The mask element at the location of.

[0061] Masking mechanisms are related to contour measurement methods. Currently, line contour sensors used for geometric cross-section measurement output measurement results using equidistant sampling. Invalid values ​​are usually filled with maximal positive / negative values ​​to ensure that each sampling yields a sequence of the same length, which greatly facilitates data acquisition and processing. However, without masking when inputting into the model, invalid values ​​will directly participate in propagation, submerging the features of the normal measurement area and causing the network to fail.

[0062] The masking mechanism is used to prevent invalid data from participating in attention and normalization calculations. Based on this masking mechanism, further masking normalization methods and masking attention mechanisms can be combined to effectively adapt to this type of data source and improve model performance.

[0063] S4: Location Code

[0064] Sine position coding is used to enhance sequence position information:

[0065] ,

[0066] in, This is the positional encoding matrix. The sequence index determines the frequency of the sine wave used in the current encoding; pos is the sequence position; d model As the feature dimension, d in this invention model =l.

[0067] This invention employs fixed sinusoidal positional encoding to introduce positional information into the input sequence, enabling the model to learn the spatial geometric relationships between cross-sections in an industrial contour sequence. Since the industrial contour data, after downsampling and block processing, retains only height values, the original sampled position coordinates no longer directly participate in network calculations. Therefore, positional encoding can recover the relative positional relationships between sequence elements, improving the model's ability to represent continuous geometric structures. Compared to learnable positional encoding, fixed sinusoidal positional encoding requires no additional training parameters, reducing the risk of overfitting on small industrial sample data. Furthermore, due to the continuous periodicity of trigonometric functions, it can still generate reasonable positional representations when the length of the sequence to be processed exceeds the length of the training phase, thus possessing better extrapolation capabilities and being suitable for reconstructing industrial contour sequences of varying lengths.

[0068] S5: Encoder Feature Extraction

[0069] The input sequence is fed into a masked Transformer encoder, which includes a multi-head self-attention module, a feedforward neural network, residual connections, and a layer normalization module. The self-attention calculation formula is as follows:

[0070] ;

[0071] Where Q is the query matrix; K is the key vector matrix; V is the value matrix; d k For feature dimensions; For the mask matrix, This is the matrix transpose operator.

[0072] The encoder stage involves self-attention calculations, where Q, K, and V are derived from the encoder input sequence after mask normalization. Therefore, all three matrices are batch-size input sequences, as shown in S2. Similarly, during attention calculations, Q, K, and V need to be operated on with the mask to ensure that the attention does not include invalid positions. Industrial contour data differs significantly from natural language sequences. To ensure consistent contour sequence length for each sampling, line laser contour sensors typically use extremely positive or negative values ​​as invalid values ​​to fill missing regions. Therefore, invalid data is discretely distributed throughout the sequence, rather than concentrated only at the beginning and end. If the standard Transformer's self-attention mechanism is directly used, these invalid positions also participate in the similarity calculation between the query matrix and the key matrix, in the matrix product QK. T This results in abnormally high correlation scores. After Softmax normalization, abnormal locations will occupy a large attention weight, causing the model to incorrectly focus attention on invalid regions, while the correlation between true contour features is significantly weakened. Ultimately, this leads to distorted attention distribution and affects the encoder's ability to learn effective geometric information.

[0073] Furthermore, since self-attention computation has the characteristic of global information interaction, an invalid position will not only affect its own corresponding attention weight, but will also propagate to the entire sequence through the attention matrix, causing invalid information to spread to the feature representation of all positions.

[0074] This invention proposes a masked attention mechanism. When calculating attention scores, a data validity mask is introduced into the correlation calculation between the query matrix and the key matrix. Attention associations are only allowed between valid data points, while the attention weights corresponding to invalid locations are masked or assigned minimum values, causing them to approach zero after Softmax calculation and thus not participate in subsequent feature aggregation. Subsequently, the value matrix V is weighted and summed only according to the valid attention weights, achieving effective propagation of feature information.

[0075] Through the masked attention mechanism, this invention can effectively prevent invalid values ​​from interfering with the global feature modeling process, so that the attention weights learned by the encoder can truly reflect the spatial correlation of the industrial contour section, significantly improve the model's attention to effective geometric features, and provide a more reliable global feature representation for the subsequent decoder to complete high-precision contour reconstruction.

[0076] The feedforward network layer structure is as follows:

[0077] ;

[0078] in, and The coefficient matrix is ​​a real number field. and For linear bias, The activation function is the standard sigmoid function, which will not be elaborated further. The input x of the feedforward layer is the intermediate result obtained after the input of the encoder of this layer passes through the normalization layer, multi-head attention, and normalization again. Its shape is the same as the input in S2.

[0079] Residual connection:

[0080] Residual connections achieve cross-layer transfer of original feature information by directly adding the sub-layer input and sub-layer output. The calculation formula is as follows:

[0081] ;

[0082] Where C is the module input, F(C) is the sub-layer output, and Y is the output after residual connection.

[0083] In the encoder, each sublayer employs a residual connection structure. Specifically, the input and output of the multi-head self-attention module are added element-wise, and then normalized through a masking layer to obtain the input for the next sublayer. The feedforward neural network also uses a residual connection method that adds the input and output. This invention utilizes residual connections to preserve the original geometric features of industrial contours, alleviating the gradient vanishing problem in deep network training, improving model convergence speed and training stability, while simultaneously preventing excessive smoothing of local contour features during multi-layer propagation.

[0084] Layer normalization:

[0085] Standard normalization is a common component of neural networks, used to prevent gradient explosion and improve model training stability. The variance is calculated as follows:

[0086] ;

[0087] in, The mean is given by s, where s is the length of the downsampled sequence. The final output is:

[0088] ;

[0089] in, To enter the normalized sequence; This is the mean. For the normalized result, The variance under the mask, To ensure the value is a very small positive value, preventing division by zero. For linear bias, It is a linear transformation matrix. This is the final output result after normalization.

[0090] S6: Mask layer normalization

[0091] To avoid invalid values ​​affecting the calculation of statistics, this embodiment proposes a layer normalization method based on data masks. The mask mean calculation method used in the model is as follows:

[0092] ;

[0093] The variance is calculated as follows:

[0094] ;

[0095] The normalization result is:

[0096] ;

[0097] Final output:

[0098] ;

[0099] And through a mask: This enables the forced masking of invalid values, ensuring that model predictions are output only at valid locations.

[0100] in, To enter the normalized sequence; This is the mean value after removing invalid values. for The mask value corresponding to the position, and Sum is the summation operation. Normalization results The variance under the mask, To ensure the value is a very small positive value, preventing division by zero. For linear bias, It is a linear transformation matrix. This is the final output result after normalization.

[0101] This normalization method effectively avoids the influence of invalid values ​​on the statistical calculation results. In industrial contour measurement, to ensure that each acquisition yields a contour sequence of consistent length, line laser contour sensors typically use preset invalid values ​​to fill in unmeasured areas. These invalid values ​​are generally extremely large positive or negative values, such as ±10. 9 The order of magnitude; while the geometric feature height of the actual contour is typically only around 10. -1 ~10 -2 The difference is on the order of magnitude, approximately 10 orders of magnitude. If the traditional layer normalization method is directly used to calculate the mean and variance of all data, extreme invalid values ​​will dominate the calculation of the statistics, causing the mean and variance to deviate significantly from the distribution range of the true geometric features. This results in the effective data being compressed to an extremely small range or even close to zero after normalization, thus completely submerging the true contour features with invalid values.

[0102] Furthermore, since the layer normalization result is directly used as the input to the subsequent multi-head self-attention module and feedforward neural network, erroneous statistics will cause a severe shift in feature representation, preventing the attention weights from accurately focusing on the effective contour region. This can lead to abnormal gradient propagation, difficulty in model training convergence, or even complete failure. Simultaneously, invalid values ​​may accumulate and amplify during residual connections and multi-layer network propagation, continuously interfering with feature extraction in subsequent layers and further reducing model stability and reconstruction accuracy.

[0103] This invention proposes a layer normalization method based on data validity masks. Only data marked as valid by the mask are used in the calculation of the mean and variance. Invalid positions are not included in the statistics during the normalization process, and the mask is used again in the output stage to force the corresponding positions to zero. This method effectively maintains the true statistical characteristics of valid contour data, avoids the contamination of normalization results by extreme invalid values, improves the stability of model numerical calculations, and ensures that subsequent attention calculations and feature extraction are based on real industrial contour information. Therefore, it significantly improves the training stability, convergence speed, and reconstruction accuracy of industrial contour sequences under conditions containing a large amount of missing data, occluded regions, and noise interference.

[0104] Unlike natural language processing, which only performs overall masking on sequence padding, the mask layer normalization method proposed in this invention can be directly applied to continuous industrial contour data, which is more in line with the data characteristics of industrial measurement equipment such as line laser contour sensors, and has better engineering applicability and robustness.

[0105] S7: Decoder Reconstruction

[0106] The decoder includes: a self-attention layer; a cross-attention layer; a feedforward network; residual connections; and a layer normalization module. Its cross-attention calculation method is the same as in the previous step. The difference is that in the decoder's self-attention calculation, Q, K, and V all come from the decoder's input sequence after mask normalization. Additionally, in the cross-attention calculation, Q is the normalized result of the target sequence, and K and V come from the encoder's output memory matrix.

[0107] The calculation formula for cross-attention layers is the same as that for self-attention layers:

[0108] ;

[0109] Where Q is the query matrix; K is the key vector matrix; V is the value matrix; d k For feature dimensions. For the mask matrix, This is the matrix transpose operator, where Q, K, and V follow the standard Transformer definition.

[0110] The decoder stage includes two computational methods: self-attention and cross-attention layers. All three matrices are batch-size input sequences, as shown in S2. Similarly, when calculating attention, Q, K, and V all need to be operated on with the mask to ensure that the attention does not include invalid positions. The role of masked attention has already been described in detail in S5 and will not be repeated here.

[0111] The feedforward network layer structure is as follows:

[0112] ;

[0113] in, and The coefficient matrix is ​​a real number field. and For linear bias, The activation function is the standard sigmoid function, which will not be elaborated further. The input x of the feedforward layer is the intermediate result obtained after the input of the encoder of this layer passes through the normalization layer, multi-head attention, and normalization again. Its shape is the same as the input in S2.

[0114] Residual connectivity;

[0115] By directly adding the sub-layer input and sub-layer output, the original feature information is transferred across layers. The calculation formula is as follows:

[0116] ;

[0117] Where C is the module input, F(C) is the sub-layer output, and Y is the output after residual connection.

[0118] Since cross-attention further fuses encoder information based on the self-attention extraction results, this invention treats the two attention sub-modules as a whole and performs residual connection. Specifically, the input features before entering the self-attention module are added element-wise with the output of the cross-attention module.

[0119] Layer normalization module:

[0120] Standard normalization is a common component of neural networks, used to prevent gradient explosion and improve model training stability. It requires calculating the variance:

[0121] ;

[0122] in, Let be the mean, and s be the length of the input sequence. Final output:

[0123] ;

[0124] in, To enter the normalized sequence; This is the mean. Normalization results The variance under the mask, To ensure the value is a very small positive value, preventing division by zero. For linear bias, It is a linear transformation matrix. This is the final output result after normalization.

[0125] The decoder will eventually output the reconstructed sequence:

[0126] .

[0127] Where Y represents the final output, and each element is y. i , indicating that its length is the same as X, and the length of the sampled sequence is s.

[0128] S8: Loss Function Construction

[0129] The model is jointly trained using Huber Loss and Sequence Constraint Loss (BCE). The Huber Loss expression is: where the weights of both can be determined by Coe. weight The adjustment is that this model primarily uses Huber Loss, therefore Coe weight <0.5, and this weight allocation will be used as a hyperparameter in subsequent training experiments.

[0130] ;

[0131] ;

[0132] ;

[0133] in, For the overall loss, Here, N represents the weighting coefficients, and N is the number of sequences. For the loss function of Huber Loss, L BCE The loss function for BCE loss is... The true value at the corresponding position in the target sequence. The reconstructed values ​​predicted by the model. The smoothing coefficient (threshold) is used to control the switching position of the loss function from quadratic error to linear error, where all the values ​​involved in the loss calculation are model predictions calculated using a mask.

[0134] This invention employs Huber Loss as the reconstruction loss function. When the prediction error is small, quadratic error is used for optimization to ensure high convergence accuracy. When the prediction error is large, linear error is used for optimization to reduce the impact of outliers on model training and improve the model's robustness to industrial contour noise and locally missing data. Simultaneously, by combining a data validity mask, loss is calculated only for valid locations, while invalid locations are not included in error backpropagation, thereby further improving model training stability and the accuracy of industrial contour sequence reconstruction.

[0135] Furthermore, BCE loss is introduced as an auxiliary constraint. BCE constrains the model's prediction results at each sampling location, ensuring that the reconstructed sequence maintains consistency with the target sequence in overall distribution. Unlike Huber Loss, which primarily focuses on numerical errors, BCE enhances the model's ability to preserve contours and continuous structures, reducing boundary blurring, local breaks, and misclassification, making the model output closer to the target sequence in both overall shape and local structure.

[0136] Ultimately, this invention employs Huber Loss and BCE loss for joint optimization. These two methods complement each other, ensuring local numerical accuracy while also considering overall geometric shape, thereby further improving the reconstruction quality of industrial contour sequences and the model's generalization ability.

[0137] Example 2

[0138] This embodiment provides an industrial contour sequence reconstruction system based on temporal Transformer, including a data loading unit, a model processing unit, a training unit, and an inference unit.

[0139] The data loading unit is used to load, downsample, and segment high-dimensional structured binary sequence data, and generate validity masks. The data loading unit divides the long sequence data into a fixed number of vector blocks, each containing a preset number of feature dimensions, and generates a corresponding validity mask for each feature location to mark invalid values.

[0140] The model processing unit constructs a Transformer model containing an encoder and decoder, and performs mask-aware feature extraction and attention computation on the input sequence. The unit includes a mask-aware embedding module, which linearly projects the high-dimensional vector, normalizes it using a validity mask, and forces the feature values ​​at invalid locations to zero. The unit also includes a mask-aware layer normalization module, which calculates statistics only for valid data locations and forces the output at invalid locations to zero. The attention mechanism of the unit includes an attention correction module, which introduces a linearly decaying bias based on temporal distance into the attention score to achieve a control effect where the attention weight of historical information further away from the current time step is lower. The decoder includes optional self-attention and cross-attention modules; the cross-attention module uses the encoder's output as key-value pairs.

[0141] The training unit is used to train the model using a mask loss function. The training unit employs a mask loss function, which calculates the reconstruction error only for positions corresponding to valid masks; invalid positions are not included in the loss calculation.

[0142] The inference unit performs non-autoregressive reconstruction of the input sequence, outputting a high-dimensional vector sequence with the same dimension as the input. The inference unit employs a non-autoregressive approach, using the source sequence as the initial input to the decoder and reconstructing the entire sequence through a single forward propagation.

[0143] Example 3

[0144] This embodiment illustrates the workflow of the system described in Embodiment 2.

[0145] The high-dimensional time series vector sequence reconstruction system of the present invention mainly includes the following steps: data preparation and loading, model building, model training, model inference and output.

[0146] 1. Data loading and preprocessing

[0147] This embodiment uses a structured binary file format. Each record contains a 32-bit integer key and a sequence of values ​​in a float32 array whose length is equal to the number of sensor sampling points. The specific processing flow is as follows:

[0148] (1) Traverse multiple sets of data files in the specified directory (.dat is the source sequence, .tgt is the target sequence).

[0149] (2) Implement memory-mapped lazy loading through the following steps:

[0150] First, the Z component of the sensor sampling point coordinates is extracted, downsampled, and the downsampled sequence is segmented according to a certain size to obtain the initial input sequence of the model. Then, the target sequence is binarized (threshold, default 50) and normalized to the interval [-1, 1]. Subsequently, a validity mask (datamask) is generated, where valid positions are True and invalid positions (close to DATAERRVALUE = -1000000000) are False.

[0151] Similar to data loading for all deep learning models, this method supports proportional partitioning of the training / validation set and random combination of training data.

[0152] 2. Model Building

[0153] This embodiment describes the structure of the constructed model. The structure of the Transformer model with masked structure and Encoder / Decoder structure constructed in this invention is as follows:

[0154] (1) The high-dimensional vector is input to the embedding layer, first mapped to the model dimension through a linear projection layer, and then normalized by combining the data mask. The process is as follows:

[0155] 1) The mask mean used in the model is calculated as follows: ;

[0156] 2) Variance is calculated as follows: ;

[0157] 3) The normalization result is: .

[0158] (2) The position encoding adopts the standard sinusoidal position encoding, and the maximum length is configurable, with a default of 5000. The specific content of the sinusoidal position encoding is described in step S4 of Example 1, and will not be repeated here.

[0159] (3) The encoder consists of N stacked layers, each layer containing: multi-head self-attention (the number of heads h can be adjusted); mask normalization layer, feedforward neural network and residual connection.

[0160] (4) The decoder consists of N stacked layers, each layer containing: self-attention; cross-attention; mask normalization layer, feedforward network and residual connection.

[0161] (5) Attention adjustment: After the decoder calculates the attention score, a linear decay bias is added: This bias is superimposed on the causal mask and added to the attention score to achieve the decay effect of the attention weight being lower as the distance is greater.

[0162] (6) The output layer is a linear projection layer that directly outputs a reconstructed vector sequence with the same dimension as the input.

[0163] Specifically as follows:

[0164] ;

[0165] Among them, Y out Y is the final output sequence, A is the linear transformation matrix, and B is the bias matrix.

[0166] 3. Perform model training

[0167] The specific training steps are as follows:

[0168] (1) Construct a training batch and automatically generate the corresponding input sequence and target sequence data mask;

[0169] (2) The model is forward propagated to obtain the output;

[0170] (3) Calculate the mask L1 loss (or Huber loss): Note that the error is only calculated at the valid positions; the invalid positions are forced to zero by the mask and do not participate in the loss calculation; the loss function can be the combined loss function of Huber Loss and BCE loss in Example 1.

[0171] (4) The Adam optimizer and custom learning rate scheduling are used, along with a certain proportion of Warmup;

[0172] (5) Supports gradient accumulation and adapts to large model training;

[0173] (6) Validate after each round of training and save the optimal model.

[0174] 4. Model Reasoning

[0175] The model reasoning process used in this invention is as follows:

[0176] (1) Input the source sequence into the encoder to obtain the memory;

[0177] (2) Use the source sequence itself as the initial input to the decoder (or use mean padding).

[0178] (3) The complete output is obtained by passing through the decoder once;

[0179] (4) Output the reconstructed sequence corresponding to the number of input sequences and perform data visualization.

[0180] (5) Split the output column by column and write it to a file in the same binary format as the input.

[0181] Unlike language models that reason word by word, the model in this invention outputs all sequences in a single propagation.

[0182] In this embodiment, the parameter configuration shown in Table 1 is used:

[0183] Table 1. Parameter Configuration Table for the Model

[0184]

[0185] The model of this invention is implemented using the PyTorch deep learning framework. The input data first undergoes height value extraction and downsampling, then a data validity mask is constructed, and the data is fed into a Transformer encoder and decoder for feature extraction and sequence reconstruction. During model training, mask layer normalization and residual structures are used to improve training stability. The final output is a reconstructed industrial contour sequence. The results are as follows... Figure 5 , 6 As shown.

[0186] Figure 5The composite material ply morphology cross-sectional data continuously acquired by the centerline profile sensor is sampled to obtain the result in (a), where the horizontal axis represents the number of sampling points and the vertical axis represents the number of sequences. (b) shows the corresponding sequence data containing only defect locations after reconstruction and filtering. These two sets of data are the data formats used during model training in this invention. The final output of the model will be mapped to a value range between -1 and 1, as shown in Figure (c). When the model performs continuous sequence processing, its processing results can be transformed into... Figure 6 The binarized image data shown here more intuitively reflects the processing results.

[0187] It should be noted that this model processes cross-sectional data acquired by a line laser sensor, which is geometric contour data, unlike time-series signals or 3D point clouds. Furthermore, this data can be input into the model without requiring conversion or mapping algorithms. Currently, there is no dedicated network model for this data type.

[0188] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for industrial contour sequence reconstruction and filtering based on temporal Transformer, characterized in that: The input industrial contour data first undergoes height value extraction, downsampling, and block processing. Then, a data validity mask is constructed and fed into the Transformer encoder and decoder for feature extraction and sequence reconstruction. During model training, mask layer normalization and residual structure are used to improve training stability. Finally, the reconstructed industrial contour sequence is output.

2. The industrial contour sequence reconstruction and filtering method based on temporal Transformer according to claim 1, characterized in that: Height sequences are extracted from industrial contour data collected by sensors, and a time-series input matrix is ​​constructed.

3. The industrial contour sequence reconstruction and filtering method based on temporal Transformer according to claim 2, characterized in that: The height sequence is downsampled, specifically as follows: To reduce the computational complexity of long sequences, the model downsamples the original input sequence after input: ； in: Let s be the new sequence obtained after downsampling, and s be the length of the downsampled sequence; in the above formula: ； Where m is the sampling step size, which is related to the resolution of the sensor. If the sensor n=1600, the sampling accuracy is 0.05, and the highest detection accuracy is 0.2 mm, then m can be up to 4. The input sequence is then divided into fixed-length blocks: ； in, For a fixed-length sequence block, Let be the length of each sequence after segmentation. At this point, each input is constructed as a set of sequences of the same length; the number of sequences is: 。 4. The industrial contour sequence reconstruction and filtering method based on temporal Transformer according to claim 1, characterized in that: To address the large number of invalid values ​​in cross-sectional data in industrial measurements, a data validity mask is constructed: ； Where M is a binary validity mask matrix, where a value of 1 indicates that the data is valid and can participate in the calculation, and a value of 0 indicates that the data is invalid. This is a flag indicating an invalid value.

5. The industrial contour sequence reconstruction and filtering method based on temporal Transformer according to claim 1, characterized in that: The masked Transformer encoder includes: a multi-head self-attention module, a feedforward neural network, residual connections, and a layer normalization module.

6. The industrial contour sequence reconstruction and filtering method based on temporal Transformer according to claim 1, characterized in that: The decoder includes: Self-attention layer, encoder-decoder cross-attention layer, feedforward network, residual connection and layer normalization module; The decoder will eventually output the reconstructed sequence: ； Where Y represents the final output, and each element is y. i Its length is the same as the input X, which is the length s of the sampled sequence.

7. The industrial contour sequence reconstruction and filtering method based on temporal Transformer according to claim 1, characterized in that: In mask layer normalization, the mask mean is calculated as follows: ； The variance is calculated as follows: ； The normalization result is: ； Final output: ； And through a mask: This enables the forced masking of invalid values, ensuring that model predictions are output only at valid locations; in, To enter the normalized sequence; This is the mean value after removing invalid values. for The mask value corresponding to the position, and Sum is the summation operation. For the normalized result, The variance under the mask, To ensure the value is a very small positive value, preventing division by zero. For linear bias, It is a linear transformation matrix. This is the final output result after normalization.

8. The industrial contour sequence reconstruction and filtering method based on temporal Transformer according to claim 1, characterized in that: The model is jointly trained using Huber Loss and Sequence Constraint Loss (BCE). The expression for Huber Loss is: ； ； ； in, Let L be the loss function of Huber Loss. BCE The loss function for BCE loss is... This represents the true value at the corresponding position in the target sequence. The reconstructed values ​​predicted by the model. The smoothing coefficient is used to control the switching position of the loss function from quadratic error to linear error. The values ​​used in the loss calculation are all model predictions calculated using a mask.

9. An industrial contour sequence reconstruction system based on temporal Transformer, used to implement the method described in any one of claims 1-8, characterized in that: It includes a data loading unit, a model processing unit, a training unit, and an inference unit; The data loading unit is used to load, downsample, and segment high-dimensional structured binary sequence data, and generate a validity mask. The model processing unit is used to construct a Transformer model containing an encoder and a decoder, and to perform mask-aware feature extraction and attention calculation on the input sequence. The training unit is used to train the model using the mask loss function; The inference unit is used to perform non-autoregressive reconstruction on the input sequence and output a high-dimensional vector sequence with the same dimension as the input.

Images