Self-supervised generative pre-training-fine tuning method based on learnable mask
Through the self-supervised generative pre-training-fine-tuning method based on learnable masks, the problem of industrial label-free time sequence data utilization is solved, the data learning ability and feature extraction effect of deep learning models are improved, and the application of different downstream tasks is adapted to different downstream tasks.
Patent Information
- Application Number
- CN202411023061.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-15
AI Technical Summary
In industrial scenarios, many time-series data lack labels, making it difficult for deep learning models to train and cannot effectively utilize a large amount of label-free time-series data.
Using a self-supervised generative pre-training-fine-tuning method based on learnable masks, the optimal mask sequence is obtained by constructing a multi-channel timing data set, using cosine similarity calculation, and generative pre-training of the Transformer encoder and two-stage fine-tuning , extract timing features and adapt to different downstream tasks.
Make full use of label-free data to improve the data learning ability of deep learning models, improve the timing feature extraction effect, adapt to short-term segments and long-term window data, and achieve good performance in different downstream tasks.
Smart Images

Figure CN118964986A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of time series prediction and fault diagnosis, and in particular relates to a self-supervised generative pre-training-fine-tuning method based on a learnable mask. Background Art
[0002] Timing signals in the industrial field can be used in many fields such as predictive control, anomaly detection, equipment operation status, health status monitoring, etc., and are crucial to the stability, safety and production efficiency of the production process.
[0003] Driven by a large amount of industrial data, time series signal processing methods based on deep learning have been widely used. However, supervised deep learning methods require a large amount of labeled data to train models. In real industrial scenarios, there are a large number of unlabeled industrial process data recorded by sensors, which cannot be used to train deep learning models. The cost of labeling data sets is relatively high, resulting in a large amount of time series data being unable to be used. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a self-supervised generative pre-training-fine-tuning method based on learnable masks, which solves the problem that a large amount of industrial unlabeled time series data cannot be effectively utilized in the prior art.
[0005] The present invention adopts the following technical solutions to solve the above technical problems:
[0006] The self-supervised generative pre-training-fine-tuning method based on learnable masks includes the following steps:
[0007] Step 1: Construct a multi-channel time series data set with a size of the number of channels × the total length of the time series signal;
[0008] Step 2: Segmentation of time series data: each channel in the multi-channel time series data set is divided into multiple segments at equal intervals, and each segment contains data points of only one channel;
[0009] Step 3: construct a learnable mask network based on cosine similarity, output the optimal mask sequence distribution in the current time window, and set the time sequence segment specified by the optimal mask sequence to 0 for masking;
[0010] Step 4: Generative pre-training: pre-train the Transformer encoder to reconstruct the masked time series segments;
[0011] Step 5: Two-stage fine-tuning: Replace the output layer with the prediction layer and the classification layer, and perform two-stage fine-tuning on the pre-trained Transformer encoder according to different types of downstream tasks.
[0012] In step 2, a two-dimensional data set is transformed into three dimensions after segmentation, where the three dimensions are the length of each segment, the number of channels, and the number of segments in each channel.
[0013] The learnable mask network based on cosine similarity consists of a temporal feature extraction module and a cosine similarity calculation module; the temporal feature extraction module consists of a convolutional layer and a multi-head attention layer; the convolutional layer extracts temporal features for each temporal segment, and the multi-head attention layer extracts temporal features of the channel for the sequence in the entire input time window; the final mask sequence distribution is then calculated through cosine similarity.
[0014] The convolutional layer extracts temporal features for each segment:
[0015]
[0016] in, is the output of the convolutional layer, w j represents the jth convolution kernel matrix, k represents the convolution kernel length, and the Relu activation function is selected to introduce nonlinear features. After the convolution layer, the temporal features of each fragment in the low-dimensional space are expressed as:
[0017]
[0018] in, Represents the nth time segment of the ith channel.
[0019] The multi-head attention layer consists of three matrices: query matrix Q, key matrix K and value matrix V, which are defined as follows:
[0020]
[0021] in, as well as represents the learnable linear matrix, d k represents the dimensions of the query vector and the key vector; the attention score is obtained by i and (K i ) T The result is obtained by performing a dot multiplication operation and finally mapped to the [0, 1] interval through the Softmax function:
[0022]
[0023] in, is the output of an attention head in the i-th channel. The output of the multi-head attention layer will concatenate the results of all heads:
[0024]
[0025] Where h represents the number of heads, W o is the weight matrix of multiple attention heads. Finally, the temporal features extracted from the entire input time window by the linear layer are:
[0026]
[0027] Among them, W1, W2, b1, b2 represent the learnable parameters in the linear layer.
[0028] The cosine similarity calculation module calculates the cosine similarity between each segment and the entire window time series feature, selects the segments containing the most useful time series features for masking, and the cosine similarity score is The calculation formula is as follows:
[0029]
[0030] Then the required mask sequence distribution is obtained according to the cosine similarity score:
[0031]
[0032] Among them, I i,n represents the sequence of fragments arranged in descending order of cosine similarity scores, Indicates the fragment sequence number that will be masked.
[0033] The Transformer encoder adds a layer normalization method:
[0034]
[0035] Among them, x i represents the i-th feature in the input matrix, X represents all features, ε is a small positive number, in order to prevent the denominator from being 0, γ i and β i are two learnable matrix parameters, μ(X) and σ(X) representing the mean and variance of the input features.
[0036] During the pre-training process, the training task of the Transformer encoder is set to reconstruct the masked fragments and learn the distribution characteristics of the data itself. The loss function is the mean square error MSE:
[0037]
[0038] in, represents the mask-reconstruction loss during pre-training, N mask represents the number of masked fragments, x i_mask represents the true value of the masked fragment, Represents the predicted value of the model.
[0039] In step 5, the two-stage fine-tuning method mainly includes: in the first stage, the Transformer encoder parameters are frozen and the prediction / classification output layer is fine-tuned; in the second stage, the entire model is fine-tuned, where
[0040] For the prediction task, the fine-tuning loss function of the model Defined as:
[0041]
[0042] Among them, N p Indicates the length of the predicted time series signal, x i_p represents the real time series data points, Represents the time series data points predicted by the model;
[0043] For classification tasks, the fine-tuning loss function of the model is Defined as:
[0044]
[0045] Among them, N c represents the number of samples in the data set, C represents the number of classification categories, and y i_c,c Indicates whether the i_cth sample belongs to the true label of category c, if it does, it is 1, otherwise it is 0. It represents the probability that the model predicts that the i_cth sample belongs to category c. log(·) is the natural logarithm function, which is used to calculate the logarithm value of the predicted probability.
[0046] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, all or part of the steps of the method are called.
[0047] Compared with the prior art, the present invention has the following beneficial effects:
[0048] 1. Through the generative pre-training paradigm of mask-reconstruction, we can make full use of the unlabeled data generated in the industrial production process to improve the data learning ability of the deep learning model.
[0049] 2. The temporal feature extraction process of time series data uses convolutional layers and multi-head autonomous attention layers to adapt to the characteristics of short time series fragment data and long time window data to achieve better feature extraction effects.
[0050] 3. The learnable mask network uses cosine similarity to provide a quantification method for time series segments that contain useful time series features. It masks time series segments that contain more useful time series features, allowing the model to learn more potential features of the data.
[0051] 4. The stage-by-stage fine-tuning method enables the model to achieve better results in different downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 Flowchart of the self-supervised generative pre-training-fine-tuning method based on learnable masks proposed in this invention.
[0053] Figure 2 Figure 2: The structure diagram of the self-supervised generative pre-training model based on learnable masks. DETAILED DESCRIPTION
[0054] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0055] The self-supervised generative pre-training-fine-tuning method based on learnable masks includes the following steps:
[0056] Step 1: Construct a multi-channel time series data set with a size of the number of channels × the total length of the time series signal;
[0057] Step 2: Segmentation of time series data: each channel in the multi-channel time series data set is divided into multiple segments at equal intervals, and each segment contains data points of only one channel;
[0058] Step 3: construct a learnable mask network based on cosine similarity, output the optimal mask sequence distribution in the current time window, and set the time sequence segment specified by the optimal mask sequence to 0 for masking;
[0059] Step 4: Generative pre-training: pre-train the Transformer encoder to reconstruct the masked time series segments;
[0060] Step 5: Two-stage fine-tuning: Replace the output layer with the prediction layer and the classification layer, and perform two-stage fine-tuning on the pre-trained Transformer encoder according to different types of downstream tasks.
[0061] Specific embodiments, such as Figure 1 , Figure 2 As shown,
[0062] A self-supervised generative pre-training-fine-tuning method based on learnable masks, the method flow is as follows Figure 1 As shown in the figure, the structure of the self-supervised generative pre-training model based on learnable masks is as follows Figure 2 As shown, the specific steps include:
[0063] 1) Build a training dataset
[0064] The public Paderborn University bearing dataset and Cincinnati Intelligent Maintenance System dataset were selected as the original database. The selected data types are motor current and vibration signals with a sampling frequency of 64 khz, and mechanical parameters (including force, speed and torque) with a sampling frequency of 4 khz. Each bearing in the dataset was tested and recorded under four different experimental conditions of speed, load torque and radial force. The Cincinnati Intelligent Maintenance System dataset contains data from 4 bearings on the same shaft, driven by an AC motor, and a failure test experiment was performed to construct a bearing vibration signal dataset. The experimental conditions are: speed 2000 rpm, radial load 6000 lbs applied to the bearing, sampling frequency 20 khz, and a total of 20480 data points were recorded.
[0065] The dataset constructed in the embodiment includes 6 sub-datasets constructed from data of 6 bearings (K001-K006) in the Paderborn University bearing dataset, and 3 sub-datasets from the Cincinnati Intelligent Maintenance System dataset. The datasets are shown in Table 1.
[0066] Table 1 Dataset description
[0067]
[0068] The data set size is the number of channels × the total length of the time series signal.
[0069] 2) Time series data segmentation processing
[0070] For input multi-channel time series data Where C and L represent the number of channels in the current time window and the length of the input sequence respectively. Performed separately, where i = {1, 2, ..., C}. To ensure the subsequent mask operation, the length and moving step of each segment are the same, both set to P, that is, non-overlapping segmentation. The length of the segment obtained is N, N = L / P. The time series data after segmentation becomes a three-dimensional data set:
[0071] 3) The learnable mask network based on cosine similarity consists of a temporal feature extraction module and a cosine similarity calculation module. The temporal feature extraction module consists of a convolutional layer and a multi-head attention layer. The convolutional layer extracts temporal features for each temporal segment, and the multi-head attention layer extracts temporal features of the channel for the sequence in the entire input time window. The final mask sequence distribution is then calculated through cosine similarity.
[0072] The cosine similarity-based learnable mask network extracts temporal features for each segment through a convolutional layer:
[0073]
[0074] in is the output of the convolutional layer, w j Represents the jth convolution kernel matrix, and k represents the convolution kernel length. For multidimensional time series with nonlinear relationships, convolution operations require activation functions to improve feature extraction capabilities. Here, the Relu activation function is selected to introduce nonlinear features. After the convolution layer, the time series features of each segment in the low-dimensional space are expressed as:
[0075]
[0076] in Represents the nth time segment of the ith channel.
[0077] The multi-head attention layer extracts temporal features from the entire input time window. Its operation mainly includes three matrices: query matrix Q, key matrix K and value matrix V, which are defined as follows:
[0078]
[0079] in as well as represents a learnable linear matrix, d k represents the dimension of query vector and key vector. The attention score is obtained by i and (K i ) T The result is obtained by performing a dot multiplication operation and finally mapped to the [0, 1] interval through the Softmax function:
[0080]
[0081] in is the output of an attention head in the i-th channel. The output of the multi-head attention layer is the concatenation of the results of all heads:
[0082]
[0083] Where h represents the number of heads, W O is the weight matrix of multiple attention heads. Finally, the temporal features extracted from the entire input time window by the linear layer are:
[0084]
[0085] Where W1, W2, b1, b2 represent the learnable parameters in the linear layer.
[0086] The cosine similarity calculation module calculates the cosine similarity between each segment and the entire window time series feature to select the most useful time series features in each segment for masking. The calculation formula is as follows:
[0087]
[0088] Then the required mask sequence distribution is obtained according to the cosine similarity score:
[0089]
[0090] Among them I i,n represents the sequence of fragments arranged in descending order of cosine similarity scores, Indicates the fragment sequence number that will be masked.
[0091] 4) Transformer encoder performs feature extraction based on the self-attention mechanism and adds a layer normalization method:
[0092]
[0093] where x i represents the i-th feature in the input matrix, X represents all features, ε is a small positive number, in order to prevent the denominator from being 0, γ i and β i are two learnable matrix parameters, μ(X) and σ(X) representing the mean and variance of the input features.
[0094] During the pre-training process, the training task of the Transformer encoder is set to reconstruct the masked fragments and learn the distribution characteristics of the data itself. The loss function is the mean square error MSE:
[0095]
[0096] in represents the mask-reconstruction loss during pre-training, N mask represents the number of masked fragments, x i_mask express
[0097] Mask out the true value of the fragment, Represents the predicted value of the model.
[0098] 5) The pre-trained Transformer encoder is adapted to different types of downstream tasks through a two-stage fine-tuning strategy. The two-stage fine-tuning strategy mainly includes: the first stage freezes the Transformer encoder parameters and fine-tunes the prediction / classification output layer; the second stage fine-tunes the entire model. Specifically, for the prediction task, the model's fine-tuning loss function Defined as:
[0099]
[0100] Where N p Indicates the length of the predicted time series signal, x i_p represents the real time series data points, Represents the time series data points predicted by the model. For classification tasks, the fine-tuning loss function of the model Defined as:
[0101]
[0102] Where N c represents the number of samples in the data set, C represents the number of classification categories, and y i_c,c Indicates whether the i_cth sample belongs to the true label of category c, if it does, it is 1, otherwise it is 0. It indicates the probability that the model predicts that the i_cth sample belongs to category c. The output is usually processed by the Softmax function so that the sum of the predicted probabilities of all categories is 1. log(·) is the natural logarithm function, which is used to calculate the logarithm value of the predicted probability.
[0103] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A self-supervised generative pre-training-fine-tuning method based on learnable masks, characterized by: The steps include: Step 1: Construct a multi-channel time series data set with a size of the number of channels × the total length of the time series signal; Step 2: Segmentation of time series data: each channel in the multi-channel time series data set is divided into multiple segments at equal intervals, and each segment contains data points of only one channel; Step 3: construct a learnable mask network based on cosine similarity, output the optimal mask sequence distribution in the current time window, and set the time sequence segment specified by the optimal mask sequence to 0 for masking; Step 4: Generative pre-training: pre-train the Transformer encoder to reconstruct the masked time series segments; Step 5: Two-stage fine-tuning: Replace the output layer with the prediction layer and the classification layer, and perform two-stage fine-tuning on the pre-trained Transformer encoder according to different types of downstream tasks.
2. The self-supervised generative pre-training-fine-tuning method based on learnable masks according to claim 1, characterized in that: In step 2, a two-dimensional data set is transformed into three dimensions after segmentation, where the three dimensions are the length of each segment, the number of channels, and the number of segments in each channel.
3. The self-supervised generative pre-training-fine-tuning method based on learnable masks according to claim 1, characterized in that: The learnable mask network based on cosine similarity consists of a temporal feature extraction module and a cosine similarity calculation module; the temporal feature extraction module consists of a convolutional layer and a multi-head attention layer; the convolutional layer extracts temporal features for each temporal segment, and the multi-head attention layer extracts temporal features of the channel for the sequence in the entire input time window; the final mask sequence distribution is then calculated through cosine similarity.
4. The self-supervised generative pre-training-fine-tuning method based on learnable masks according to claim 3, characterized in that: The convolutional layer extracts temporal features for each segment: in, is the output of the convolutional layer, w j represents the jth convolution kernel matrix, k represents the convolution kernel length, and the Relu activation function is selected to introduce nonlinear features. After the convolution layer, the temporal features of each fragment in the low-dimensional space are expressed as: in, Represents the nth time segment of the ith channel.
5. The self-supervised generative pre-training-fine-tuning method based on learnable masks according to claim 3, characterized in that: The multi-head attention layer consists of three matrices: query matrix Q, key matrix K and value matrix V, which are defined as follows: in, as well as represents the learnable linear matrix, d k represents the dimensions of the query vector and the key vector; the attention score is obtained by i and (K i ) T The result is obtained by performing a dot multiplication operation and finally mapped to the [0, 1] interval through the Softmax function: in, is the output of an attention head in the i-th channel. The output of the multi-head attention layer will concatenate the results of all heads: Where h represents the number of heads, W O is the weight matrix of multiple attention heads. Finally, the temporal features extracted from the entire input time window by the linear layer are: Among them, W1, W2, b1, b2 represent the learnable parameters in the linear layer.
6. The self-supervised generative pre-training-fine-tuning method based on learnable masks according to claim 3, characterized in that: The cosine similarity calculation module calculates the cosine similarity between each segment and the entire window time series feature, selects the segments containing the most useful time series features for masking, and the cosine similarity score is The calculation formula is as follows: Then the required mask sequence distribution is obtained according to the cosine similarity score: Among them, I i,n represents the sequence of fragments arranged in descending order of cosine similarity scores, Indicates the fragment sequence number that will be masked.
7. The self-supervised generative pre-training-fine-tuning method based on learnable masks according to claim 1, characterized in that: The Transformer encoder adds a layer normalization method: Among them, x i represents the i-th feature in the input matrix, X represents all features, ε is a small positive number, in order to prevent the denominator from being 0, γ i and β i are two learnable matrix parameters, μ(X) and σ(X) representing the mean and variance of the input features.
8. The self-supervised generative pre-training-fine-tuning method based on learnable masks according to claim 7, characterized in that: During the pre-training process, the training task of the Transformer encoder is set to reconstruct the masked fragments and learn the distribution characteristics of the data itself. The loss function is the mean square error MSE: in, represents the mask-reconstruction loss during pre-training, N mask represents the number of masked fragments, x i_mask represents the true value of the masked fragment, Represents the predicted value of the model.
9. The self-supervised generative pre-training-fine-tuning method based on learnable masks according to claim 1, characterized in that: In step 5, the two-stage fine-tuning method mainly includes: in the first stage, the Transformer encoder parameters are frozen and the prediction / classification output layer is fine-tuned; in the second stage, the entire model is fine-tuned, where For the prediction task, the fine-tuning loss function of the model Defined as: Among them, N p Indicates the length of the predicted time series signal, x i_p represents the real time series data points, Represents the time series data points predicted by the model; For classification tasks, the fine-tuning loss function of the model is Defined as: Among them, N c represents the number of samples in the data set, C represents the number of classification categories, and y i_c,c Indicates whether the i_cth sample belongs to the true label of category c, if it does, it is 1, otherwise it is 0. It represents the probability that the model predicts that the i_cth sample belongs to category c. log(·) is the natural logarithm function, which is used to calculate the logarithm value of the predicted probability.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, all or part of the steps of the method described in any one of claims 1 to 9 are called.
Citation Information
Cited By
Fault prediction and detection method and device based on intelligent model, equipment and medium
CN120493070A
Large model training method, system and equipment for low-code development and medium
CN121031672A
Intelligent control system and method based on time sequence filling network
CN122194709A
Intelligent Control System and Method Based on Temporal Filling Network
CN122194709B
Distributed multi-sensor data fusion unsupervised learning method and system based on multi-dimensional mask
CN122433018A
