System and method for semi-supervised few-shot time series anomaly detection and classification based on pre-trained model guided fine-tuning

Abstract

The system for semi-supervised few-shot time series anomaly detection and classification based on pre-trained model guided fine-tuning includes the following modules: a data input and pre-processing module, a model pre-training module, a guided fine-tuning module, an anomaly detection module, a model module and a storage module; the method for semi-supervised few-shot time series anomaly detection and classification based on pre-trained model guided fine-tuning includes the following steps: (1) the data input and pre-processing module pre-processes the time series data; (2) the model module performs model pre-training; (3) the guided fine-tuning module instructs the model module to perform guided fine-tuning; (4) the anomaly detection module integrates the trained pre-trained model and the fine-tuned guidance vector, and instructs the model module to perform anomaly detection and classification tasks.

Description

Technical Field

[0001] The present invention relates to a system and method for semi-supervised few-sample time series anomaly detection and classification based on fine-tuning guided by a pre-trained model, and belongs to the field of information technology, in particular to the field of network operation and maintenance technology. Background Art

[0002] Network operations and maintenance (O&M) primarily encompass anomaly detection, root cause location, fault classification, and fault remediation, each of which is closely linked. Anomaly detection is a prerequisite for network O&M, aiming to identify anomalies or rare events in time series data. Anomaly detection is a critical task for monitoring the various states (i.e., indicators) of entities (such as manufacturing systems and internet services), which are typically characterized by multivariate or univariate time series. Accurate anomaly detection is crucial in practical engineering projects, as its accuracy and efficiency significantly impact the overall workflow. Accurate anomaly detection can trigger timely troubleshooting, ensuring the accuracy and efficiency of subsequent network O&M work.

[0003] The traditional approach to anomaly detection currently involves domain experts manually establishing static thresholds for each monitored metric (e.g., network throughput, link congestion, CPU utilization, etc.). These experts then utilize various extreme value-based analysis methods to classify time series data with high anomaly scores as anomalies. Alternatively, they employ clustering and statistical analysis to detect anomalies, or employ fast Fourier transforms to detect regions with potential anomalies. However, these traditional methods face the challenge of adapting to the complex time series data of the real world. With the significant increase in network services and server scale, the cost of these manual anomaly detection methods is increasing. Furthermore, these methods suffer from a series of issues, including high labor and material costs, low accuracy and efficiency, and poor versatility.

[0004] To sum up, how to effectively solve the above-mentioned problems existing in traditional manual-based anomaly detection has become a technical problem that urgently needs to be solved in the field of network operation and maintenance technology. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to invent a system and method for time series anomaly detection and classification, which realizes time series anomaly detection in a semi-supervised manner and ensures the accuracy, efficiency and versatility of time series anomaly detection.

[0006] To achieve the above objectives, the present invention proposes a semi-supervised few-shot time series anomaly detection and classification system based on fine-tuning guided by a pre-trained model. The system includes the following modules:

[0007] Data input and preprocessing module: This module is used to preprocess a large amount of unlabeled time series data, a small number of labeled time series data, and time series data to be detected. The preprocessing includes but is not limited to the following operations: data cleaning, feature selection, data normalization, and sliding time window division.

[0008] The data cleaning operation refers to: deleting illegal values ​​from the original sequence;

[0009] The feature selection operation refers to: performing feature selection on the multidimensional time series after data cleaning, deleting sequences that are meaningless to the task in the multidimensional time series, including all-zero sequences and pure stationary sequences;

[0010] The data normalization operation refers to: performing zero-mean normalization on the time series data after feature selection, that is, subtracting the mean of each time series data from the time series data and dividing it by the standard deviation of the time series data;

[0011] The sliding time window division operation refers to: performing sliding time window division on the time series data to divide it into multiple fixed-size time windows w, each time window is regarded as a vector, and multiple vectors form a matrix;

[0012] Model pre-training module: The function of this module is to request unlabeled time series data from the data input and pre-processing module and instruct the model module to perform model pre-training. After the model pre-training of the model module is completed, the model pre-training module requests the storage module to save the pre-trained model, and the storage module returns whether the saving is successful.

[0013] The model pre-training includes two pre-training tasks, namely the masked time series modeling pre-training task MTM and the continuous time series inference pre-training task CTI;

[0014] The content of the masked time series modeling pre-training task MTM is: randomly masking a certain proportion of the time series data that has completed preprocessing. The training goal of the masked time series modeling pre-training task is to make the model output the masked time series data;

[0015] The content of the continuous time series inference pre-training task CTI is to train the model's ability to infer time series data, that is, to determine whether two time series segments are continuous segments from the same time series;

[0016] In the masked time series modeling pre-training task MTM, a certain proportion of time series data is randomly masked by constructing a mask matrix To achieve, Each element m ijRepresents a Boolean mask representing the jth time window w of the i-th time series ij Is it masked? If the time window w ij If it is not masked, then m ij =1, otherwise m ij =0;

[0017] In the continuous time series inference pre-training task CTI, a segmentation label matrix is ​​constructed To indicate whether time series data needs to be inferred, split the label matrix Each element of c ij , specifically defined as follows: c ij Represents the time window w ij Is the position of the continuous time series segmentation point, that is, the j-th time window w of the i-th time series ij Whether an inference task is needed; c ij =0 means no inference task is required; c ij =1 means that an inference task is required;

[0018] The total loss function L used by the model pre-training module during model pre-training is total The definition is as follows:

[0019] L total =αL MTM +(1-α)L CTI

[0020] In the above formula, α is the scale parameter, ranging from [0,1];

[0021] In the above formula, L MTM represents the loss function for the masked time series modeling pre-training task, L CTI Represents continuous time series inference

[0022] Loss function for the pre-training task;

[0023] L MTM The specific definitions are as follows:

[0024]

[0025] In the above formula, represents the embedding vector matrix of the unlabeled time series data in the pre-training dataset, ε i Represents the embedding vector matrix The i-th row vector in represents the i-th time window w of the unlabeled time series data i Embedding vector of; E[] represents the mathematical expectation, y ij Denotes the embedding vector εi The true value of the jth component in , Denotes the embedding vector ε i The mask reconstruction value of the jth component within ||ε i || represents the embedding vector ε i length;

[0026] L CTI The specific definitions are as follows:

[0027]

[0028] In the above formula, y^ is the probability that the model predicts that the sample is a positive sample; y is the sample label, which is 1 if the sample is a positive sample, otherwise it is 0; E[] represents the mathematical expectation.

[0029] Guidance fine-tuning module: The function of this module is to first request the storage module to read the pre-trained model, and the storage module returns the trained pre-trained model; then the guidance fine-tuning module requests the labeled time series data from the data input and preprocessing module, instructing the model module to perform guidance fine-tuning; after the guidance fine-tuning is completed, the guidance fine-tuning module requests the storage module to store the guidance vector, and the storage module returns whether the storage is successful;

[0030] Anomaly Detection Module: This module's function is to first request the time series data to be detected from the data input and preprocessing module; request the storage module to read the trained pre-trained model, and the storage module returns the trained pre-trained model; request the storage module to read the fine-tuned guidance vector, and the storage module returns the fine-tuned guidance vector; then the anomaly detection module integrates the trained pre-trained model and the fine-tuned guidance vector, instructing the model module to perform anomaly detection and classification tasks; after detection is completed, output the anomaly detection and classification results;

[0031] Model module: The functions of this module are: according to the instructions of the model pre-training module, perform the model pre-training operation; according to the instructions of the guidance fine-tuning module, perform the guidance fine-tuning operation; according to the instructions of the anomaly detection module, perform the anomaly detection operation; the model module is composed of an encoder network and a decoder network; the encoder network is composed of an embedded coding sublayer and N E The decoder network consists of an embedding coding sublayer and N D The decoder blocks are stacked; the embedding coding sublayer of the encoder network and the embedding coding sublayer of the decoder network have the same structure;

[0032] Storage module: This module is responsible for storing the pre-trained model and the fine-tuned guidance vector.

[0033] The function of the embedding coding sublayer is to generate an embedding matrix for the input data Each element of is e ij , represents the embedding matrix The i-th row vector ε i The jth component of ;

[0034]

[0035]

[0036] in, It is represented by the parameter θ emb The jth component of the i-th row vector of the embedding matrix generated by the convolutional neural network for the input data; Represents the input of the embedding coding sublayer. For the encoder network, Where ⊙ represents the masking operation on the time series data. represents time series data, represents the mask matrix and represents the segmentation label matrix, for the decoder network, in represents the output of the decoder network from time 1 to time t-1; p ij represents the position code of the jth component of the i-th time window, d model represents the dimension of the input data of the embedding coding sublayer, which is equal to the size of the time window of the input data |w|; d emb Indicates the dimension of the generated embedding vector.

[0037] The function of the encoder block of the encoder network is to learn a meaningful representation of each time series data and transform the time series data into The embedding matrix Encoded as a feature vector H in the hidden representation space en , so that the decoder network can perform MTM tasks and CTI tasks, and prepare for subsequent anomaly detection and classification tasks; the feature vector H en The formula is as follows:

[0038]

[0039] In the above formula, Encoders() represents the N of the encoder network EAn encoder block; the encoder block mainly consists of three parts: a multi-head multi-scale attention sublayer, a multi-scale attention fusion sublayer, and a feedforward fully connected sublayer; residual and normalization are performed after the multi-scale attention fusion sublayer and the feedforward fully connected sublayer;

[0040] The function of the decoder network is to perform the MTM task, CTI task and anomaly detection and classification task; the input of the decoder network is the embedding matrix generated by the embedding coding sublayer for the decoder network and the encoder network outputs the time series data The encoded feature vector H en , the output of the decoder network at time t for:

[0041]

[0042] In the above formula, Decoders() represents the N of the decoder network D A decoder block; the decoder block mainly consists of four parts: a masked multi-head multi-scale attention sublayer, a multi-scale attention fusion sublayer, a multi-head attention sublayer and a feedforward fully connected sublayer; residual and normalization are performed after the masked multi-head multi-scale attention sublayer, the multi-scale attention fusion sublayer and the feedforward fully connected sublayer;

[0043] The feedforward fully connected sublayer structure in the encoder block and the decoder block is exactly the same. Both use a fully connected linear neural network consisting of two linear transformations, with the ReLU function activated in the middle. The formula is as follows:

[0044]

[0045] Among them, x1 represents the input of the feedforward fully connected sublayer, b1 and b2 are learnable parameters;

[0046] The specific contents of the residual and normalization operations in the encoder block and decoder block are:

[0047] The residual connection feeds the input data x2 directly to the sub-layer module to prevent overfitting and learning failure caused by too many stacked module layers. Normalization uses layer normalization, which only normalizes the layer dimensions. The overall formula for residual and normalization is:

[0048] Res&Norm(x2)=LayerNorm(x2+Sublayer(x2))

[0049] Among them, Res&Norm(x2) represents the result of residual and normalization operations; LayerNorm() represents layer normalization; Sublayer(x2) represents the output of the sublayer module.

[0050] The multi-head multi-scale attention sublayer and the masked multi-head multi-scale attention sublayer have the same structure, both including Gaussian kernel attention units, multi-head interpretable guided attention units and graph attention units;

[0051] The multi-head multi-scale attention sublayer and the masked multi-head multi-scale attention sublayer both process the time series data at three scales: small, medium, and large. That is, the Gaussian kernel attention unit is used to process the small-scale time series data in the time series data window to obtain the result The results are obtained by using multi-head interpretable guided attention units to process the medium-scale time series data between time series data windows. Use the graph attention unit to process the large-scale time series data between time series data and obtain the result

[0052] The specific content of the Gaussian kernel attention unit processing the small-scale time series in the time series data window is:

[0053]

[0054] In the above formula, [α' ij ] represents a matrix with N rows and N columns, where N is equal to the size of the window |w|, and each element of the matrix is ​​α' ij ; exp() represents the exponential function; the element σ in the i-th row and j-th column of the scale variable matrix Σ ij , is a learnable scale variable corresponding to the jth component of the i-th window of the time series; the scale variable matrix Σ is implemented using a fully connected linear neural network, that is, Where W σ are the learnable parameters of the fully connected linear neural network, is the embedding matrix of the time series data output by the embedding coding sublayer;

[0055] The multi-head interpretable guided attention unit is composed of N h The single-head interpretable guided attention unit is composed of a single-head interpretable guided attention sub-unit and a pluggable and learnable attention guidance sub-unit;

[0056] The method of using a multi-head interpretable guided attention unit to process the mesoscale time series data between time series data windows refers to processing the mesoscale time series data between time series data windows, specifically:

[0057]

[0058] In the above formula, is an attention score matrix, each entry of which is a multi-head interpretable guidance attention score; N h is the total number of heads of the multi-head interpretable guided attention unit; W O is a learnable parameter for the final multi-head linear mapping; Q h represents the query matrix of the h-th head's single-head interpretable guided attention unit, K h represents the key matrix of the h-th head's single-head interpretable guidance attention unit; d k Represents the dimension of the hidden layer vector of the encoder block or decoder block, softmax() represents the softmax function, and [;] represents the matrix connection operation; The key matrix K of the single-head interpretable guidance attention unit with the h-th head is composed of the pluggable and learnable attention guidance subunit h The added key guidance matrix is ​​obtained using a fully connected linear neural network transformation;

[0059] The specific content of using the graph attention unit to process large-scale time series between time series is as follows: Graph attention can model the relationship between nodes in any graph. Each time series can be regarded as a node in the graph, and each edge represents the relationship between the features of two time series. In this way, the relationship between multiple time series is captured;

[0060] Processing results Is an attention matrix, the element β' in the i-th row and j-th column of the matrix ij Calculate according to the following formula:

[0061]

[0062] In the above formula, Indicates a connection operation. and W P is a learnable parameter, LeakyReLU is a nonlinear activation function; x i and x j They represent the i-th and j-th time series respectively, and n represents the number of time series;

[0063] The function of the multi-scale attention fusion sublayer is to fuse the features of the three scales processed by the multi-head multi-scale attention sublayer. The specific calculation method is as follows:

[0064] Calculate the fused attention score matrix according to the following formula

[0065]

[0066] In the above formula, LayerNorm() is the layer normalization operation;

[0067]

[0068] η1=ELU(W4γ1+W5γ2+W6γ3+b4)

[0069] η2=W7η1+b5

[0070] In the above formula, GLU I Represents the input gated linear unit, GLU, used to control the input O Represents an output-gated linear unit used to control the output:

[0071] GLU I (n)=softmax(W8n+b6)⊙(W9n+b7)

[0072] GLU O (η)=sigmoid(W 10 η+b8)⊙(W 11,ω η+b9)

[0073] ELU stands for Exponential Linear Unit; W1 to W 11 and b1 to b9 are learnable parameters;

[0074] The output X' of the multi-scale attention fusion sub-layer is calculated as follows:

[0075]

[0076] Where V represents the value matrix of the multi-head interpretable guided attention unit; [;] represents the matrix connection operation; φ V represents the value guidance matrix added by the pluggable and learnable attention guidance subunit to the value matrix V of the multi-head interpretable guidance attention unit;

[0077] The specific calculation process of the query matrix Q, key matrix K and value matrix V of the encoder block and decoder block of different layers is as follows:

[0078] The query matrix of the hth head of the first encoder block The key matrix of the h-th head A matrix of values ​​shared by all heads The embedding matrix output by the embedding coding sublayer of the encoder network It is obtained by the fully connected linear neural network transformation of the first layer encoder block, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the aforementioned first-layer encoder block;

[0079] The query matrix of the h-th head of the encoder block of the remaining layers The key matrix of the h-th head A matrix of values ​​shared by all heads The output data of the encoder block of the previous layer of the encoder block It is obtained by the fully connected linear neural network transformation of the encoder block of this layer, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the encoder block mentioned above;

[0080] The query matrix of the hth head of the first layer decoder block is The key matrix of the h-th head A matrix of values ​​shared by all heads The embedding matrix output by the embedding encoding sublayer of the decoder network It is obtained by the fully connected linear neural network transformation of the first layer decoder block, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the aforementioned first-layer decoder block;

[0081] The query matrix of the h-th head of the decoder block of the remaining layers The output data of the decoder block of the previous layer of the decoder block of this layer It is obtained by the fully connected linear neural network transformation of the decoder block of this layer, that is, In the above formula are the learning parameters of the fully connected linear neural network of the decoder block in this layer;

[0082] The key matrix of the h-th head of the decoder block of the remaining layers A matrix of values ​​shared by all heads The output data of the encoder network It is obtained by the fully connected linear neural network transformation of the decoder block of this layer, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the decoder block in this layer;

[0083] The key guidance matrix φ of each layer encoder block and decoder block is K Shared, the value guidance matrix φ of each layer encoder block and decoder block V Shared; φ K and φ V The specific calculation process is as follows:

[0084] φ K and φ V The key guidance matrix and the value guidance matrix added by the pluggable and learnable attention guidance sub-unit to the key matrix and the value guidance matrix in the encoder block and the decoder block respectively; represents the key guidance matrix added by the pluggable, learnable attention guidance subunit to the key matrix of the h-th head in the l-th layer encoder block and decoder block; represents the value guidance matrix added by the pluggable, learnable attention guidance subunit to the value matrices of all heads in the l-th layer encoder block and decoder block; is the guidance vector The length of is set to k times the length of the time series window, that is, k is a positive integer; d k is the dimension of the hidden layer vector of the encoder block or decoder block; the key guidance matrix Sum value guidance matrix It is obtained by using the fully connected linear neural network transformation, that is, In the above formula and are the learnable parameters of the fully connected linear neural network, X K_init and X V_init is a randomly initialized matrix.

[0085] The specific work content of the guidance fine-tuning module is:

[0086] After the model is pre-trained, the guidance vector needs to be combined with the pre-trained model Fine-tuning is performed to adapt to different scenarios and tasks; the guidance vector is a randomly initialized vector It is transformed by a fully connected linear neural network, that is, according to the following formula:

[0087]

[0088] Calculate, where is the learning parameter of the fully connected linear neural network; the specific process is:

[0089] Fix the parameters of the model module by adding the guidance vector to be learned in the input time series data And the abnormal type label of the time series, through fine-tuning training, obtain the prediction results that enable the model module to predict The optimal guidance vector that is closest to the true label Y Right now:

[0090]

[0091] In the above formula, Embedding matrix representing the input time series data with anomaly type labels from the guided fine-tuning dataset; Represents the prediction result of the model, and Y represents the true label of the input time series data;

[0092] The loss function used in training is the prediction result The cross entropy loss function L between the true label Y tuning :

[0093]

[0094] in, represents the embedding matrix of the input time series data with anomaly type labels from the guided fine-tuning dataset, E[] represents the mathematical expectation; the stochastic gradient descent algorithm is used for guided fine-tuning training.

[0095] The specific working content of the anomaly detection module is: inputting the data into the time series data to be detected after preprocessing by the preprocessing module The embedding matrix obtained by the embedding coding sublayer The trained pre-trained model parameters Θ and the fine-tuned guidance vector Input to the model module, instructing the model module to perform anomaly detection and classification tasks;

[0096] The output of the anomaly detection module is the label of anomaly detection and classification

[0097]

[0098] That is, the corresponding exception type.

[0099] The present invention also proposes a method for semi-supervised few-sample time series anomaly detection and classification based on fine-tuning guided by a pre-trained model, the method comprising the following steps:

[0100] (1) The data input and preprocessing module preprocesses a large amount of unlabeled time series data, a small number of labeled time series data, and time series data to be detected;

[0101] (2) the model pre-training module requests the unlabeled time series data from the data input and pre-processing module, and instructs the model module to perform model pre-training; after the model pre-training of the model module is completed, the model pre-training module requests the storage module to save the pre-trained pre-trained model, and the storage module returns whether the saving is successful;

[0102] (3) The guidance fine-tuning module requests the storage module to read the pre-trained pre-trained model, and the storage module returns the pre-trained model; then the guidance fine-tuning module requests the labeled time series data from the data input and preprocessing module, instructing the model module to perform guidance fine-tuning; after the guidance fine-tuning is completed, the guidance fine-tuning module requests the storage module to store the guidance vector, and the storage module returns whether the storage is successful;

[0103] (4) The anomaly detection module first requests the time series data to be detected from the data input and preprocessing module; requests the storage module to read the trained pre-trained model, and the storage module returns the trained pre-trained model; then requests the storage module to read the fine-tuned guidance vector, and the storage module returns the fine-tuned guidance vector; then the anomaly detection module integrates the trained pre-trained model and the fine-tuned guidance vector, and instructs the model module to perform anomaly detection and classification tasks; after the detection is completed, the results of anomaly detection and classification are output.

[0104] The beneficial effect of the present invention lies in that the system and method of the present invention utilize a large amount of unlabeled data for system pre-training and a small amount of labeled data for fine-tuning the guidance vector, thus avoiding the problem of extensive manual labeling and detection, thereby significantly saving manpower. By incorporating guidance fine-tuning, only the guidance vector, which takes up very little space, needs to be stored for different scenarios. Only one pre-trained model of the system needs to be stored, rather than multiple models that take up a lot of space. This significantly saves material resources and solves the problems of low accuracy, low efficiency, and poor versatility of existing time series anomaly detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 Schematic diagram of the module of the semi-supervised few-sample time series anomaly detection and classification system based on fine-tuning guided by pre-training model proposed in the present invention;

[0106] Figure 2 Schematic diagram of the network structure of the encoder block of the encoder network proposed by the present invention;

[0107] Figure 3 Schematic diagram of the network structure of the decoder block of the decoder network proposed in the present invention;

[0108] Figure 4This is a schematic diagram of the network structure of the multi-head multi-scale attention sub-layer proposed in this invention;

[0109] Figure 5 is a schematic diagram of a graph attention unit according to an embodiment of the present invention;

[0110] Figure 6 2. It is a schematic diagram of the process of the multi-scale attention fusion sublayer fusing and calculating the features of three scales in an embodiment of the present invention;

[0111] Figure 7 This is a schematic diagram of the working process of the guiding fine-tuning module in an embodiment of the present invention;

[0112] Figure 8 This is a workflow diagram of the method for semi-supervised few-sample time series anomaly detection and classification based on fine-tuning guided by a pre-trained model proposed in the present invention;

[0113] Figure 9 This is a flowchart of the steps of the method for semi-supervised few-sample time series anomaly detection and classification based on fine-tuning guided by a pre-trained model proposed in the present invention; DETAILED DESCRIPTION

[0114] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0115] See also Figure 1 This paper introduces a semi-supervised few-shot time series anomaly detection and classification system based on fine-tuning guided by a pre-trained model. The system includes the following modules:

[0116] Data input and preprocessing module: This module is used to preprocess a large amount of unlabeled time series data, a small number of labeled time series data, and time series data to be detected. The preprocessing includes but is not limited to the following operations: data cleaning, feature selection, data normalization, and sliding time window division.

[0117] Table 1 shows an example of a few-sample labeled multidimensional time series data.

[0118] Table 1

[0119]

[0120]

[0121] The data cleaning operation refers to: for each value x of the input original time series ij Perform data cleaning and delete illegal data. For example, for the "CPU usage" indicator, any x ij >100 and x ijValues ​​<0 are illegal data.

[0122] The feature selection operation refers to: performing feature selection on the multidimensional time series after data cleaning, deleting sequences that are meaningless to the task in the multidimensional time series, including all-zero sequences and pure stationary sequences;

[0123] The data normalization operation refers to: performing zero-mean normalization on the time series data after feature selection, that is, subtracting the mean of each time series data from the time series data and dividing it by the standard deviation of the time series data;

[0124] The sliding time window division operation refers to: performing sliding time window division on the time series data to divide it into multiple fixed-size time windows w, each time window is regarded as a vector, and multiple vectors form a matrix;

[0125] In an embodiment, the following sliding time window division operation is adopted:

[0126]

[0127] Here, s represents the sliding window step size, which can be set according to actual conditions, for example, s=1, the sliding window length |w|=j2-j1, the size is fixed, and can be set according to actual conditions, for example, |w|=3.

[0128] Model pre-training module: The function of this module is to request unlabeled time series data from the data input and pre-processing module and instruct the model module to perform model pre-training. After the model pre-training of the model module is completed, the model pre-training module requests the storage module to save the pre-trained model, and the storage module returns whether the saving is successful.

[0129] The model pre-training includes two pre-training tasks, namely the masked time series modeling pre-training task MTM and the continuous time series inference pre-training task CTI;

[0130] The content of the masked time series modeling pre-training task MTM is: randomly masking a certain proportion of the time series data that has completed preprocessing. The training goal of the masked time series modeling pre-training task is to make the model output the masked time series data;

[0131] The content of the continuous time series inference pre-training task CTI is to train the model's ability to infer time series data, that is, to determine whether two time series segments are continuous segments from the same time series;

[0132] In the masked time series modeling pre-training task MTM, a certain proportion (e.g. 30%) of the time series data is randomly masked by constructing a mask matrix To achieve, Each element m ij Represents a Boolean mask representing the time window w ij Whether (the jth time window of the i-th time series) is masked, if the time window w ij If it is not masked, then m ij =1, otherwise m ij =0;

[0133] Table 2 shows a mask matrix Instance of .

[0134] Table 2

[0135] 1 0 1 … 1 1 1 … 0 0 1 … … … … …

[0136] In the continuous time series inference pre-training task CTI, a segmentation label matrix is ​​constructed To indicate whether time series data needs to be inferred; split label matrix Each element of c ij , specifically defined as follows: c ij Represents the time window w ij Is the position of the continuous time series segmentation point, that is, the j-th time window w of the i-th time series ij Whether an inference task is needed; c ij =0 means no inference task is required; c ij =1 means that an inference task is required;

[0137] Table 3 shows a segmentation label matrix Instance of .

[0138] Table 3

[0139] 0 1 0 0 0 0 … 0 0 0 0 1 0 … 0 0 1 0 0 0 … … … … … … … …

[0140] The total loss function L used by the model pre-training module during model pre-training is total The definition is as follows:

[0141] L total =αL MTM +(1-α)L CTI

[0142] In the above formula, α is a proportional parameter, ranging from [0, 1]. In the embodiment, α is 0.5. MTM represents the loss function for the masked time series modeling pre-training task, L CTI represents the loss function for the continuous time series inference pre-training task;

[0143] LMTM The specific definitions are as follows:

[0144]

[0145] In the above formula, represents the embedding vector matrix of the unlabeled time series data in the pre-training dataset, ε i Represents the embedding vector matrix The i-th row vector in represents the i-th time window w of the unlabeled time series data i Embedding vector of; E[] represents the mathematical expectation, y ij Denotes the embedding vector ε i The true value of the jth component in , Denotes the embedding vector ε i The mask reconstruction value of the jth component within ||ε i || represents the embedding vector ε i length;

[0146] L CTI The specific definitions are as follows:

[0147]

[0148] In the above formula, y^ is the probability that the model predicts that the sample is a positive sample; y is the sample label, which is 1 if the sample is a positive sample, otherwise it is 0; E[] represents the mathematical expectation.

[0149] Guidance fine-tuning module: The function of this module is to first request the storage module to read the pre-trained model, and the storage module returns the trained pre-trained model; then the guidance fine-tuning module requests the labeled time series data from the data input and preprocessing module, instructing the model module to perform guidance fine-tuning; after the guidance fine-tuning is completed, the guidance fine-tuning module requests the storage module to store the guidance vector, and the storage module returns whether the storage is successful;

[0150] Anomaly Detection Module: This module's function is to first request the time series data to be detected from the data input and preprocessing module; request the storage module to read the trained pre-trained model, and the storage module returns the trained pre-trained model; request the storage module to read the fine-tuned guidance vector, and the storage module returns the fine-tuned guidance vector; then the anomaly detection module integrates the trained pre-trained model and the fine-tuned guidance vector, instructing the model module to perform anomaly detection and classification tasks; after detection is completed, output the anomaly detection and classification results;

[0151] Model module: The functions of this module are: according to the instructions of the model pre-training module, perform the model pre-training operation; according to the instructions of the guidance fine-tuning module, perform the guidance fine-tuning operation; according to the instructions of the anomaly detection module, perform the anomaly detection operation; the model module is composed of an encoder network and a decoder network; the encoder network is composed of an embedded coding sublayer and N E (In the embodiment, N E The decoder network consists of an embedded coding sublayer and N D (In the embodiment, N D The value is 16) decoder blocks are stacked; the embedded coding sublayer of the encoder network and the embedded coding sublayer of the decoder network have the same structure;

[0152] Storage module: This module is responsible for storing the pre-trained model and the fine-tuned guidance vector.

[0153] The function of the embedding coding sublayer is to generate an embedding matrix for the input data Each element of is e ij , represents the embedding matrix The i-th row vector ε i The jth component of ;

[0154]

[0155] in, It is represented by the parameter θ emb The jth component of the i-th row vector of the embedding matrix generated by the convolutional neural network for the input data; Represents the input of the embedding coding sublayer. For the encoder network, Where ⊙ represents the masking operation on the time series data. represents time series data, represents the mask matrix and represents the segmentation label matrix, for the decoder network, in represents the output of the decoder network from time 1 to time t-1; p ij represents the position code of the jth component of the i-th time window, d model represents the dimension of the input data of the embedding coding sublayer, which is equal to the size of the time window of the input data |w|; d emb Indicates the dimension of the generated embedding vector.

[0156] In an embodiment, the structure of the embedded coding sublayer is consistent with that of the embedded coding sublayer in Transformer (see reference: Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need [J]. arXiv, 2017).

[0157] The function of the encoder block of the encoder network is to learn a meaningful representation of each time series data and transform the time series data into The embedding matrix Encoded as a feature vector H in the hidden representation space en , so that the decoder network can perform MTM tasks and CTI tasks, and prepare for subsequent anomaly detection and classification tasks; the feature vector H en The formula is as follows:

[0158]

[0159] In the above formula, Encoders() represents the N of the encoder network E encoder blocks;

[0160] See also Figure 2 The encoder block mainly consists of three parts: a multi-head multi-scale attention sublayer, a multi-scale attention fusion sublayer, and a feed-forward fully connected sublayer; residual and normalization are performed after the multi-scale attention fusion sublayer and the feed-forward fully connected sublayer;

[0161] The function of the decoder network is to perform the MTM task, CTI task and anomaly detection and classification task; the input of the decoder network is the embedding matrix generated by the embedding coding sublayer for the decoder network and the encoder network outputs the time series data The encoded feature vector H en , the output of the decoder network at time t for:

[0162]

[0163] In the above formula, Decoders() represents the N of the decoder network D decoder blocks;

[0164] See also Figure 3 The decoder block mainly consists of four parts: a masked multi-head multi-scale attention sublayer, a multi-scale attention fusion sublayer, a multi-head attention sublayer and a feedforward fully connected sublayer; residual and normalization are performed after the masked multi-head multi-scale attention sublayer, the multi-scale attention fusion sublayer and the feedforward fully connected sublayer;

[0165] The feedforward fully connected sublayer structure in the encoder block and the decoder block is exactly the same. Both use a fully connected linear neural network consisting of two linear transformations, with the ReLU function activated in the middle. The formula is as follows:

[0166]

[0167] Among them, x1 represents the input of the feedforward fully connected sublayer, b1 and b2 are learnable parameters;

[0168] The specific contents of the residual and normalization operations in the encoder block and decoder block are:

[0169] The residual connection feeds the input data x2 directly to the sub-layer module to prevent overfitting and learning failure caused by too many stacked module layers. Normalization uses layer normalization, which only normalizes the layer dimensions. The overall formula for residual and normalization is:

[0170] Res&Norm(x2)=LayerNorm(x2+Sublayer(x2))

[0171] Among them, Res&Norm(x2) represents the result of residual and normalization operations; LayerNorm() represents layer normalization; Sublayer(x2) represents the output of the sublayer module.

[0172] See also Figure 4 The multi-head multi-scale attention sublayer and the masked multi-head multi-scale attention sublayer have the same structure, both including Gaussian kernel attention units, multi-head interpretable guided attention units and graph attention units;

[0173] In an embodiment, the attention calculation network in the multi-head interpretable guided attention unit draws on the idea of ​​the attention calculation network in the Transformer; the mask calculation process in the masked multi-head multi-scale attention sublayer adopts a calculation process consistent with that in the Transformer.

[0174] The multi-head multi-scale attention sublayer and the masked multi-head multi-scale attention sublayer both process the time series data at three scales: small, medium, and large. That is, the Gaussian kernel attention unit is used to process the small-scale time series data in the time series data window to obtain the result The results are obtained by using multi-head interpretable guided attention units to process the medium-scale time series data between time series data windows. Use the graph attention unit to process the large-scale time series data between time series data and obtain the result

[0175] The specific content of the Gaussian kernel attention unit processing the small-scale time series in the time series data window is:

[0176]

[0177] In the above formula, [α' ij ] represents a matrix with N rows and N columns, where N is equal to the size of the window |w|, and each element of the matrix is ​​α' ij ; exp() represents the exponential function; the element σ in the i-th row and j-th column of the scale variable matrix Σ ij , is a learnable scale variable corresponding to the jth component of the i-th window of the time series; the scale variable matrix Σ is implemented using a fully connected linear neural network, that is, Where W σ are the learnable parameters of the fully connected linear neural network, is the embedding matrix of the time series data output by the embedding coding sublayer;

[0178] The multi-head interpretable guided attention unit is composed of N h (In the embodiment, the value is 12) single-head interpretable guided attention units; the single-head interpretable guided attention unit is composed of a single-head interpretable attention sub-unit and a pluggable and learnable attention guidance sub-unit;

[0179] The method of using a multi-head interpretable guided attention unit to process the mesoscale time series data between time series data windows refers to processing the mesoscale time series data between time series data windows, specifically:

[0180]

[0181] In the above formula, is an attention score matrix, each entry of which is a multi-head interpretable guidance attention score; N h is the total number of heads of the multi-head interpretable guided attention unit; W O is a learnable parameter for the final multi-head linear mapping; Q h represents the query matrix of the h-th head's single-head interpretable guided attention unit, K h represents the key matrix of the h-th head's single-head interpretable guidance attention unit; d k Represents the dimension of the hidden layer vector of the encoder block or decoder block, softmax() represents the softmax function, and [;] represents the matrix connection operation; The key matrix K of the single-head interpretable guidance attention unit with the h-th head is composed of the pluggable and learnable attention guidance subunit h The added key guidance matrix is ​​obtained using a fully connected linear neural network transformation; the key guidance matrix The addition of is different from the calculation process of the attention calculation network in Transformer.

[0182] See also Figure 5 The specific content of using the graph attention unit to process large-scale time series between time series is as follows: Graph attention can model the relationship between nodes in any graph. Each time series can be regarded as a node in the graph, and each edge represents the relationship between the features of two time series. In this way, the relationship between multiple time series can be captured;

[0183] Processing results Is an attention matrix, the element β' in the i-th row and j-th column of the matrix ij Calculate according to the following formula:

[0184]

[0185] In the above formula, Indicates a connection operation. and W P is a learnable parameter, LeakyReLU is a nonlinear activation function; x i and x j They represent the i-th and j-th time series respectively, and n represents the number of time series;

[0186] See also Figure 6 The function of the multi-scale attention fusion sublayer is to fuse the features of the three scales processed by the multi-head multi-scale attention sublayer. The specific calculation method is as follows;

[0187] Calculate the fused attention score matrix according to the following formula

[0188]

[0189] In the above formula, LayerNorm() is the layer normalization operation;

[0190]

[0191] η1=ELU(W4γ1+W5γ2+W6γ3+b4)

[0192] η2=W7η1+b5

[0193] In the above formula, GLU IRepresents the input gated linear unit, GLU, used to control the input O Represents an output-gated linear unit used to control the output:

[0194] GLU I (n)=softmax(W8n+b6)⊙(W9n+b7)

[0195] GLU O (η)=sigmoid(W 10 η+b8)⊙(W 11,ω η+b9)

[0196] ELU stands for Exponential Linear Unit; W1 to W 11 and b1 to b9 are learnable parameters;

[0197] The output X' of the multi-scale attention fusion sub-layer is calculated as follows:

[0198]

[0199] Where V represents the value matrix of the multi-head interpretable guided attention unit; [;] represents the matrix connection operation; φ V represents the value guidance matrix added by the pluggable and learnable attention guidance subunit to the value matrix V of the multi-head interpretable guidance attention unit;

[0200] The specific calculation process of the query matrix Q, key matrix K and value matrix V of the encoder block and decoder block of different layers is as follows:

[0201] The query matrix of the hth head of the first encoder block The key matrix of the h-th head A matrix of values ​​shared by all heads The embedding matrix output by the embedding coding sublayer of the encoder network It is obtained by the fully connected linear neural network transformation of the first layer encoder block, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the aforementioned first-layer encoder block;

[0202] The query matrix of the h-th head of the encoder block of the remaining layers The key matrix of the h-th head A matrix of values ​​shared by all heads The output data of the encoder block of the previous layer of the encoder block It is obtained by the fully connected linear neural network transformation of the encoder block of this layer, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the encoder block mentioned above;

[0203] The query matrix of the hth head of the first layer decoder block is The key matrix of the h-th head A matrix of values ​​shared by all heads The embedding matrix output by the embedding encoding sublayer of the decoder network It is obtained by the fully connected linear neural network transformation of the first layer decoder block, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the aforementioned first-layer decoder block;

[0204] The query matrix of the h-th head of the decoder block of the remaining layers The output data of the decoder block of the previous layer of the decoder block of this layer It is obtained by the fully connected linear neural network transformation of the decoder block of this layer, that is, In the above formula are the learning parameters of the fully connected linear neural network of the decoder block in this layer;

[0205] The key matrix of the h-th head of the decoder block of the remaining layers A matrix of values ​​shared by all heads The output data of the encoder network It is obtained by the fully connected linear neural network transformation of the decoder block of this layer, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the decoder block in this layer;

[0206] The key guidance matrix φ of each layer encoder block and decoder block is K Shared, the value guidance matrix φ of each layer encoder block and decoder block V Shared; φ K and φ V The specific calculation process is as follows:

[0207] φ K and φ V The key guidance matrix and the value guidance matrix added by the pluggable and learnable attention guidance sub-unit to the key matrix and the value guidance matrix in the encoder block and the decoder block respectively; represents the key guidance matrix added by the pluggable, learnable attention guidance subunit to the key matrix of the h-th head in the l-th layer encoder block and decoder block; represents the value guidance matrix added by the pluggable, learnable attention guidance subunit to the value matrices of all heads in the l-th layer encoder block and decoder block; is the guidance vector The length of is set to k times the length of the time series window, that is, k is a positive integer, and in the embodiment, the value is 5 to 8; d k is the dimension of the hidden layer vector of the encoder block or decoder block; the key guidance matrix Sum value guidance matrix It is obtained by using the fully connected linear neural network transformation, that is, In the above formula and are the learnable parameters of the fully connected linear neural network, X K_init and X V_init is a randomly initialized matrix.

[0208] See also Figure 7 The specific work content of the guidance fine-tuning module is:

[0209] After the model is pre-trained, the guidance vector needs to be combined with the pre-trained model Fine-tuning is performed to adapt to different scenarios and tasks; the guidance vector is a randomly initialized vector It is transformed by a fully connected linear neural network, that is, according to the following formula:

[0210]

[0211] Calculate, where is the learning parameter of the fully connected linear neural network; the specific process is:

[0212] Fix the parameters of the model module by adding the guidance vector to be learned in the input time series data And the abnormal type label of the time series, through fine-tuning training, obtain the prediction results that enable the model module to predict The optimal guidance vector that is closest to the true label Y Right now:

[0213]

[0214] In the above formula, Embedding matrix representing the input time series data with anomaly type labels from the guided fine-tuning dataset; Represents the prediction result of the model, and Y represents the true label of the input time series data;

[0215] The loss function used in training is the prediction result The cross entropy loss function L between the true label Y tuning :

[0216]

[0217] in, represents the embedding matrix of the input time series data with anomaly type labels from the guided fine-tuning dataset, E[] represents the mathematical expectation; the stochastic gradient descent algorithm is used for guided fine-tuning training.

[0218] The specific working content of the anomaly detection module is: inputting the data into the time series data to be detected after preprocessing by the preprocessing module The embedding matrix obtained by the embedding coding sublayer The trained pre-trained model parameters Θ and the fine-tuned guidance vector Input to the model module, instructing the model module to perform anomaly detection and classification tasks;

[0219] The output of the anomaly detection module is the label of anomaly detection and classification

[0220]

[0221] That is, the corresponding exception type.

[0222] See also Figure 8 and Figure 9 The present invention introduces a method for semi-supervised few-shot time series anomaly detection and classification based on fine-tuning guided by a pre-trained model. The method includes the following steps:

[0223] (1) The data input and preprocessing module preprocesses a large amount of unlabeled time series data, a small number of labeled time series data, and time series data to be detected;

[0224] (2) the model pre-training module requests the unlabeled time series data from the data input and pre-processing module, and instructs the model module to perform model pre-training; after the model pre-training of the model module is completed, the model pre-training module requests the storage module to save the pre-trained pre-trained model, and the storage module returns whether the saving is successful;

[0225] (3) The guidance fine-tuning module requests the storage module to read the pre-trained pre-trained model, and the storage module returns the pre-trained model; then the guidance fine-tuning module requests the labeled time series data from the data input and preprocessing module, instructing the model module to perform guidance fine-tuning; after the guidance fine-tuning is completed, the guidance fine-tuning module requests the storage module to store the guidance vector, and the storage module returns whether the storage is successful;

[0226] (4) The anomaly detection module first requests the time series data to be detected from the data input and preprocessing module; requests the storage module to read the trained pre-trained model, and the storage module returns the trained pre-trained model; then requests the storage module to read the fine-tuned guidance vector, and the storage module returns the fine-tuned guidance vector; then the anomaly detection module integrates the trained pre-trained model and the fine-tuned guidance vector, and instructs the model module to perform anomaly detection and classification tasks; after the detection is completed, the results of anomaly detection and classification are output.

[0227] The inventors conducted a large number of experiments on the system and method proposed in the present invention. In the experiments, they first selected datasets of multiple scenes, such as the Mars exploration datasets SMAP (Soil Moisture Active Passive satellite) and MSL (Mars Science Laboratory rover) ( https: / / nsidc.org / data / smap / data , https: / / github.com / khundman / telemanom, https: / / s3-us-west-2.amazonaws.com / telemanom / data.zip), the Secure Water Treatment (SWaT) water resource anomaly detection dataset (https: / / itrust.sutd.edu.sg / itrust-labs_datasets / dataset_info), the SMD (Server Machine Dataset) dataset for real server scenarios (https: / / github.com / NetManAIOps / OmniAnomaly / tree / master / ServerMachineDataset), and the MSDS (Multi-Source Distributed System Data for AI-powered Analytics) dataset of multivariate time series indicator data generated by distributed systems (https: / / zenodo.org / record / 3549604#.Y7pYGS-KG3w). In addition, there are some specific scenario data sets, such as e-commerce, banking data sets and home broadband data sets, etc. The selected data sets are pre-processed; then the model is pre-trained to obtain a pre-trained model. Experiments have shown that the model has achieved excellent results in two pre-training tasks; using the pre-trained model, combined with part of the labeled data in the data set, the model is guided and fine-tuned to obtain a streamlined guidance vector; finally, the guidance vector is used to detect and classify anomalies in the data set. Compared with the answer label, more than 96% of the test set anomaly detections are correct, and more than 90% of the test set anomaly classifications are correct. Overall, good experimental results are obtained, indicating that the system and method proposed in the present invention are effective and feasible.

Claims

1. A semi-supervised few-shot time series anomaly detection and classification system based on fine-tuning guided by a pre-trained model, characterized by: The system includes the following modules: Data input and preprocessing module: This module is used to preprocess large amounts of unlabeled time series data, small amounts of labeled time series data, and time series data to be tested. The preprocessing includes but is not limited to the following operations: data cleaning, feature selection, data normalization, and sliding time window partitioning. The time series data includes CPU usage, TCP handshake latency, and memory usage. The data cleaning operation refers to: deleting illegal values ​​from the original sequence; The feature selection operation refers to: performing feature selection on the multidimensional time series after data cleaning, deleting sequences that are meaningless to the task in the multidimensional time series, including all-zero sequences and pure stationary sequences; The data normalization operation refers to: performing zero-mean normalization on the time series data after feature selection, that is, subtracting the mean of the time series data after feature selection and dividing it by the standard deviation of the time series data; The sliding time window division operation refers to: performing sliding time window division on the time series data to divide it into multiple fixed-size time windows w, each time window is regarded as a vector, and multiple vectors form a matrix; Model pre-training module: The function of this module is to request unlabeled time series data from the data input and pre-processing module and instruct the model module to perform model pre-training; After the model pre-training of the model module is completed, the model pre-training module requests the storage module to save the pre-trained model, and the storage module returns whether the saving is successful; The model pre-training includes two pre-training tasks, namely the masked time series modeling pre-training task MTM and the continuous time series inference pre-training task CTI; The content of the masked time series modeling pre-training task MTM is: randomly masking a certain proportion of the time series data that has completed preprocessing. The training goal of the masked time series modeling pre-training task is to make the model output the masked time series data; The content of the continuous time series inference pre-training task CTI is to train the model's ability to infer time series data, that is, to determine whether two time series segments are continuous segments from the same time series; In the masked time series modeling pre-training task MTM, a certain proportion of time series data is randomly masked by constructing a mask matrix To achieve, Each element m ij Represents a Boolean mask representing the jth time window w of the i-th time series ij Is it masked? If the time window w ij If it is not masked, then m ij =1, otherwise m ij =0; In the continuous time series inference pre-training task CTI, a segmentation label matrix is ​​constructed To indicate whether time series data needs to be inferred, split the label matrix Each element of c ij , specifically defined as follows: c ij Represents the time window w ij Is the position of the continuous time series segmentation point, that is, the j-th time window w of the i-th time series ij Whether an inference task is needed; c ij =0 means no inference task is required; c ij =1 means that an inference task is required; The total loss function L used by the model pre-training module during model pre-training is total The definition is as follows: L total =αL MTM +(1-α)L CTI In the above formula, α is the scale parameter, ranging from [0,1]; In the above formula, L MTM represents the loss function for the masked time series modeling pre-training task, L CTI represents the loss function for the continuous time series inference pre-training task; L MTM The specific definitions are as follows: In the above formula, represents the embedding vector matrix of the unlabeled time series data in the pre-training dataset, ε i Represents the embedding vector matrix The i-th row vector in represents the i-th time window w of the unlabeled time series data i Embedding vector of ; E[] represents the mathematical expectation, y ij Denotes the embedding vector ε i The true value of the jth component in , Denotes the embedding vector ε i The mask reconstruction value of the jth component within ||ε i || represents the embedding vector ε i length; L CTI The specific definitions are as follows: In the above formula, y^ is the probability that the model predicts that the sample is a positive sample; y is the sample label, which is 1 if the sample is a positive sample, otherwise it is 0; E[] represents the mathematical expectation; Guidance fine-tuning module: The function of this module is to first request the storage module to read the pre-trained model, and the storage module returns the trained pre-trained model; then the guidance fine-tuning module requests the labeled time series data from the data input and preprocessing module, instructing the model module to perform guidance fine-tuning; after the guidance fine-tuning is completed, the guidance fine-tuning module requests the storage module to store the guidance vector, and the storage module returns whether the storage is successful; Anomaly detection module: The function of this module is to first request the time series data to be detected from the data input and preprocessing module; Requesting the storage module to read the trained pre-trained model, the storage module returns the trained pre-trained model; requesting the storage module to read the fine-tuned guidance vector, the storage module returns the fine-tuned guidance vector; then the anomaly detection module fuses the trained pre-trained model with the fine-tuned guidance vector and instructs the model module to perform anomaly detection and classification tasks; After the detection is completed, the results of anomaly detection and classification are output; Model module: The functions of this module are: according to the instructions of the model pre-training module, perform the model pre-training operation; according to the instructions of the guidance fine-tuning module, perform the guidance fine-tuning operation; according to the instructions of the anomaly detection module, perform the anomaly detection operation; the model module is composed of an encoder network and a decoder network; the encoder network is composed of an embedded coding sublayer and N E The decoder network consists of an embedding coding sublayer and N D The decoder blocks are stacked; the embedding coding sublayer of the encoder network and the embedding coding sublayer of the decoder network have the same structure; Storage module: This module is responsible for storing the pre-trained model and the fine-tuned guidance vector; The function of the encoder block of the encoder network is to learn a meaningful representation of each time series data and transform the time series data into The embedding matrix Encoded as a feature vector H in the hidden representation space en , so that the decoder network can perform MTM tasks and CTI tasks, and prepare for subsequent anomaly detection and classification tasks; the feature vector H en The formula is as follows: In the above formula, Encoders() represents the N of the encoder network E An encoder block; the encoder block mainly consists of three parts: a multi-head multi-scale attention sublayer, a multi-scale attention fusion sublayer, and a feedforward fully connected sublayer; residual and normalization are performed after the multi-scale attention fusion sublayer and the feedforward fully connected sublayer; The function of the decoder network is to perform the MTM task, CTI task and anomaly detection and classification task; the input of the decoder network is the embedding matrix generated by the embedding coding sublayer for the decoder network and the encoder network outputs the time series data The encoded feature vector H en , the output of the decoder network at time t for: In the above formula, Decoders() represents the N of the decoder network D A decoder block; the decoder block mainly consists of four parts: a masked multi-head multi-scale attention sublayer, a multi-scale attention fusion sublayer, a multi-head attention sublayer and a feedforward fully connected sublayer; residual and normalization are performed after the masked multi-head multi-scale attention sublayer, the multi-scale attention fusion sublayer and the feedforward fully connected sublayer; The feedforward fully connected sublayer structure in the encoder block and the decoder block is exactly the same. Both use a fully connected linear neural network consisting of two linear transformations, with the ReLU function activated in the middle. The formula is as follows: Among them, x1 represents the input of the feedforward fully connected sublayer, b1 and b2 are learnable parameters; The specific contents of the residual and normalization operations in the encoder block and decoder block are: The residual connection feeds the input data x2 directly to the sub-layer module to prevent overfitting and learning failure caused by too many stacked module layers. Normalization uses layer normalization, which only normalizes the layer dimensions. The overall formula for residual and normalization is: Res&Norm(x2)=LayerNorm(x2+Sublayer(x2)) Among them, Res&Norm(x2) represents the result of residual and normalization operation; LayerNorm() represents layer normalization; Sublayer(x2) represents the output of sublayer module; The multi-head multi-scale attention sublayer and the masked multi-head multi-scale attention sublayer have the same structure, both including Gaussian kernel attention units, multi-head interpretable guided attention units and graph attention units; The multi-head multi-scale attention sublayer and the masked multi-head multi-scale attention sublayer both process the time series data at three scales: small, medium, and large. That is, the Gaussian kernel attention unit is used to process the small-scale time series data in the time series data window to obtain the result The results are obtained by using multi-head interpretable guided attention units to process the medium-scale time series data between time series data windows. Use the graph attention unit to process the large-scale time series data between time series data and obtain the result The specific content of the Gaussian kernel attention unit processing the small-scale time series in the time series data window is: In the above formula, [α' ij ] represents a matrix with N rows and N columns, where N is equal to the size of the window |w|, and each element of the matrix is ​​α' ij ; exp() represents the exponential function; the element σ in the i-th row and j-th column of the scale variable matrix Σ ij , is a learnable scale variable corresponding to the jth component of the i-th window of the time series; the scale variable matrix Σ is implemented using a fully connected linear neural network, that is, Where W σ are the learnable parameters of the fully connected linear neural network, is the embedding matrix of the time series data output by the embedding coding sublayer; The multi-head interpretable guided attention unit is composed of N h The single-head interpretable guided attention unit is composed of a single-head interpretable guided attention sub-unit and a pluggable and learnable attention guidance sub-unit; The method of using a multi-head interpretable guided attention unit to process the mesoscale time series data between time series data windows refers to processing the mesoscale time series data between time series data windows, specifically: In the above formula, is an attention score matrix, each entry of which is a multi-head interpretable guidance attention score; N h is the total number of heads of the multi-head interpretable guided attention unit; W O is a learnable parameter for the final multi-head linear mapping; Q h represents the query matrix of the h-th head’s single-head interpretable guided attention unit, K h represents the key matrix of the h-th head’s single-head interpretable guided attention unit; d k Represents the dimension of the hidden layer vector of the encoder block or decoder block, softmax() represents the softmax function, and [;] represents the matrix connection operation; is the key matrix K of the single-head interpretable guided attention unit with the h-th head, which is composed of the pluggable and learnable attention guidance subunit h The added key guidance matrix is ​​obtained using a fully connected linear neural network transformation; The specific content of using the graph attention unit to process large-scale time series between time series is as follows: Graph attention can model the relationship between nodes in any graph, each time series is regarded as a node in the graph, and each edge represents the relationship between the features of two time series; in this way, the relationship between multiple time series is captured; Processing results Is an attention matrix, the element β' in the i-th row and j-th column of the matrix ij Calculate according to the following formula: In the above formula, Indicates a connection operation. and W P is a learnable parameter, LeakyReLU is a nonlinear activation function; x i and x j They represent the i-th and j-th time series respectively, and n represents the number of time series; The function of the multi-scale attention fusion sublayer is to fuse the features of the three scales processed by the multi-head multi-scale attention sublayer. The specific calculation method is as follows: Calculate the fused attention score matrix according to the following formula In the above formula, LayerNorm() is the layer normalization operation; η1=ELU(W4γ1+W5γ2+W6γ3+b4) η2=W7η1+b5 In the above formula, GLU I Represents the input gated linear unit, GLU, used to control the input O Represents an output-gated linear unit used to control the output: GLU I (η)=softmax(W8η+b6)⊙(W9η+b7) GLU O (η)=sigmoid(W 10 η+b8)⊙(W 11,ω (the+b9) ELU stands for Exponential Linear Unit; W1 to W 11 and b1 to b9 are learnable parameters; The output X' of the multi-scale attention fusion sub-layer is calculated as follows: Where V represents the value matrix of the multi-head interpretable guided attention unit; [;] represents the matrix connection operation; φ V Represents the value guidance matrix added by the pluggable and learnable attention guidance sub-unit to the value matrix V of the multi-head interpretable guidance attention unit.

2. The system for semi-supervised few-shot time series anomaly detection and classification based on pre-trained model-guided fine-tuning according to claim 1, characterized in that: The function of the embedding coding sublayer is to generate an embedding matrix for the input data Each element of is e ij , represents the embedding matrix The i-th row vector ε i The jth component of ; in, Represented by the parameter θ emb The jth component of the i-th row vector of the embedding matrix generated by the convolutional neural network for the input data; Represents the input of the embedding coding sublayer. For the encoder network, Where ⊙ represents the mask operation on time series data. represents time series data, represents the mask matrix and represents the segmentation label matrix, for the decoder network, in represents the output of the decoder network from time 1 to time t-1; p ij represents the positional encoding of the jth component of the i-th time window, d model represents the dimension of the input data of the embedding coding sublayer, which is equal to the size of the time window of the input data |w|; d emb Indicates the dimension of the generated embedding vector.

3. The semi-supervised few-shot time series anomaly detection and classification system based on pre-trained model-guided fine-tuning according to claim 1, characterized in that: The specific work content of the guidance fine-tuning module is: After the model is pre-trained, the guidance vector needs to be combined with the pre-trained model Fine-tuning is performed to adapt to different scenarios and tasks; the guidance vector is a randomly initialized vector It is transformed by a fully connected linear neural network, that is, according to the following formula: Calculate, where is the learning parameter of the fully connected linear neural network; the specific process is: Fix the parameters of the model module by adding the guidance vector to be learned in the input time series data And the abnormal type label of the time series, through fine-tuning training, obtain the prediction results that enable the model module to predict The optimal guidance vector that is closest to the true label Y Right now: In the above formula, Embedding matrix representing the input time series data with anomaly type labels from the guided fine-tuning dataset; Represents the prediction result of the model, and Y represents the true label of the input time series data; The loss function used in training is the prediction result The cross entropy loss function L between the true label Y tuning : in, represents the embedding matrix of the input time series data with anomaly type labels from the guided fine-tuning dataset, E[] represents the mathematical expectation; the stochastic gradient descent algorithm is used for guided fine-tuning training.

4. The semi-supervised few-shot time series anomaly detection and classification system based on pre-trained model-guided fine-tuning according to claim 1, characterized in that: The query matrix Q, key matrix K and value matrix V of the encoder block and decoder block are calculated as follows: The query matrix of the hth head of the first encoder block The key matrix of the h-th head A matrix of values ​​shared by all heads The embedding matrix output by the embedding coding sublayer of the encoder network It is obtained by the fully connected linear neural network transformation of the first layer encoder block, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the aforementioned first-layer encoder block; The query matrix of the h-th head of the encoder block of the remaining layers The key matrix of the h-th head A matrix of values ​​shared by all heads The output data of the encoder block in the previous layer of the encoder block It is obtained by the fully connected linear neural network transformation of the encoder block, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the aforementioned encoder block; The query matrix of the hth head of the first layer decoder block is The key matrix of the h-th head A matrix of values ​​shared by all heads The embedding matrix output by the embedding encoding sublayer of the decoder network It is obtained by the fully connected linear neural network transformation of the first layer decoder block, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the aforementioned first-layer decoder block; The query matrix of the h-th head of the decoder block of the remaining layers The output data of the decoder block of the previous layer of the decoder block of this layer It is obtained by the fully connected linear neural network transformation of the decoder block of this layer, that is, In the above formula are the learning parameters of the fully connected linear neural network of the decoder block in this layer; The key matrix of the h-th head of the decoder block of the remaining layers A matrix of values ​​shared by all heads The output data of the encoder network It is obtained by the fully connected linear neural network transformation of the decoder block of this layer, that is, In the above formula and are the learning parameters of the fully connected linear neural network of the decoder block in this layer.

5. The semi-supervised few-shot time series anomaly detection and classification system based on pre-trained model-guided fine-tuning according to claim 1, characterized in that: The key guidance matrix φ of the encoder block and decoder block K Each layer shares the guidance matrix φ of the encoder block and the decoder block. V Shared on each floor; K and φ V The specific calculation process is as follows: φ K and φ V The key guidance matrix and the value guidance matrix added by the pluggable and learnable attention guidance sub-unit to the key matrix and the value guidance matrix in the encoder block and the decoder block respectively; represents the key guidance matrix added by the pluggable, learnable attention guidance subunit to the key matrix of the h-th head in the l-th layer encoder block and decoder block; represents the value guidance matrix added by the pluggable, learnable attention guidance subunit to the value matrices of all heads in the l-th layer encoder block and decoder block; is the guidance vector The length of is set to k times the length of the time series window, that is, k is a positive integer; d k is the dimension of the hidden layer vector of the encoder block or decoder block; the key guidance matrix Sum value guidance matrix It is obtained by using the fully connected linear neural network transformation, that is, In the above formula and are the learnable parameters of the fully connected linear neural network, X K_init and X V_init is a randomly initialized matrix.

6. The semi-supervised few-shot time series anomaly detection and classification system based on pre-trained model-guided fine-tuning according to claim 1, characterized in that: The specific working content of the anomaly detection module is: inputting the data into the time series data to be detected after preprocessing by the preprocessing module The embedding matrix obtained by the embedding coding sublayer The trained pre-trained model parameters Θ and the fine-tuned guidance vector Input to the model module, instructing the model module to perform anomaly detection and classification tasks; The output of the anomaly detection module is the label of anomaly detection and classification That is, the corresponding exception type.

7. A method for semi-supervised few-sample time series anomaly detection and classification based on the system of claim 1, characterized in that: The method comprises the following steps: (1) The data input and preprocessing module preprocesses a large amount of unlabeled time series data, a small number of labeled time series data, and time series data to be detected; (2) the model pre-training module requests unlabeled time series data from the data input and pre-processing module, and instructs the model module to perform model pre-training; After the model pre-training of the model module is completed, the model pre-training module requests the storage module to save the pre-trained pre-trained model, and the storage module returns whether the saving is successful; (3) The guidance fine-tuning module requests the storage module to read the pre-trained pre-trained model, and the storage module returns the pre-trained model; then the guidance fine-tuning module requests the labeled time series data from the data input and preprocessing module, instructing the model module to perform guidance fine-tuning; after the guidance fine-tuning is completed, the guidance fine-tuning module requests the storage module to store the guidance vector, and the storage module returns whether the storage is successful; (4) The anomaly detection module first requests the time series data to be detected from the data input and preprocessing module; Requesting the storage module to read the trained pre-trained model, the storage module returns the trained pre-trained model; then requesting the storage module to read the fine-tuned guidance vector, the storage module returns the fine-tuned guidance vector; then the anomaly detection module fuses the trained pre-trained model with the fine-tuned guidance vector and instructs the model module to perform anomaly detection and classification tasks; After the detection is completed, the results of anomaly detection and classification are output.

Images