Time series data anomaly detection method and device based on pre-training model and product

Abstract

The invention provides a time series data anomaly detection method, device and product based on a pre-training model, and relates to the technical field of time series data anomaly detection.The method comprises the steps that time series data is preprocessed to obtain an input sequence, and the time series data is composed of multiple pieces of multi-dimensional monitoring data arranged in a time sequence; enabling the input sequence to pass through a convolution self-attention pre-training model to generate an output sequence, the output sequence being a segment-level representation of the input sequence, and the output sequence comprising local features extracted from the input sequence; inputting the output sequence and the input sequence into a Transform model to generate a reconstruction sequence; according to the reconstruction sequence and the input sequence, abnormal feature knowledge is combined, an abnormal detection result is generated, and the abnormal detection result represents monitoring data in an abnormal state in the time sequence data.

Description

Technical Field

[0001] The present application relates to the technical field of time series data anomaly detection, and in particular to a time series data anomaly detection method, device and product based on a pre-trained model. Background Art

[0002] The proliferation of sensors and devices in contemporary Internet of Things (IoT) platforms has led to the generation of large amounts of complex multi-dimensional sensor data. Moreover, in many data-driven industries, including financial markets, network security, and medical monitoring, the identification of abnormal data and the guarantee of service reliability have become critical. Therefore, accurate and timely anomaly detection of multi-dimensional time series data has also received increasing attention.

[0003] In the prior art, the Transformer model has become a popular tool in the field of time series anomaly recognition due to its parallel computing and global feature extraction capabilities, providing significant progress in this field. However, the detection accuracy of anomaly detection of time series using the Transformer model is insufficient and still needs to be optimized. Therefore, it is urgent to propose a time series data anomaly detection method, device and product based on a pre-trained model to improve the accuracy of the Transformer model in anomaly detection of multidimensional time series. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a method, device and product for detecting anomalies in time series data based on a pre-trained model, so as to overcome the above problems or at least partially solve the above problems.

[0005] According to a first aspect of an embodiment of the present application, a method for detecting anomalies in time series data based on a pre-trained model is provided, the method comprising: Preprocessing the time series data to obtain an input sequence, wherein the time series data consists of a plurality of multi-dimensional monitoring data arranged in time sequence; Passing the input sequence through a convolutional self-attention pre-training model to generate an output sequence, wherein the output sequence is a segment-level representation of the input sequence, and the output sequence includes local features extracted from the input sequence; Inputting the output sequence and the input sequence into a Transformer model to generate a reconstructed sequence; According to the reconstructed sequence and the input sequence, combined with abnormal feature knowledge, an abnormality detection result is generated, and the abnormality detection result represents the monitoring data in the time series data that is in an abnormal state.

[0006] In a possible implementation, the convolutional self-attention pre-training model includes: a first position encoding module, a feature extraction module and a first output projection module; wherein the feature extraction module is composed of N multi-head attention blocks and N TCE blocks alternately connected in series; Each of the TCE blocks includes: an input projection layer, an upsampling layer, a feedforward layer, and a downsampling layer; The step of passing the input sequence through a convolutional self-attention pre-training model to generate an output sequence comprises: Using the first position encoding module, the input sequence and a zero matrix of the same shape are connected along the feature dimension to generate an initial sequence; Inputting the initial sequence into a feature extraction module, the N multi-head attention blocks and the N TCE blocks alternately perform feature extraction to obtain feature information; The feature information is input into the first output projection module for processing to generate the output sequence.

[0007] In a possible implementation, the feature extraction is performed alternately by the N multi-head attention blocks and the N TCE blocks, including: After receiving the kth attention information output by the kth multi-head attention block, the kth TCE block converts the kth attention information through the input projection layer of the kth TCE block to obtain the kth projection information; The upsampling layer of the kth TCE block performs convolution processing on the kth projection information to obtain the kth upsampling information; The feedforward layer of the kth TCE block performs dimension transformation on the kth up-sampled information to obtain the kth feedforward information; The downsampling layer of the kth TCE block performs convolution processing on the kth feedforward information to obtain the kth downsampling information; The kth TCE block introduces nonlinear transformation, processes the kth projection information, the kth downsampling information and the kth feedforward information, outputs the kth feature information, and inputs the kth feature information into the k+1th multi-head attention block.

[0008] In a possible implementation, the convolutional self-attention pre-trained model is trained by the following steps: Acquire sample time series data; the sample time series data refers to multidimensional monitoring data arranged in chronological order; Performing sliding window operation on the sample time series data to obtain a plurality of subsequence data with the same length; Inputting the subsequence data into the convolutional self-attention pre-training model to obtain an output result; Calculating a mean square error between the output result and the subsequence data as a loss function value; With the goal of minimizing the loss function value, updating the parameters of the convolutional self-attention pre-training model; Repeat the above steps until the loss function value converges, end the training, and obtain the completed convolutional self-attention pre-training model.

[0009] In a possible implementation, the Transformer model includes: a second position encoding module, an encoder, a decoder, and a second output projection module; and inputting the output sequence and the input sequence into the Transformer model to generate a reconstructed sequence includes: Merging the position information of the output sequence and the input sequence through the second position encoding module of the Transformer model to obtain merged information; Encoding the combined information using the encoder to obtain an encoding result; The encoding result is used as K and V matrices, and the last window data of the input sequence is used as Q matrix to input into the decoder for decoding to obtain a decoding result; The second output projection module generates the reconstructed sequence according to the decoding result.

[0010] In a possible implementation, the Transformer model is trained by the following steps: Acquire sample time series data; the sample time series data refers to multidimensional monitoring data arranged in chronological order; Performing sliding window operation on the sample time series data to obtain a plurality of subsequence data with the same length; Inputting the subsequence data into the convolutional self-attention pre-training model to obtain an output result; Inputting the output result and the subsequence data into the Transformer model to obtain a reconstructed sample sequence; Calculating a mean square error between the reconstructed sample sequence and the last window data of the subsequence data as a loss function value; According to the loss function value, updating the parameters of the Transformer model; Repeat the above steps until the loss function value converges, end the training, and obtain the trained Transformer model.

[0011] In a possible implementation, generating an anomaly detection result according to the reconstructed sequence and the input sequence in combination with anomaly feature knowledge includes: Calculating a reconstruction error based on the reconstructed sequence and the input sequence; Dividing the reconstruction error into a plurality of window errors through a sliding window operation; Using the Sigmoid function, the value in each window error is mapped to the range of [0, 1]; Using a Softmax function to generate a probability distribution result according to the value in the window error; Multiplying the probability distribution result by the length of the window error to obtain the abnormal feature knowledge, wherein the abnormal feature knowledge represents the relative abnormality degree at the corresponding time point; Multiplying the abnormal feature knowledge and the window error to obtain an abnormal score at a corresponding time point; The anomaly scores at each time point are integrated to obtain the anomaly detection result.

[0012] In a possible implementation manner, a reconstruction error is calculated according to the reconstructed sequence and the input sequence according to the following calculation formula: ; in, is the reconstruction error, is the reconstruction sequence, is the input sequence, M is the dimension of the input sequence, is the length of the input sequence.

[0013] The second aspect of the embodiment of the present application further provides a time series data anomaly detection device, which is used to execute the time series data anomaly detection method described in the first aspect, and the device includes: A preprocessing module, used for preprocessing the time series data to obtain an input sequence, wherein the time series data is composed of a plurality of multi-dimensional monitoring data arranged in time order; A convolutional self-attention pre-trained model, used to generate an output sequence according to the input sequence, wherein the output sequence is a segment-level representation of the input sequence, and the output sequence includes local features extracted from the input sequence; A Transformer model, used for generating a reconstructed sequence according to the output sequence and the input sequence; The anomaly detection module is used to generate an anomaly detection result based on the reconstructed sequence and the input sequence in combination with the anomaly feature knowledge, wherein the anomaly detection result represents the monitoring data in the time series data that is in an abnormal state.

[0014] The third aspect of the embodiment of the present application also provides an electronic device, including a memory, a processor and a computer program stored on the memory, and the processor executes the computer program to implement the steps in the time series data anomaly detection method based on a pre-trained model described in the first aspect of the embodiment of the present application.

[0015] The fourth aspect of the embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the time series data anomaly detection method based on a pre-trained model described in the first aspect of the embodiment of the present application are implemented.

[0016] The fifth aspect of the embodiments of the present application also provides a computer program product. When the computer program product is run on an electronic device, the processor is enabled to implement the steps in the time series data anomaly detection method based on a pre-trained model as described in the first aspect of the embodiments of the present application.

[0017] An embodiment of the present application provides a method for detecting anomalies in time series data based on a pre-trained model, the method comprising: pre-processing time series data to obtain an input sequence, the time series data consisting of multiple multidimensional monitoring data arranged in chronological order; passing the input sequence through a convolutional self-attention pre-training model to generate an output sequence, the output sequence being a segment-level representation of the input sequence, the output sequence including local features extracted from the input sequence; inputting the output sequence and the input sequence into a Transformer model to generate a reconstructed sequence; generating an anomaly detection result based on the reconstructed sequence and the input sequence in combination with abnormal feature knowledge, the anomaly detection result representing monitoring data in an abnormal state in the time series data.

[0018] The embodiment of the present application extracts local features (more fine-grained feature information) from time series data through a convolutional self-attention pre-training model, and generates a segment-level representation (i.e., an output sequence) that accurately reflects the time series features. By inputting the output sequence (using the extracted local features as supplementary information) and the input sequence (corresponding to the time series data) into the Transformer model for anomaly detection, the Transformer's ability to accurately identify abnormal data that deviates from expected behavior is enhanced, thereby improving the accuracy and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 It is a flowchart of the steps of a time series data anomaly detection method provided by an embodiment of the present application; Figure 2 It is a structural schematic diagram of a convolutional self-attention pre-training model provided in an embodiment of the present application; Figure 3 It is a structural diagram of a Transformer model provided in an embodiment of the present application; Figure 4 It is a schematic diagram of a process for generating anomaly detection results provided by an embodiment of the present application; Figure 5 This is a schematic diagram of the structure of a time series data anomaly detection device provided in an embodiment of the present application. Figure 6 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.

[0022] The proliferation of sensors and devices in contemporary Internet of Things (IoT) platforms has led to the generation of large amounts of complex, high-dimensional sensor data. As a result, the challenge of anomaly detection in large amounts of multidimensional time series data has become increasingly prominent. Identification of anomalies and assurance of service reliability have become critical in numerous data-driven industries, including financial markets, cybersecurity, and medical monitoring. As a result, anomaly detection in multivariate time series data has been a hot research area both in academic institutions and in commercial settings.

[0023] For accurate anomaly detection, Transformer models have also received increasing attention for anomaly detection in time series data. Transformer models have shown promising results in extracting global feature representations and temporal correlations of multivariate time series by leveraging the self-attention mechanism and global modeling capabilities of the Transformer model. However, due to the sparsity of anomalies, local anomalies have limited impact on the global pattern of time series. This poses a challenge to anomaly detection models based on transformer models due to their insufficient ability to extract local features.

[0024] The lack of sufficient anomaly information in the global representation may lead to the neglect of key anomaly points. Other classic anomaly detection models, such as LSTM-NDT and CAE-M, have been shown to perform well in modeling short-term cycles of time series. However, the complexity of these models leads to slow processing speed and high computational cost, which hinders their ability to effectively capture long-term dependencies. Although these traditional anomaly detection models are able to model either global or local patterns, they do not recognize the importance of integrating these two aspects in an organic way. This one-sided modeling approach hinders the effective capture of the interaction between global and local patterns, ultimately leading to a decrease in anomaly detection accuracy. Therefore, achieving a balance between global and local pattern modeling in the model is a major challenge, and it is necessary to consider both model complexity and robustness capabilities in global and local pattern modeling. In addition, traditional deep learning algorithms are data-driven methods, which essentially learn patterns and make decisions based on a large number of examples. Although traditional deep learning models have achieved considerable success in a range of fields in recent years, purely data-driven methods have inherent limitations and may not produce expected results in many instances. This is due to the fact that these methods rely on training data, which may not contain samples from specific scenarios. As a result, the model may not be able to understand and handle these scenarios correctly, which may lead to a decrease in its generalization ability.

[0025] In view of the above problems, the embodiments of the present application propose a method, device and product for detecting anomalies in time series data based on a pre-trained model to improve the accuracy of the Transformer model in detecting anomalies in multi-dimensional time series. In the following, in conjunction with the accompanying drawings, a method for detecting anomalies in time series data based on a pre-trained model provided by the embodiments of the present application is described in detail through some embodiments and their application scenarios.

[0026] The first aspect of the present application embodiment provides a time series data anomaly detection method based on a pre-trained model, referring to Figure 1 , Figure 1 A flowchart of a method for detecting anomalies in time series data provided by an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes: Step S101 , preprocessing the time series data to obtain an input sequence, wherein the time series data is composed of a plurality of multi-dimensional monitoring data arranged in time sequence.

[0027] In this embodiment, time series data refers to data composed of multiple multi-dimensional monitoring data arranged in time order. Specifically, time series data can be positioned as Among them, the time series data X contains data points at T moments, Represents a data point at time t, and the feature dimension of each data point is M. Anomaly detection is performed on time series data, that is, each data point is detected to see if it is in an abnormal state.

[0028] Time series data can be univariate (one-dimensional) or multivariate (multi-dimensional) time series data. In univariate time series data, one variable changes over time, such as monthly traffic data, annual average temperature, and daily weather data; in multivariate time series data, the data consists of multiple observed variables that fluctuate over time, and these variables are usually recorded at similar time points and show correlations with each other. Exemplarily, for factory production data as time series data, the time series data represents the data monitored by various monitoring equipment within an hour (i.e., multi-dimensional monitoring data), including production speed information, product quality information, machine operating temperature information, and power consumption information at the corresponding moment. Anomaly detection is performed on the time series data, that is, detection is performed on whether the data at a certain moment in this hour is in an abnormal state, and specifically which dimension (production speed information, product quality information, machine operating temperature information, or power consumption information) of the information is abnormal.

[0029] In this embodiment, the acquired time series data is preprocessed and the time series data can be cropped so that the length of the sequence data obtained after cropping conforms to the input of the convolutional self-attention pre-training model to generate an input sequence.

[0030] Step S102, passing the input sequence through a convolutional self-attention pre-training model to generate an output sequence, wherein the output sequence is a segment-level representation of the input sequence, and the output sequence includes local features extracted from the input sequence.

[0031] The combination of the convolution module and the self-attention module can effectively improve the feature extraction effect. Based on this, the embodiment of the present application designs a convolutional self-attention pre-training model (CAPNets), which can effectively learn and extract local features in the input sequence and generate segment-level representations (i.e., output sequences) containing rich feature information. As a means of effectively integrating the advantages of convolution and self-attention, the convolutional self-attention pre-training model unifies the interaction between local and non-local features in the time series, thereby facilitating the capture of abnormal information.

[0032] In a possible implementation, the convolutional self-attention pre-training model includes: a first position encoding module, a feature extraction module and a first output projection module; wherein the feature extraction module is composed of N multi-head attention blocks and N TCE blocks connected in series alternately. Each TCE block includes: an input projection layer, an upsampling layer, a feedforward layer, and a downsampling layer.

[0033] In this embodiment, refer to Figure 2 , Figure 2 A structural diagram of a convolutional self-attention pre-training model is shown, Figure 2 As shown in Figure 2, the convolutional self-attention pre-trained model consists of a first position encoding module (such as Figure 2 PositionalEncoding module shown in FIG. 1 ), feature extraction module and first output projection module (such as Figure 2 The Output Projection module shown in Figure 1 is composed of a series of N multi-head attention blocks (such as Figure 2 The Multi-HeadAttention Block module shown in Figure 2 The TCE Block module shown in FIG. 1 is composed of a multi-head attention block, a TCE block, a multi-head attention block, a TCE block, and so on. N represents a positive integer greater than 1.

[0034] The step S102, wherein the input sequence is passed through a convolutional self-attention pre-training model to generate an output sequence, comprises: Step S1021: Using the first position encoding module, the input sequence and a zero matrix of the same shape are connected along the feature dimension to generate an initial sequence.

[0035] First, input the sequence I’ , using the first position encoding module in the convolutional self-attention pre-training model as shown in the following formula 1, the input sequence I’ Zero matrix of the same shape as C , connect along the feature dimension and perform position encoding to generate the initial sequence I 0.

[0036] Formula 1; in, I’ is the input sequence; C is a zero matrix; I 0 is the initial sequence. I 0∈R k×b×M , I’ ∈R k×b×2M , where b is the batch size, M is the dimension size, and k is the length of the subsequence data, that is, the sliding window size (indicating the number of multidimensional monitoring data in each subsequence data).

[0037] Step S1022: input the initial sequence into a feature extraction module, and the N multi-head attention blocks and the N TCE blocks alternately perform feature extraction to obtain feature information.

[0038] like Figure 2 As shown, in the process of feature extraction module extracting features from the initial sequence, according to the alternating series order of N multi-head attention blocks and TCE blocks, after each multi-head attention block is processed, the processing result is sent to the connected TCE block for processing. After the TCE block is processed, it continues to be sent to the next connected multi-head attention block for processing until the last TCE block is processed, and the output result of the feature extraction module is obtained.

[0039] Among them, the processing flow of each multi-head attention block in the feature extraction module is as follows: Figure 2 As shown, each multi-head attention block (i.e. Figure 2 The Multi-Head Attention Block shown in Figure 1) consists of two parts: the multi-head attention layer (i.e. Figure 2 Multi-Head Attention shown in Figure 2), residual connections and layer normalization (i.e. Figure 2 Add&Norm as shown). For the initial input sequence, it is first linearly transformed to convert it into a query matrix Q, a key matrix K and a value matrix V. Then, the multi-head feedforward layer generates h subspaces respectively. Q i (query matrix), K i (key matrix) and V i (value matrix), where i∈{1,…h}. Then, as shown in Formula 2, for each subspace, the scaled dot product attention method is used, so that the attention information can be extracted from the above h different subspaces.

[0040] Formula 2: in, H i is the attention information, Q i is the query matrix of the i-th subspace; K i is the key matrix of the i-th subspace; V i is the value matrix of the i-th subspace.

[0041] Finally, as shown in Formula 3, the attention information obtained from the h subspaces is connected to produce the final result of multi-head attention. This result contains the attention information from different subspaces, allowing the model to comprehensively consider various correlations.

[0042] Formula 3; Among them, the scaled dot product attention method can be performed according to the following formula 4 to extract the attention information of the corresponding subspace: Formula 4; in, Q is the query matrix; K is the key matrix; V is the value matrix, T is the total length of the time series data, d k is the scaling factor constant.

[0043] In a possible implementation, the step S1022, wherein the N multi-head attention blocks and the N TCE blocks alternately perform feature extraction, includes: Step S1022-1, after receiving the kth attention information output by the kth multi-head attention block, the kth TCE block converts the kth attention information through the input projection layer of the kth TCE block to obtain the kth projection information, where k is any positive integer less than N.

[0044] Specifically, let the input information of the k-th multi-head attention block be , the kth attention information output by the kth multi-head attention block is , the k-th multi-head attention block extracts attention information according to the following formula 5: Formula 5; like Figure 2 As shown, each of the TCE blocks includes: an input projection layer (such as Figure 2 Conv1d shown), an upsampling layer (such as Figure 2 Convlution Module shown), a feed-forward layer (such as Figure 2 Feed Forward as shown), a downsampling layer (such as Figure 2 The k-th TCE block receives the k-th attention information output by the k-th multi-head attention block. After that, the input projection layer (such as Figure 2 Conv1d shown in the figure converts the k-th attention information according to formula 6 to obtain the k-th projection information .

[0045] Formula 6; The purpose of the input projection layer is to convert the k-th attention information Converted into the kth downsampling information output by the downsampling layer Same shape.

[0046] Step S1022-2: The upsampling layer of the k-th TCE block performs convolution processing on the k-th projection information to obtain k-th upsampling information.

[0047] Specifically, the kth projection information is obtained by processing the input projection layer After that, the upsampling layer performs convolution processing on the k-th projection information according to Formula 7 to obtain the k-th upsampling information This upsampling layer does not change the feature dimension of the time series data, but performs a convolution operation along the time dimension to obtain additional contextual information.

[0048] Formula 7; Step S1022-3: The feedforward layer of the kth TCE block performs dimension transformation on the kth up-sampled information to obtain the kth feedforward information.

[0049] Specifically, after the upsampling layer is processed, the kth upsampling information is obtained. Afterwards, the feed-forward layer of the kth TCE block (such as Figure 2 According to Formula 8, the k-th up-sampled information is transformed in dimension to obtain the k-th feed-forward information .

[0050] Formula eight.

[0051] The feed-forward layer transforms the dimensions of the time series data, helping the model learn complex relationships and dependencies between features.

[0052] Step S1022-4: The downsampling layer of the kth TCE block performs convolution processing on the kth feedforward information to obtain the kth downsampling information.

[0053] Specifically, the kth feedforward information is output in the feedforward layer Afterwards, the downsampling layer of the kth TCE block (such as Figure 2 The Convlution Module shown in the figure) uses Formula 9 to calculate the k-th feedforward information. Perform convolution processing to capture information globally from the features and obtain the kth down-sampling information : Formula nine.

[0054] Step S1022-5, the kth TCE block introduces nonlinear transformation, processes the kth projection information, the kth downsampling information and the kth feedforward information, outputs the kth feature information, and inputs the kth feature information into the k+1th multi-head attention block.

[0055] Specifically, the kth downsampling information is output in the downsampling layer After that, the kth TCE introduces a nonlinear transformation (such as Figure 2 Add&Swish shown in the figure) processes the k-th projection information, the k-th down-sampling information and the k-th feedforward information according to formula 10 to obtain the k-th feature information The role of the Swish function is to enhance the expressive power of the neural network by introducing nonlinear transformations, so that the neural network can learn more complex patterns and features, effectively alleviate the gradient vanishing problem, and make the neural network more stable during the training process.

[0056] Formula 10; The embodiment of the present application constructs a multi-level structure of TCE blocks, so that the combination of these layers can improve the model's ability to understand and model time series, and extract higher-level and more abstract feature representations.

[0057] Step S1023: input the feature information into the first output projection module for processing to generate the output sequence.

[0058] The first output projection module (such as Figure 2 The Output Projection shown in Figure 1 is composed of a linear layer and a sigmoid function layer. The first output projection module is used to restore the shape of the data and transform the dimension of the feature information so that the feature dimension is converted from 2M to M. Finally, the output sequence P is obtained, P∈R k×b×M The output sequence P is a segment-level representation of the input sequence. The output sequence includes local features extracted from the input sequence. Therefore, in the subsequent anomaly detection, the local feature information can be combined for calculation and analysis to obtain more accurate anomaly detection results.

[0059] In a possible implementation, the convolutional self-attention pre-trained model is trained by the following steps: Step S201, obtaining sample time series data; the sample time series data refers to multi-dimensional monitoring data arranged in chronological order.

[0060] Specifically, first obtain sample time series data, which is multidimensional monitoring data at multiple moments within a certain time period obtained through real monitoring. For example, the sample time series data is factory production data, which represents the data (i.e., multidimensional monitoring data) obtained by monitoring various monitoring devices within a certain time period, including production speed information, product quality information, machine operating temperature information, and power consumption information at the corresponding moment. The sample time series data is normalized according to the following formula 11: Formula XI; in, Represents the multidimensional monitoring data corresponding to time t in the sample time series data The normalized result is is the maximum value of the multidimensional monitoring data at all time points in the time series data, is the minimum value of the multidimensional monitoring data at all time points in the time series data, is a non-zero vector used to prevent division by zero.

[0061] Step S202, performing sliding window operation on the sample time series data to obtain a plurality of subsequence data with the same length.

[0062] Specifically, since the length of the obtained sample time series data may not be consistent and may be longer than the length of the input sequence, a sliding window operation is performed using a sliding window with a time step of 1 to obtain multiple subsequence data W with consistent lengths. This is shown in the following formula 12: Formula 12; in, is the t-th subsequence data, k is the length of the subsequence data, that is, the sliding window size (indicating the number of multidimensional monitoring data in each subsequence data). At time t, if the available data points are less than k, then Insert k−t occurrences of η 1 is used to fill the missing data points. T subsequence data are obtained Among them, W t ∈R 1×b×M .

[0063] Step S203: input the subsequence data into the convolutional self-attention pre-training model to obtain an output result.

[0064] Specifically, the obtained multiple subsequence data are input into the convolutional self-attention pre-training model in batches, and each subsequence data can correspond to an output result of the model output. The subsequence data and the zero matrix of the same shape are connected along the feature dimension using the first position encoding module to generate a sample initial sequence; the sample initial sequence is input into the feature extraction module, and feature extraction is performed alternately by N multi-head attention blocks and N TCE blocks to obtain sample feature information; the sample feature information is input into the first output projection module for processing to generate an output result.

[0065] Step S204: Calculate the mean square error between the output result and the subsequence data as a loss function value.

[0066] Specifically, the mean square error between the output result and the subsequence data is calculated according to the following formula 13: Formula XIII; Among them, k is the length of the subsequence data, that is, the sliding window size; b is the batch size of the subsequence data, and M is the dimension size of the subsequence data; Output the result of dimension K (the range of K is 1-M) for the j-th batch of data at the i-th moment; It is the subsequence data of dimension K of the j-th batch of data at the i-th moment.

[0067] Step S205, updating the parameters of the convolutional self-attention pre-training model with the goal of minimizing the loss function value.

[0068] Step S206, repeat the above steps until the loss function value converges, end the training, and obtain the trained convolutional self-attention pre-training model.

[0069] Specifically, with the goal of minimizing the loss function value (i.e., mean square error), parameter backpropagation is performed to update each module in the convolutional self-attention pre-training model (the first output projection module and each multi-head attention block and TCE block in the feature extraction module) in order to generate a segment-level representation with richer local information.

[0070] In an embodiment of the present application, in order to improve the accuracy of anomaly detection of time series data by the Transformer model, a unique convolutional self-attention pre-training model (CAPNets) is proposed, through which local feature information is extracted from time series data to enhance the local pattern modeling capability of the Transformer model. Among them, the convolutional self-attention pre-training model is a pre-training model based on multi-head attention blocks and TCE blocks. The model uses TCE blocks to capture local feature information in the global modeling process of time series, so that it can effectively extract fine-grained patterns and generate segment-level representations that accurately reflect the feature patterns of time series. These segment-level representations include rich local contextual information. The introduction of this information can enhance the ability of the Transformer model to accurately identify instances that deviate from expected behavior, which helps to improve the accuracy of the Transformer model in detecting anomalies of time series data.

[0071] Step S103: input the output sequence and the input sequence into a Transformer model to generate a reconstructed sequence.

[0072] After step S102, the convolutional self-attention pre-trained model is used to extract feature information from the input time series data, that is, the output sequence (segment-level representation containing local feature information), and the output sequence is sent to the Transformer model together with the input sequence. The output sequence is used to compensate for the limited ability of the Transformer model in extracting fine-grained local feature models, thereby giving the Transformer model a more accurate time series reconstruction capability.

[0073] In a possible implementation, the Transformer model includes: a second position encoding module, an encoder, a decoder, and a second output projection module; Reference Figure 3 , Figure 3 A schematic diagram of the structure of a Transformer model is shown, Figure 3 As shown, the Transformer model consists of a second position encoding module (such as Figure 3 Positional Encoding module shown), encoder (such as Figure 3 Transformer Encoder module shown), decoder (such as Figure 3 The Transformer Decoder module shown in Figure 1) and the second output projection module (as shown in Figure 2) Figure 3 The Output Projection modules shown in the figure are connected in series.

[0074] The step S103, inputting the output sequence and the input sequence into a Transformer model to generate a reconstructed sequence, includes: Step S1031, merging the position information of the output sequence and the input sequence through the second position encoding module of the Transformer model to obtain merged information.

[0075] Specifically, as shown in Formula 14, the convolutional self-attention pre-training model (such as Figure 3 The output sequence of the Pre-trained CAPNets shown P Input sequence of the self-attention pre-trained model with the original input convolution The connection is performed, and the position information is merged using a second position encoding module to obtain merged information.

[0076] Formula 14; in, For the merged information, is the input sequence, P is the output sequence, ∈R k×b×2M , PE Represents the position encoding function.

[0077] Step S1032: Encode the merged information using the encoder to obtain an encoding result.

[0078] Specifically, the encoder (such as Figure 3 The Transformer Encoder module shown in the figure is defined as follows: Formula 15; Formula 16; in, For the merged information, is the intermediate encoding information, is the encoding result. Among them, , ∈R k×b×2M .

[0079] Step S1033: Use the encoding result as K and V matrices, and use the last window data of the input sequence as Q matrix to input into the decoder for decoding to obtain a decoding result.

[0080] Specifically, for a decoder (such as Figure 3 The Transformer Decoder module shown in FIG. 1 is a Transformer Decoder module, whose input is the Q matrix (i.e., query matrix), the K matrix (i.e., key matrix), and the V matrix (i.e., value matrix). In this embodiment, in order to further speed up the reasoning speed of the Transformer model, the entire input sequence is not considered. Instead of inputting it into the decoder as the Q matrix, the last window data set (i.e., the last window data) is selected from the input sequence as the Q matrix. Specifically, the input sequence It is a sequence of data consisting of multiple data points arranged in time order, such as ={ e t-k+1 , e t-k+2 , ……e t-1 , e t}∈R k×b×M , where the last window data is the last data point in the input sequence e t The last window data is used as the Q matrix, and the encoding result is input into the decoder as the K and V matrices, allowing the Transformer model to obtain the current time point and the previous time point. The dependency between consecutive time points enables it to emphasize local contextual information, further enhancing the Transformer model's sensitivity to local information, thereby improving the model's reasoning efficiency.

[0081] Specifically, the definition of the decoder is shown in the following formula: ; ; ; ; in, e t is the last window data, is the encoding result, To decode the result, , i ∈{0,1,2,3}.

[0082] Step S1034: the second output projection module generates the reconstructed sequence according to the decoding result.

[0083] Specifically, the decoder outputs the decoding result Traverse an output projection layer (the second output projection module) and retrieve the corresponding e t The reconstruction sequence is as follows: ; in, To decode the result, R To reconstruct the sequence, .

[0084] In a possible implementation, the Transformer model is trained by the following steps: Step S301, obtaining sample time series data; the sample time series data refers to multi-dimensional monitoring data arranged in chronological order.

[0085] Specifically, first obtain sample time series data, which is multi-dimensional monitoring data at multiple moments within a certain time period obtained through real monitoring. For example, the sample time series data is factory production data, which represents data (i.e., multi-dimensional monitoring data) obtained by monitoring various monitoring devices within a certain time period, including production speed information, product quality information, machine operating temperature information, and power consumption information at the corresponding moment. The sample time series data can be normalized according to the calculation process of Formula 11 above.

[0086] Step S302, sliding a window on the sample time series data to obtain a plurality of subsequence data of the same length. The sliding window process in step S202 above can be used to generate a plurality of subsequence data W of the same length. Where W t ∈R 1 ×b×M .

[0087] Step S303: input the subsequence data into the convolutional self-attention pre-training model to obtain an output result.

[0088] Specifically, for the multiple subsequence data obtained ={ e t-k+1 , e t-k+2 , ……e t-1 , e t}∈R k×b×M , and are input into the convolutional self-attention pre-trained model in batches. Each subsequence data can correspond to an output result of the model output. In this step, the output result of the model output is a segment-level representation of the input subsequence data, and the output sequence includes the local features extracted from the input subsequence data.

[0089] Step S304: input the output result and the subsequence data into the Transformer model to obtain a reconstructed sample sequence.

[0090] Specifically, the output result and the subsequence data are input into the Transformer model for sequence reconstruction to obtain a reconstructed sample sequence. Specifically, the output result and the subsequence data are merged for position information through the second position encoding module of the Transformer model to obtain merged sample information; the merged information is encoded using an encoder to obtain a sample encoding result; the sample encoding result is used as the K and V matrices, and the last window data of the subsequence data is used as the Q matrix to input the decoder for decoding to obtain a sample decoding result; the second output projection module generates a reconstructed sample sequence according to the sample decoding result.

[0091] Step S305 , calculating the mean square error between the reconstructed sample sequence and the last window data of the subsequence data as the loss function value.

[0092] Specifically, the loss function value is calculated according to the following formula: ;

[0093] Specifically, R is the reconstructed sample sequence, e tis the last window data, b is the batch size of the subsequence data, M is the feature dimension of the subsequence data, is the original data of dimension j in the i-th batch of data at time t, is the reconstructed data of dimension j in the i-th batch of data at time t.

[0094] Step S306: Update the parameters of the Transformer model according to the loss function value.

[0095] Step S307, repeat the above steps until the loss function value converges, end the training, and obtain the trained Transformer model.

[0096] Specifically, according to the loss function value, the parameters are passed back and each module in the Transformer model (encoder, decoder and second output projection module) is updated, so that the data points in the normal state in the original input subsequence data are more similar to the data points at the corresponding positions in the reconstructed sample sequence, while the data points in the abnormal state in the original input subsequence data are more different from the data points at the corresponding positions in the reconstructed sample sequence.

[0097] Step S104, generating an anomaly detection result according to the reconstructed sequence and the input sequence in combination with abnormal feature knowledge, wherein the anomaly detection result represents monitoring data in an abnormal state in the time series data.

[0098] Specifically, after the reconstruction sequence is generated by the Transformer model, the reconstruction error is calculated based on the reconstruction sequence and the input sequence, and the difference between abnormal points and normal points is amplified in combination with the abnormal feature knowledge, thereby enhancing the model's ability to distinguish normal and abnormal instances, so as to improve the accuracy of decision-making (determining whether there are abnormal data points in time series data) and the generalization of the model.

[0099] Reference Figure 4 , Figure 4 A schematic diagram of a process for generating anomaly detection results is shown. In a possible implementation, generating anomaly detection results based on the reconstructed sequence and the input sequence in combination with abnormal feature knowledge includes: Step S1041: Calculate a reconstruction error according to the reconstructed sequence and the input sequence.

[0100] The reconstruction error represents the difference between the reconstructed sequence and the input sequence. In a possible implementation, the reconstruction error is calculated based on the reconstructed sequence and the input sequence according to the following calculation formula: ; in, is the reconstruction error, is the reconstruction sequence, is the input sequence, M is the dimension of the input sequence, is the length of the input sequence.

[0101] Step S1042: dividing the reconstruction error into a plurality of window errors through a sliding window operation.

[0102] Specifically, a sliding window with a time step of 1 is used to perform a sliding window operation on the calculated reconstruction error to obtain multiple window errors with the same length. S i ,Right now , where M is the dimension of the input sequence, is the length of the input sequence, L is the length of the sliding window. .

[0103] Step S1043, using the Sigmoid function to map the value in each window error to the range of [0, 1].

[0104] Step S1044, using the Softmax function to generate a probability distribution result according to the value of the window error.

[0105] Specifically, Figure 4 As shown in the figure, the Sigmoid function is used to map the reconstruction error of each point in the window error to the range of [0,1], and then the Softmax function is used to convert the reconstruction error of each point in the window error into a probability distribution, so that the error value of each point is assigned a certain probability according to its size, a larger error value is assigned a higher probability, and a smaller error value is assigned a lower probability, thereby quantifying the relative abnormality level of each point in the error window according to the probability distribution result.

[0106] Step S1045, multiplying the probability distribution result by the length of the window error to obtain the abnormal feature knowledge, where the abnormal feature knowledge represents the relative abnormality degree at the corresponding time point.

[0107] Specifically, the calculation process of abnormal feature knowledge is shown in the following formula: ; in, For abnormal feature knowledge (such as Figure 4 Anomaly Feature Knowledge, as shown in is the window error, L is the length of the window error, is the probability distribution result. Therefore, by using the common abnormal feature knowledge, the abnormal score between normal points and abnormal points is further amplified, thus enhancing the effectiveness of anomaly detection.

[0108] Step S1046, multiplying the abnormal feature knowledge and the window error to obtain an abnormal score at the corresponding time point.

[0109] Specifically, Figure 4 As shown, the abnormal score at time point t is calculated according to the following formula (corresponding to Figure 4 Anomaly Score in ; in, is the anomaly score at time point t, For abnormal characteristic knowledge, is the window error.

[0110] Step S1047, integrating the anomaly scores at each time point to obtain the anomaly detection result.

[0111] Specifically, as shown in the following formula, at each time point (according to the length of the input sequence from the first moment to The anomaly score of the point at the time is obtained to obtain the final anomaly score.

[0112] ; in, is the anomaly score at time point t The value of the last element in the calculation to get the final anomaly score Then, the peak over threshold (POT) method is used to determine the target threshold. If the anomaly score of a certain dimension i∈{1, 2, ...M} at a certain time point in the time series data exceeds the target threshold, it is determined that the dimension is in an abnormal state at that time point. Then, an anomaly detection result is generated, which is used to indicate whether the multidimensional monitoring data at each time point in the time series data is in an abnormal state, and the information of the corresponding dimension in the abnormal state. Exemplarily, in the application scenario of production status monitoring, according to the anomaly detection results, it can be known that the monitoring data at time A is in an abnormal state, and it can be known that the monitoring data of the dimension of the machine operating temperature at time A is in an abnormal state.

[0113] The embodiment of the present application extracts local features (more fine-grained feature information) from time series data through a convolutional self-attention pre-training model, and generates a segment-level representation (i.e., an output sequence) that accurately reflects the time series features. By inputting the output sequence (using the extracted local features as supplementary information) and the time series data (corresponding to the input sequence) into the Transformer model for anomaly detection, the limited ability of the Transformer model in extracting fine-grained local feature models is compensated, and the ability of the Transformer to accurately identify abnormal data that deviates from expected behavior is enhanced, thereby improving the accuracy and generalization ability of the model.

[0114] The second aspect of the embodiment of the present application further provides a time series data anomaly detection device, which is used to execute the time series data anomaly detection method described in the first aspect, referring to Figure 5 , Figure 5 A schematic diagram of the structure of a time series data anomaly detection device is shown. Figure 5 As shown, the device comprises: A preprocessing module, used for preprocessing the time series data to obtain an input sequence, wherein the time series data is composed of a plurality of multi-dimensional monitoring data arranged in time order; A convolutional self-attention pre-trained model, used to generate an output sequence according to the input sequence, wherein the output sequence is a segment-level representation of the input sequence, and the output sequence includes local features extracted from the input sequence; A Transformer model, used for generating a reconstructed sequence according to the output sequence and the input sequence; The anomaly detection module is used to generate an anomaly detection result based on the reconstructed sequence and the input sequence in combination with the anomaly feature knowledge, wherein the anomaly detection result represents the monitoring data in the time series data that is in an abnormal state.

[0115] In a possible implementation, the convolutional self-attention pre-training model includes: a first position encoding module, a feature extraction module and a first output projection module; wherein the feature extraction module is composed of N multi-head attention blocks and N TCE blocks alternately connected in series; Each of the TCE blocks includes: an input projection layer, an upsampling layer, a feedforward layer, and a downsampling layer; According to the input sequence, an output sequence is generated, including: Using the first position encoding module, the input sequence and a zero matrix of the same shape are connected along the feature dimension to generate an initial sequence; Inputting the initial sequence into a feature extraction module, the N multi-head attention blocks and the N TCE blocks alternately perform feature extraction to obtain feature information; The feature information is input into the first output projection module for processing to generate the output sequence.

[0116] In a possible implementation, the feature extraction is performed alternately by the N multi-head attention blocks and the N TCE blocks, including: After receiving the kth attention information output by the kth multi-head attention block, the kth TCE block converts the kth attention information through the input projection layer of the kth TCE block to obtain the kth projection information; The upsampling layer of the kth TCE block performs convolution processing on the kth projection information to obtain the kth upsampling information; The feedforward layer of the kth TCE block performs dimension transformation on the kth up-sampled information to obtain the kth feedforward information; The downsampling layer of the kth TCE block performs convolution processing on the kth feedforward information to obtain the kth downsampling information; The kth TCE block introduces nonlinear transformation, processes the kth projection information, the kth downsampling information and the kth feedforward information, outputs the kth feature information, and inputs the kth feature information into the k+1th multi-head attention block.

[0117] In a possible implementation, the convolutional self-attention pre-trained model is trained by the following steps: Acquire sample time series data; the sample time series data refers to multidimensional monitoring data arranged in chronological order; Performing sliding window operation on the sample time series data to obtain a plurality of subsequence data with the same length; Inputting the subsequence data into the convolutional self-attention pre-training model to obtain an output result; Calculating a mean square error between the output result and the subsequence data as a loss function value; With the goal of minimizing the loss function value, updating the parameters of the convolutional self-attention pre-training model; Repeat the above steps until the loss function value converges, end the training, and obtain the completed convolutional self-attention pre-training model.

[0118] In a possible implementation, the Transformer model includes: a second position encoding module, an encoder, a decoder, and a second output projection module; and generating a reconstructed sequence according to the output sequence and the input sequence includes: Merging the position information of the output sequence and the input sequence through the second position encoding module of the Transformer model to obtain merged information; Encoding the combined information using the encoder to obtain an encoding result; The encoding result is used as K and V matrices, and the last window data of the input sequence is used as Q matrix to input into the decoder for decoding to obtain a decoding result; The second output projection module generates the reconstructed sequence according to the decoding result.

[0119] In a possible implementation, the Transformer model is trained by the following steps: Acquire sample time series data; the sample time series data refers to multidimensional monitoring data arranged in chronological order; Performing sliding window operation on the sample time series data to obtain a plurality of subsequence data with the same length; Inputting the subsequence data into the convolutional self-attention pre-training model to obtain an output result; Inputting the output result and the subsequence data into the Transformer model to obtain a reconstructed sample sequence; Calculating a mean square error between the reconstructed sample sequence and the last window data of the subsequence data as a loss function value; According to the loss function value, updating the parameters of the Transformer model; Repeat the above steps until the loss function value converges, end the training, and obtain the trained Transformer model.

[0120] In a possible implementation, generating an anomaly detection result according to the reconstructed sequence and the input sequence in combination with anomaly feature knowledge includes: Calculating a reconstruction error based on the reconstructed sequence and the input sequence; Dividing the reconstruction error into a plurality of window errors through a sliding window operation; Using the Sigmoid function, the value in each window error is mapped to the range of [0, 1]; Using a Softmax function to generate a probability distribution result according to the value in the window error; Multiplying the probability distribution result by the length of the window error to obtain the abnormal feature knowledge, wherein the abnormal feature knowledge represents the relative abnormality degree at the corresponding time point; Multiplying the abnormal feature knowledge and the window error to obtain an abnormal score at a corresponding time point; The anomaly scores at each time point are integrated to obtain the anomaly detection result.

[0121] In a possible implementation manner, a reconstruction error is calculated according to the reconstructed sequence and the input sequence according to the following calculation formula: ; in, is the reconstruction error, is the reconstruction sequence, is the input sequence, M is the dimension of the input sequence, is the length of the input sequence.

[0122] The present application also provides an electronic device, referring to Figure 6 , Figure 6 Schematic diagram of an electronic device proposed in an embodiment of the present application. Figure 6 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in the time series data anomaly detection method based on a pre-trained model disclosed in the embodiment of the present application.

[0123] An embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the time series data anomaly detection method based on a pre-trained model as disclosed in the embodiment of the present application are implemented.

[0124] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the processor is enabled to implement the steps of the time series data anomaly detection method based on a pre-trained model as disclosed in the embodiment of the present application.

[0125] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0126] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0129] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0130] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0131] The above is a detailed introduction to the time series data anomaly detection method, device and product based on a pre-trained model provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for detecting anomalies in time series data based on a pre-trained model, characterized in that: The method comprises: Preprocessing the time series data to obtain an input sequence, wherein the time series data consists of a plurality of multi-dimensional monitoring data arranged in time sequence; Passing the input sequence through a convolutional self-attention pre-training model to generate an output sequence, wherein the output sequence is a segment-level representation of the input sequence, and the output sequence includes local features extracted from the input sequence; Inputting the output sequence and the input sequence into a Transformer model to generate a reconstructed sequence; According to the reconstructed sequence and the input sequence, combined with abnormal feature knowledge, an abnormality detection result is generated, and the abnormality detection result represents the monitoring data in the time series data that is in an abnormal state.

2. The time series data anomaly detection method according to claim 1, characterized in that: The convolutional self-attention pre-training model includes: a first position encoding module, a feature extraction module and a first output projection module; wherein the feature extraction module is composed of N multi-head attention blocks and N TCE blocks alternately connected in series; Each of the TCE blocks includes: an input projection layer, an upsampling layer, a feedforward layer, and a downsampling layer; The step of passing the input sequence through a convolutional self-attention pre-training model to generate an output sequence comprises: Using the first position encoding module, the input sequence and a zero matrix of the same shape are connected along the feature dimension to generate an initial sequence; Inputting the initial sequence into a feature extraction module, the N multi-head attention blocks and the N TCE blocks alternately perform feature extraction to obtain feature information; The feature information is input into the first output projection module for processing to generate the output sequence.

3. The time series data anomaly detection method according to claim 2, characterized in that: The feature extraction is performed alternately by the N multi-head attention blocks and the N TCE blocks, including: After receiving the kth attention information output by the kth multi-head attention block, the kth TCE block converts the kth attention information through the input projection layer of the kth TCE block to obtain the kth projection information; The upsampling layer of the kth TCE block performs convolution processing on the kth projection information to obtain the kth upsampling information; The feedforward layer of the kth TCE block performs dimension transformation on the kth up-sampled information to obtain the kth feedforward information; The downsampling layer of the kth TCE block performs convolution processing on the kth feedforward information to obtain the kth downsampling information; The kth TCE block introduces nonlinear changes, processes the kth projection information, the kth downsampling information and the kth feedforward information, outputs the kth feature information, and inputs the kth feature information into the k+1th multi-head attention block.

4. The time series data anomaly detection method according to claim 2, characterized in that: The convolutional self-attention pre-trained model is trained through the following steps: Acquire sample time series data; the sample time series data refers to multidimensional monitoring data arranged in chronological order; Performing sliding window operation on the sample time series data to obtain a plurality of subsequence data with the same length; Inputting the subsequence data into the convolutional self-attention pre-training model to obtain an output result; Calculating a mean square error between the output result and the subsequence data as a loss function value; With the goal of minimizing the loss function value, updating the parameters of the convolutional self-attention pre-training model; Repeat the above steps until the loss function value converges, end the training, and obtain the completed convolutional self-attention pre-training model.

5. The time series data anomaly detection method according to claim 1, characterized in that: The Transformer model includes: a second position encoding module, an encoder, a decoder, and a second output projection module; the output sequence and the input sequence are input into the Transformer model to generate a reconstructed sequence, including: Merging the position information of the output sequence and the input sequence through the second position encoding module of the Transformer model to obtain merged information; Encoding the combined information using the encoder to obtain an encoding result; The encoding result is used as K and V matrices, and the last window data of the input sequence is used as Q matrix to input into the decoder for decoding to obtain a decoding result; The second output projection module generates the reconstructed sequence according to the decoding result.

6. The method for detecting anomalies in time series data according to claim 5, characterized in that: The Transformer model is trained through the following steps: Acquire sample time series data; the sample time series data refers to multidimensional monitoring data arranged in chronological order; Performing sliding window operation on the sample time series data to obtain a plurality of subsequence data with the same length; Inputting the subsequence data into the convolutional self-attention pre-training model to obtain an output result; Inputting the output result and the subsequence data into the Transformer model to obtain a reconstructed sample sequence; Calculating a mean square error between the reconstructed sample sequence and the last window data of the subsequence data as a loss function value; According to the loss function value, updating the parameters of the Transformer model; Repeat the above steps until the loss function value converges, end the training, and obtain the trained Transformer model.

7. The time series data anomaly detection method according to claim 1, characterized in that: The generating anomaly detection result according to the reconstructed sequence and the input sequence in combination with abnormal feature knowledge includes: Calculating a reconstruction error based on the reconstructed sequence and the input sequence; Dividing the reconstruction error into a plurality of window errors through a sliding window operation; Using the Sigmoid function, the value in each window error is mapped to the range of [0, 1]; Using a Softmax function to generate a probability distribution result according to the value in the window error; Multiplying the probability distribution result by the length of the window error to obtain the abnormal feature knowledge, wherein the abnormal feature knowledge represents the relative abnormality degree at the corresponding time point; Multiplying the abnormal feature knowledge and the window error to obtain an abnormal score at a corresponding time point; The anomaly scores at each time point are integrated to obtain the anomaly detection result.

8. The time series data anomaly detection method according to claim 7, characterized in that: The reconstruction error is calculated according to the reconstructed sequence and the input sequence according to the following calculation formula: ； in, is the reconstruction error, is the reconstruction sequence, is the input sequence, M is the dimension of the input sequence, is the length of the input sequence.

9. A time series data anomaly detection device, characterized in that: The device is applied to execute the time series data anomaly detection method according to any one of claims 1 to 8, and comprises: A preprocessing module, used for preprocessing the time series data to obtain an input sequence, wherein the time series data is composed of a plurality of multi-dimensional monitoring data arranged in time order; A convolutional self-attention pre-trained model, used to generate an output sequence according to the input sequence, wherein the output sequence is a segment-level representation of the input sequence, and the output sequence includes local features extracted from the input sequence; A Transformer model, used for generating a reconstructed sequence according to the output sequence and the input sequence; The anomaly detection module is used to generate an anomaly detection result based on the reconstructed sequence and the input sequence in combination with the anomaly feature knowledge, wherein the anomaly detection result represents the monitoring data in the time series data that is in an abnormal state.

10. An electronic device, characterized in that: It includes a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the time series data anomaly detection method according to any one of claims 1 to 8.

Images