A method and system for anomaly detection in multivariate time-series data based on a large language model

Abstract

This invention relates to the field of artificial intelligence technology and discloses a method and system for detecting anomalies in multivariate temporal data based on a large language model. The method involves acquiring multivariate temporal samples within a preset time window, performing sliding segmentation based on variables, and introducing variable identity encoding and time position encoding to form local segment representations. Each variable segment is input into a causal convolutional block, and a serialized token is constructed and fed into a large language model finely tuned using LoRA for global bidirectional attention modeling to uncover cross-variable and asynchronous response relationships. Residual fusion and global pooling are performed on the global features and the original sequence features to obtain sample-level representations, which are then output as anomaly identification results via a classification head. The system includes a data acquisition and preprocessing module, a segmentation and encoding module, a causal convolutional modeling module, a large language model global modeling module, and a feature fusion and anomaly discrimination module, which can improve the accuracy and robustness of multivariate temporal anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for detecting anomalies in multivariate time-series data based on a large language model. Background Technology

[0002] With the continuous expansion of the Industrial Internet, the Internet of Things, and information systems, a large number of devices, sensors, and business systems are constantly outputting multivariate time-series data. Multivariate time-series anomaly identification has gradually become one of the core technologies in equipment health management, production process monitoring, cybersecurity, and financial risk control. Early research often employed statistical modeling methods, such as control charts, ARIMA, and PCA, to model univariate or low-dimensional time series, judging anomalies through residuals or statistical deviations. However, these methods often had limited performance in high-dimensional, strongly correlated scenarios. In recent years, deep learning has gradually become the mainstream direction for time-series anomaly detection. Reconstruction-based or predictive methods based on autoencoders, variational autoencoders, generative adversarial networks, LSTM, and TCN structures have been widely used to characterize normal patterns and use reconstruction errors or prediction errors as anomaly scores. In multivariate scenarios, researchers have further introduced graph neural networks and attention mechanisms to learn the relationships between variables explicitly or implicitly. For example, models such as Graph Bias Network (GDN) and Graph Relationship Learning Network (GReLeN) are used to improve anomaly identification capabilities from the perspective of sensor dependencies. Meanwhile, Transformer-based temporal anomaly detection methods, such as Anomaly Transformer, TranAD, and spatiotemporal hybrid structure models, are constantly being proposed. These methods emphasize the use of self-attention mechanisms to capture long-range dependencies and complex spatiotemporal patterns, achieving superior detection performance compared to traditional methods on publicly available industrial process, network traffic, and log datasets. Overall, multivariate temporal anomaly detection technology is evolving from traditional statistical models to deep learning, especially attention and graph structure models, but it is still in a stage of continuous development.

[0003] While existing multivariate time-series anomaly identification methods have made some progress in terms of expressive power and detection accuracy, several common shortcomings remain. First, many methods simply concatenate variables from different time points and input them into LSTM, TCN, or Transformer models during feature construction. This implicitly assumes that the responses of variables to potential events at the same time step are synchronous, making it difficult to accurately reflect the intervariate response lags and sequence differences commonly found in industrial processes and network behavior. Consequently, asynchronous anomaly patterns are prone to being missed under complex operating conditions. Second, much work directly models at the original time-step granularity, lacking explicit modeling and encoding of local time segments. Variable identity and temporal location information are mostly implicitly integrated into the network structure, resulting in unclear hierarchical relationships between local causal evolution and global intervariate dependencies, limiting the ability to characterize the root causes of anomalies. Third, existing graph neural networks and Transformer methods mainly focus on static or weakly time-varying variable correlations, and their characterization of dynamically changing dependency structures and asynchronous association patterns between variables remains insufficient. While recent explorations have begun to apply large language models to time series data, these efforts have largely focused on textual descriptions, knowledge-based aids, or simple transfers, and are still immature in modeling complex asynchronous response relationships within original multivariate numerical sequences. In summary, existing technologies struggle to simultaneously and effectively integrate local causal evolution modeling within variables, asynchronous dependency modeling across variables, and global representation. This makes it difficult to identify multi-form multivariate time series anomaly patterns in a timely and accurate manner. Therefore, a multivariate time series data anomaly detection method and system based on large language models is needed to address these issues. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for detecting anomalies in multivariate time-series data based on a large language model, thus solving the aforementioned problems.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for detecting anomalies in multivariate time-series data based on a large language model, the method comprising: S1: acquiring multivariate time-series samples of a target object within a preset time window, wherein the multivariate time-series samples include at least two variable dimensions and two time steps.

[0008] S2: The multivariate time series sample is split into multiple variable time series according to the variable dimension. Each variable time series is divided into time segments according to the preset segment length and sliding step size to obtain multiple time segments of the variable. Each time segment is assigned a variable identity code and a time position code. The time segments are combined with the corresponding codes to form a first feature sequence that represents the local time series segment and its variable and time position information.

[0009] S3: Input the first feature sequence into the causal convolution block according to the variable dimension, perform causal convolution processing on the time series of each variable, and obtain the second feature sequence representing the local dynamic behavior of each variable.

[0010] S4: The second feature sequence is spliced ​​in a predetermined order to form a multivariate time series feature sequence, and the multivariate time series feature sequence is input into a global modeling module composed of a large language model to perform global sequence modeling on the multivariate time series feature sequence. The correlation and asynchronous response relationship between different variables and different time slices are modeled in the global scope of the sequence to obtain a third feature sequence containing global context information.

[0011] S5: Perform feature fusion and convergence on the third feature sequence and the multivariate time series feature sequence to obtain a global feature vector characterizing the overall behavioral characteristics of the multivariate time series sample, and input the global feature vector into the classification head network to output the anomaly identification result of the multivariate time series sample.

[0012] Furthermore, the preset segment length and sliding step size are L and R, respectively. When segmenting each variable time series of length T, a sliding window of length L and step size R is used to divide the multiple time segments sequentially from the beginning of the time series. When the length of the remaining time period at the end of the time series is less than L, the remaining time period is zero-padded to make up the length L to generate additional time segments, thereby avoiding the loss of information at the end of the time window.

[0013] Furthermore, the variable identity encoding includes a first set of embedding vectors established for each variable dimension, and the time position encoding includes a second set of embedding vectors established for each time slice. Both the first and second embedding vectors are learnable vectors of a preset dimension. When forming the first feature sequence, for each time slice, the first embedding vector corresponding to its variable and the second embedding vector corresponding to its time position are obtained. The first embedding vector and the second embedding vector are vector-added to obtain the encoded representation of the time slice, and the encoded representation is vector-superimposed with the temporal features of the time slice.

[0014] Furthermore, the causal convolutional block includes multiple one-dimensional causal convolutional layers connected sequentially along the time direction. Each one-dimensional causal convolutional layer obtains the corresponding temporal features by performing convolution calculations on the current time slice and its historical time slices. Residual connections and normalization processing are set between adjacent one-dimensional causal convolutional layers to maintain the stable transmission of local causal temporal features of each variable while expanding the temporal receptive field.

[0015] Furthermore, the global modeling module composed of the large language model is a pre-trained large language model based on a self-attention structure. When performing global sequence modeling on the multivariate temporal feature sequence, the main parameters of the pre-trained large language model are kept frozen, and low-rank adaptation parameter matrices are introduced into the projection matrices used to generate query vectors, key vectors, and value vectors in its attention sublayer. The low-rank adaptation parameter matrices are used as learnable parameters during training to perform task-related lightweight fine-tuning of the large language model without changing the main parameters.

[0016] Furthermore, the feature fusion includes: summing the third feature sequence and the multivariate time-series feature sequence element by element at corresponding sequence positions to obtain a fused feature sequence, and applying nonlinear activation to the fused feature sequence.

[0017] The feature aggregation includes: performing global pooling on the fused feature sequence along the sequence length dimension to obtain the global feature vector.

[0018] The classification head network includes at least one fully connected layer and a normalized activation function for outputting anomaly identification results.

[0019] Furthermore, the present invention also provides a multivariate time series data anomaly detection system based on a large language model. The system includes a data acquisition and preprocessing module for acquiring multivariate time series samples of a target object within a preset time window. The multivariate time series samples include at least two variable dimensions and two time steps.

[0020] The slicing and encoding module is used to split the multivariate time series sample into multiple variable time series according to the variable dimension, and to slice each variable time series according to a preset slicing length and sliding step size to obtain multiple time slices of the variable. The module assigns a variable identity code and a time position code to each time slice, and combines the time slice with the corresponding code to form a first feature sequence that represents the local time series segment and its variable and time position information.

[0021] The causal convolution modeling module inputs the first feature sequence into the causal convolution block according to the variable dimension, performs causal convolution processing on the time series of each variable, and obtains the second feature sequence representing the local dynamic behavior of each variable.

[0022] The large language model global modeling module is used to concatenate the second feature sequence in a predetermined order to form a multivariate temporal feature sequence, and input the multivariate temporal feature sequence into the global modeling module composed of the large language model to perform global sequence modeling on the multivariate temporal feature sequence. The module models the correlation and asynchronous response relationship between different variables and different time slices in the global scope of the sequence to obtain a third feature sequence containing global context information.

[0023] The feature fusion and anomaly detection module is used to fuse and converge the third feature sequence and the multivariate time series feature sequence to obtain a global feature vector that characterizes the overall behavioral features of the multivariate time series sample. The global feature vector is then input into the classification head network to output the anomaly detection result of the multivariate time series sample.

[0024] (III) Beneficial Effects

[0025] Compared with existing technologies, this invention provides a method and system for anomaly detection in multivariate time-series data based on a large language model, which has the following beneficial effects:

[0026] 1. This method and system for anomaly detection of multivariate time series data based on a large language model divides multivariate time series samples into time slices according to the variable dimension, introduces variable identity encoding and time position encoding for each time slice, and then combines causal convolutional blocks to perform local modeling of the time series of each variable. This enables the model to finely characterize the causal evolution process inside the variable at the local time slice level and explicitly perceive the structural information of "which variable it comes from and what time position it is in". This breaks through the modeling method of simply splicing the variables at the same time step and implicitly assuming synchronous response in the existing technology. By using sliding window segmentation and tail completion, structured local temporal segments are formed. Learnable variables and positional embedding vectors are then superimposed, and the temporal receptive field is expanded through multi-layer one-dimensional causal convolution and residual connections. This allows local features to retain temporal constraints while enhancing their sensitivity to slow evolution and precursory signals. This effectively alleviates the problems of insufficient local pattern representation and inadequate utilization of variable identity and temporal location information in background techniques, thereby improving the ability to capture asynchronous abnormal patterns under complex working conditions and characterize the root causes of anomalies.

[0027] 2. This method and system for anomaly detection in multivariate temporal data based on a large language model concatenates local temporal feature sequences extracted through causal convolution and inputs them into a global modeling module composed of a pre-trained large language model. Under the premise of freezing the main parameters, a low-rank adaptation parameter matrix is ​​introduced to perform lightweight fine-tuning of the attention sublayer, enabling the large language model to efficiently learn the global correlation and asynchronous response relationship between different variables and different time slices in the numerical temporal domain. Combined with operations such as residual fusion, nonlinear activation, and global pooling along the sequence dimension, local causal features and global asynchronous dependency representations are effectively converged. Finally, the anomaly identification result is output by the classification head network. Compared to existing graph neural networks or Transformer models, which mainly focus on static or weak time-varying correlations and struggle to adaptively characterize cross-variable dependency structures that change dynamically with the scene, this invention leverages the powerful long-sequence modeling capabilities and efficient parameter adaptation mechanisms of large language models. While ensuring controllable training and deployment costs, it enhances the modeling capabilities for complex time-varying dependencies and asynchronous anomaly patterns, improving the accuracy, recall, and robustness of multivariate time-series anomaly identification. It is suitable for various application scenarios such as industrial process monitoring, IoT operation and maintenance, and network security. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall process of a multivariate time-series data anomaly detection method based on a large language model provided by the present invention.

[0029] Figure 2 This is a schematic diagram of the causal convolutional block structure of a multivariate time-series data anomaly detection method based on a large language model provided by the present invention.

[0030] Figure 3 This is a schematic diagram of the bidirectional attention mechanism of a multivariate time-series data anomaly detection method based on a large language model provided by the present invention.

[0031] Figure 4 This is a schematic diagram of the LoRA fine-tuning structure for a multivariate time-series data anomaly detection method based on a large language model provided by the present invention.

[0032] Figure 5 This invention provides a model architecture diagram for a multivariate time-series data anomaly detection system based on a large language model. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Please see Figure 1 , Figure 1 This invention provides an overall flowchart of a multivariate time-series data anomaly detection method based on a large language model. The method includes: S1: acquiring multivariate time-series samples of a target object within a preset time window, wherein the multivariate time-series samples include at least two variable dimensions and two time steps.

[0036] S2: The multivariate time series sample is split into multiple variable time series according to the variable dimension. Each variable time series is divided into time segments according to the preset segment length and sliding step size to obtain multiple time segments of the variable. Each time segment is assigned a variable identity code and a time position code. The time segments are combined with the corresponding codes to form a first feature sequence that represents the local time series segment and its variable and time position information.

[0037] S3: Input the first feature sequence into the causal convolution block according to the variable dimension, perform causal convolution processing on the time series of each variable, and obtain the second feature sequence representing the local dynamic behavior of each variable.

[0038] S4: The second feature sequence is spliced ​​in a predetermined order to form a multivariate time series feature sequence, and the multivariate time series feature sequence is input into a global modeling module composed of a large language model to perform global sequence modeling on the multivariate time series feature sequence. The correlation and asynchronous response relationship between different variables and different time slices are modeled in the global scope of the sequence to obtain a third feature sequence containing global context information.

[0039] S5: Perform feature fusion and convergence on the third feature sequence and the multivariate time series feature sequence to obtain a global feature vector characterizing the overall behavioral characteristics of the multivariate time series sample, and input the global feature vector into the classification head network to output the anomaly identification result of the multivariate time series sample.

[0040] In this embodiment, the overall process of a multivariate time-series data anomaly detection method based on a large language model is as follows: Figure 1As shown, in step S1, multivariate time-series samples are collected from target objects such as industrial production lines, IoT terminals, or network devices within a preset time window. Each sample contains at least two variable dimensions and two time steps to characterize the operating status of the device within that time period. In step S2, the collected multivariate time-series samples are split into several variable time series according to the variable dimensions. Each variable time series is time-sliced ​​according to a preset slice length and sliding step size, dividing the original long sequence into continuous local time-series segments. Variable identity codes and time position codes are assigned to each time slice. The codes are combined with the time slices to form the first feature sequence, so that each segment carries both its sensor identity and its relative position within the time window, thereby providing variable and time structure information for subsequent explicit network processing. In step S3, the first feature sequence is input into a causal convolutional block according to the variable dimensions for processing. The causal convolutional block extracts the local dynamic change patterns and causal evolution relationships within each variable layer by performing convolution operations using only the current and historical time slices, outputting a representation of the local dynamic behavior of each variable. The second feature sequence is used to make the model more sensitive to time-related features such as gradual trends and early warning signs. In step S4, the second feature sequences of different variables are concatenated into a unified multivariate time-series feature sequence in a predetermined order and input into a global modeling module composed of a pre-trained large language model. This module uses a self-attention structure to perform global dependency modeling between variables and time segments in a long sequence range, automatically learning the correlation and asynchronous response relationships between different sensor signals and different time segments, thereby obtaining a third feature sequence containing global context information, realizing a unified representation of complex spatiotemporal correlation patterns. In step S5, the third feature sequence is fused and converged with the original multivariate time-series feature sequence to obtain a global feature vector that can comprehensively reflect the overall behavioral characteristics of the sample. This global feature vector is input into the classification head network for discrimination and outputs the corresponding anomaly identification results, such as normal state, different fault types, or different anomaly categories. This enables high-precision automated anomaly detection of multivariate time-series data in scenarios such as industrial monitoring, equipment health management, IoT operation and maintenance, and network security monitoring.

[0041] Furthermore, the preset segment length and sliding step size are respectively and For length of When performing time-slicing on each variable's time series, a length of [length missing] is used. Step size is The sliding window divides the time series into multiple time slices sequentially from the beginning of the time series; when the length of the remaining time period at the end of the time series is insufficient... When the remaining time period is filled with zeros to complete the length L, additional time slices are generated, thereby avoiding the loss of information at the end of the time window.

[0042] In this embodiment, for a length of Any variable time series When performing time-sharing, the preset slice length is: The sliding step size is Using a length of Step size is The sliding window moves sequentially from the beginning of the time series, dividing the original sequence into several local time segments. , of which Each time slice can be represented as Starting index Slide the window by step Incremental, in one implementation, the number of fragments Based on the length of the time window Slice length With sliding step size The relationship is determined according to a preset formula, for example, it can be determined by... Configure settings to ensure the time window is covered as completely as possible during the sliding process; when the window slides to the end of the time series, if the length of the remaining time period is less than the slice length... Then, the remaining time period will be padded with zeros at the end of the time axis to complete the length. Later used as additional time slices By adding the data to the set of time segments, the problem of discarding effective information at the end of the time window due to insufficient length is avoided while maintaining consistency in all time segment dimensions. This allows subsequent processing such as variable identity encoding, time position encoding, causal convolution, and global modeling of large language models to be performed on local time segments of uniform length, ensuring the operability of the method in engineering implementation and the integrity of the utilization of time information.

[0043] Furthermore, the variable identity encoding includes a first set of embedding vectors established for each variable dimension, and the time position encoding includes a second set of embedding vectors established for each time slice. Both the first and second embedding vectors are learnable vectors of a preset dimension. When forming the first feature sequence, for each time slice, the first embedding vector corresponding to its variable and the second embedding vector corresponding to its time position are obtained. The first embedding vector and the second embedding vector are vector-added to obtain the encoded representation of the time slice, and the encoded representation is vector-superimposed with the temporal features of the time slice.

[0044] In this embodiment, for multivariate time series samples within a preset time window, it is assumed that there are a total of Each variable dimension, and each variable is divided into [variable dimensions] in step S2. The time segment, record the first time segment The first variable The temporal feature vector of each time slice is ,in , The variable identity encoding is achieved by establishing a first embedding vector for each variable dimension, that is, constructing a first embedding vector set. The time-position encoding is achieved by establishing a second embedding vector for each time slice position, that is, constructing a set of second embedding vectors. ,in , All are preset dimensions The learnable vectors are updated along with other model parameters through backpropagation during network training; when forming the first feature sequence, for any time slice... First, retrieve the variable identity embedding vector corresponding to its own variable from the first embedding vector set. Then, extract the time position embedding vector corresponding to its time position from the second embedding vector set. The encoded representation of the time slice is obtained by vector addition of the two. Then, this encoded representation is superimposed with the temporal feature vectors of the time slices, for example, to obtain the feature vector in the first feature sequence. In this way, each local temporal segment carries both its physical "variable identity information" and "temporal location information" before entering the causal convolutional block and the large language model. This avoids confusion of different variable features in high-dimensional space and enables the model to explicitly distinguish the response of the same variable at different time positions, providing a clear structured input representation for subsequent local causal modeling and global asynchronous dependency modeling.

[0045] For further details, please refer to Figure 2 , Figure 2 This invention provides a schematic diagram of a causal convolutional block structure for a multivariate temporal data anomaly detection method based on a large language model. The causal convolutional block includes multiple one-dimensional causal convolutional layers connected sequentially along the time direction. Each one-dimensional causal convolutional layer obtains the corresponding temporal features by performing convolution calculations on the current time slice and its historical time slices. Residual connections and normalization processing are set between adjacent one-dimensional causal convolutional layers to maintain the stable transmission of local causal temporal features of each variable while expanding the temporal receptive field.

[0046] In this embodiment, for ease of explanation, it is assumed that after steps S2 to S3, for any variable The first feature sequence can be represented as a vector sequence arranged in time slice order. ,in Characterizing the variable in the first place Local features on each time slice; the causal convolutional block works independently for each variable's time series, and consists of multiple one-dimensional causal convolutional layers connected sequentially along the time direction, denoted as the input sequence of the l-th causal convolutional layer. The output sequence is In the initial layer there is For any time position The l-th layer one-dimensional causal convolution only applies the convolution operation to the current and historical positions, for example, it can be represented as ,in The kernel size is [size]. This is the expansion rate of the layer (zero padding can be used when the index is less than 1). and The learnable convolution parameters for this layer, This is the nonlinear activation function following a one-dimensional causal convolution. Based on this, the convolution output is superimposed back onto the input through residual connections and normalization, for example, it can be expressed as... ,in Normalization operators such as layer normalization can be applied to maintain the stability of the feature distribution while expanding the temporal receptive field layer by layer. This is achieved through stacking multiple layers of one-dimensional causal convolutions, with each layer having a different dilation rate. Causal convolutional blocks, while strictly adhering to temporal causality, aggregate information from the current time slice and different historical time slices from near to far, gradually capturing the multi-scale dynamic behavior of each variable, from short-term fluctuations to long-term trends, and outputting a second feature sequence. That is, the characterization variable The high-level representation of local causal temporal features provides input features with clear structure and explicit temporal order for subsequent global modeling of large language models, thereby ensuring that the present invention can fully explore the dynamic evolution patterns inside variables in asynchronous response scenarios.

[0047] For further details, please refer to Figure 3-4 , Figure 3 A schematic diagram of the bidirectional attention mechanism of a multivariate time-series data anomaly detection method based on a large language model provided by the present invention; Figure 4This diagram illustrates the LoRA fine-tuning structure of a multivariate temporal data anomaly detection method based on a large language model, as provided by this invention. The global modeling module, composed of a large language model, is a pre-trained large language model based on a self-attention structure. When performing global sequence modeling on the multivariate temporal feature sequence, the main parameters of the pre-trained large language model are frozen. Low-rank adaptation parameter matrices are introduced into the projection matrices used to generate query vectors, key vectors, and value vectors in its attention sublayer. These low-rank adaptation parameter matrices serve as learnable parameters during training, enabling task-related lightweight fine-tuning of the large language model without altering the main parameters.

[0048] In this embodiment, the global modeling module, composed of a large language model, employs a pre-trained large language model based on a self-attention structure to perform bidirectional global modeling on the multivariate temporal feature sequence processed by causal convolutional blocks. Specifically, the second feature sequence obtained in step S3 is expanded and reorganized into a one-dimensional token sequence according to the variable and temporal order. Each of them For the first The feature vectors of each time segment are first obtained by linear projection in the self-attention sublayer of the large language model to obtain the query vector, key vector, and value vector, i.e. ,in The main parameter matrix is ​​obtained through pre-training and frozen in this invention; subsequently, attention weights and output are calculated according to a bidirectional self-attention mechanism. Each position vector can simultaneously focus on features at other positions throughout the sequence, thereby capturing the correlation and asynchronous response relationships between different variables and different time slices globally. To adapt to the multivariate temporal anomaly recognition task of this invention without compromising the core capabilities of the pre-trained large language model, this embodiment introduces a LoRA low-rank adaptation structure on the projection matrix of the attention sublayer, i.e., maintaining... The parameters are frozen and remain unchanged, and are respectively superimposed on their low-rank parameter matrix pairs. For example, the effective weight of the query branch can be expressed as... ,in , For rank The learnable matrix, with key and value branches set up similarly. , Only update during training , Low-rank fitting parameters are used to achieve task-related lightweight fine-tuning of the pre-trained large language model with a small number of parameters; after layer processing through multi-head attention, residual connections, and feedforward networks, the resulting output sequence is obtained. That is, the third feature sequence containing global context information, where each correspond Figure 3Output token In this embodiment, the input and output lengths of the self-attention layer are the same, that is... This method incorporates global dependency information across variables and time segments while retaining the advantages of pre-trained models in long sequence modeling. It provides a more expressive temporal representation for subsequent feature fusion and anomaly detection steps, thereby improving the accuracy and robustness of multivariate temporal anomaly identification.

[0049] Furthermore, the feature fusion includes: summing the third feature sequence and the multivariate time-series feature sequence element by element at corresponding sequence positions to obtain a fused feature sequence, and applying nonlinear activation to the fused feature sequence;

[0050] The feature aggregation includes: performing global pooling on the fused feature sequence along the sequence length dimension to obtain the global feature vector;

[0051] The classification head network includes at least one fully connected layer and a normalized activation function for outputting anomaly identification results.

[0052] In this embodiment, the third feature sequence can be obtained after processing by the global modeling module of the large language model. ,in Characterizing the first The global context features of each time-series token, and the corresponding multivariate time-series feature sequence are denoted as follows: ,in The vectors at the same positions of the second feature sequence obtained by concatenation before inputting into the large language model; the feature fusion includes element-wise summation of the third feature sequence and the multivariate temporal feature sequence at corresponding sequence positions, for example, for any position Calculate the fusion vector And then Apply a nonlinear activation function , to obtain the fused feature sequence For example, common activation functions such as ReLU or LeakyReLU can be used to fully integrate the global dependency information contained in the third feature sequence with the local detail information in the original temporal features in the vector space, thereby improving the discriminative power and nonlinear expressive ability of the feature representation; the feature convergence includes, for the fused feature sequence Global pooling is performed along the sequence length dimension to obtain a global feature vector that does not change with the sequence length. One implementation method can use global average pooling, i.e. Alternatively, global max pooling or a combination of averaging and max pooling can be used to compress the sequence dimension while preserving the most representative overall behavioral patterns; the classification head network uses global feature vectors The input consists of at least one fully connected layer and a normalized activation function for outputting the anomaly detection result. For example, a weight matrix can be set for a multi-class anomaly detection task. Bias vector Calculate category scores Then, the predicted probabilities of each category are obtained through softmax activation. For binary classification or single-class anomaly detection tasks, sigmoid activation can be used to output the anomaly probability. This realizes the mapping of the high-dimensional global time series representation to the final anomaly identification result label or confidence score, and completes the automatic anomaly identification of multivariate time series samples.

[0053] For further details, please refer to Figure 5 , Figure 5 This invention provides a model architecture diagram for a multivariate time-series data anomaly detection system based on a large language model. The invention also provides a multivariate time-series data anomaly detection system based on a large language model, the system comprising: a data acquisition and preprocessing module, used to acquire multivariate time-series samples of a target object within a preset time window, wherein the multivariate time-series samples include at least two variable dimensions and two time steps.

[0054] The slicing and encoding module is used to split the multivariate time series sample into multiple variable time series according to the variable dimension, and to slice each variable time series according to a preset slicing length and sliding step size to obtain multiple time slices of the variable. The module assigns a variable identity code and a time position code to each time slice, and combines the time slice with the corresponding code to form a first feature sequence that represents the local time series segment and its variable and time position information.

[0055] The causal convolution modeling module inputs the first feature sequence into the causal convolution block according to the variable dimension, performs causal convolution processing on the time series of each variable, and obtains the second feature sequence representing the local dynamic behavior of each variable.

[0056] The large language model global modeling module is used to concatenate the second feature sequence in a predetermined order to form a multivariate temporal feature sequence, and input the multivariate temporal feature sequence into the global modeling module composed of the large language model to perform global sequence modeling on the multivariate temporal feature sequence. The module models the correlation and asynchronous response relationship between different variables and different time slices in the global scope of the sequence to obtain a third feature sequence containing global context information.

[0057] The feature fusion and anomaly detection module is used to fuse and converge the third feature sequence and the multivariate time series feature sequence to obtain a global feature vector that characterizes the overall behavioral features of the multivariate time series sample. The global feature vector is then input into the classification head network to output the anomaly detection result of the multivariate time series sample.

[0058] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0059] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for anomaly detection in multivariate time-series data based on a large language model, characterized in that: The method includes: S1: Obtain multivariate time-series samples collected by the target object's sensor within a preset time window from industrial production lines, IoT terminals, or network devices; S2: The collected multivariate time series samples are split into multiple variable time series according to the variable dimension. Each variable time series is divided into time segments according to the preset segment length and sliding step size. The original long sequence is divided into continuous local time segments. Variable identity code and time position code are assigned to each time segment. The code is combined with the time segment to form the first feature sequence, so that each segment carries both its sensor identity and its relative position in the time window. S3: The first feature sequence is input into the causal convolution block according to the variable dimension for processing. The causal convolution block performs convolution operation using only the current and historical time slices, extracts the local dynamic change pattern and causal evolution relationship inside each variable layer by layer, and outputs the second feature sequence used to characterize the local dynamic behavior of each variable. S4: The second feature sequence is spliced ​​in a predetermined order to form a multivariate time series feature sequence, and the multivariate time series feature sequence is input into a global modeling module composed of a large language model to perform global sequence modeling on the multivariate time series feature sequence. The correlation and asynchronous response relationship between different variables and different time slices are modeled in the global scope of the sequence to obtain a third feature sequence containing global context information. S5: Perform feature fusion and convergence on the third feature sequence and the multivariate time series feature sequence to obtain a global feature vector, and input the global feature vector into the classification head network to output the anomaly identification result of the multivariate time series sample.

2. The method for anomaly detection in multivariate time-series data based on a large language model according to claim 1, characterized in that: The preset segment length and sliding step size are respectively and For length of When performing time-slicing on each variable's time series, a length of [length missing] is used. Step size is The sliding window divides the time series into multiple time slices sequentially from the beginning of the time series; when the length of the remaining time period at the end of the time series is insufficient... When the remaining time period is filled with zeros to complete the length L, additional time slices are generated, thereby avoiding the loss of information at the end of the time window.

3. The method for anomaly detection in multivariate time-series data based on a large language model according to claim 1, characterized in that: The variable identity encoding includes a first set of embedding vectors established for each variable dimension, and the time position encoding includes a second set of embedding vectors established for each time slice. Both the first and second embedding vectors are learnable vectors of a preset dimension. When forming the first feature sequence, for each time slice, the first embedding vector corresponding to its variable and the second embedding vector corresponding to its time position are obtained. The first embedding vector and the second embedding vector are vector-added to obtain the encoding representation of the time slice, and the encoding representation is vector-superimposed with the temporal features of the time slice.

4. The method for anomaly detection in multivariate time-series data based on a large language model according to claim 1, characterized in that: The causal convolutional block includes multiple one-dimensional causal convolutional layers connected sequentially along the time direction. Each one-dimensional causal convolutional layer obtains the corresponding temporal features by performing convolution calculations on the current time slice and its historical time slices. Residual connections and normalization processing are set between adjacent one-dimensional causal convolutional layers to maintain the stable transmission of local causal temporal features of each variable while expanding the temporal receptive field.

5. The method for anomaly detection in multivariate time-series data based on a large language model according to claim 1, characterized in that: The global modeling module, composed of a large language model, is a pre-trained large language model based on a self-attention structure. When performing global sequence modeling on the multivariate temporal feature sequence, the main parameters of the pre-trained large language model are kept frozen, and low-rank adaptation parameter matrices are introduced into the projection matrices used to generate query vectors, key vectors, and value vectors in its attention sublayer. The low-rank adaptation parameter matrices are used as learnable parameters during training to perform task-related lightweight fine-tuning of the large language model without changing the main parameters.

6. The method for anomaly detection in multivariate time-series data based on a large language model according to claim 1, characterized in that: The feature fusion includes: summing the third feature sequence and the multivariate time-series feature sequence element by element at corresponding sequence positions to obtain a fused feature sequence, and applying nonlinear activation to the fused feature sequence; The feature aggregation includes: performing global pooling on the fused feature sequence along the sequence length dimension to obtain the global feature vector; The classification head network includes at least one fully connected layer and a normalized activation function for outputting anomaly identification results.

7. A multivariate time-series data anomaly detection system based on a large language model, characterized in that, The system includes: The data acquisition and preprocessing module is used to acquire multivariate time series samples of the target object sensor within a preset time window from industrial production lines, IoT terminals or network devices; The slicing and encoding module is used to split the collected multivariate time series samples into multiple variable time series according to the variable dimension. Each variable time series is sliced ​​according to the preset slicing length and sliding step size, dividing the original long sequence into continuous local time series segments. Variable identity code and time position code are assigned to each time segment. The code is combined with the time segment to form the first feature sequence, so that each segment carries both its own sensor identity and its relative position in the time window. The causal convolution modeling module inputs the first feature sequence into the causal convolution block according to the variable dimension for processing. The causal convolution block performs convolution operations using only the current and historical time slices, extracting the local dynamic change patterns and causal evolution relationships within each variable layer by layer, and outputting the second feature sequence used to characterize the local dynamic behavior of each variable. The large language model global modeling module is used to concatenate the second feature sequence in a predetermined order to form a multivariate temporal feature sequence, and input the multivariate temporal feature sequence into the global modeling module composed of the large language model to perform global sequence modeling on the multivariate temporal feature sequence. The module models the correlation and asynchronous response relationship between different variables and different time slices in the global scope of the sequence to obtain a third feature sequence containing global context information. The feature fusion and anomaly detection module is used to fuse and converge the third feature sequence and the multivariate time series feature sequence to obtain a global feature vector, and input the global feature vector into the classification head network to output the anomaly detection result of the multivariate time series sample.

Images