Abnormality detection method based on time convolution network and converter

Abstract

The invention belongs to the technical field of deep learning application, and discloses an anomaly detection method based on a time convolutional network and a converter. The method comprises the following steps: firstly, carrying out preprocessing and feature extraction on a CERT r4.2 data set; performing data windowing according to the timeline of each user; constructing a model by using a time convolutional network, a converter and a classifier; dividing the data set into a training set and a test set, inputting data in the training set and windowed data taking a user as a unit into the constructed model and the training model, finally inputting data in the test set into the trained model, detecting abnormal behaviors, and counting the working performance of the training model. According to the method, the local features generated by the time convolution network and the global dependency information provided by the converter are effectively fused, so that more comprehensive feature representation is formed, the support capability of a subsequent classifier is enhanced, the overall detection performance of long-time malicious behaviors is improved, and the method has a relatively good application value.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, specifically to an anomaly detection method based on temporal convolutional networks and transformers. Background Technology

[0002] In the field of cybersecurity, insider threat detection remains a highly challenging problem, and insider attacks have become a serious challenge for many organizations. Related data shows that as early as 2018, 90% of organizations faced the risk of insider attacks; in 2019, approximately 60% of organizations experienced such attacks; and by 2023, 74% of organizations acknowledged an increase in the frequency of insider attacks. Because insiders have authorization access to organizational assets, they are often more likely than external attackers to compromise the confidentiality, availability, or integrity of data, causing damage up to 20 times greater than that caused by external attackers, with an average cost of up to $4.5 million per data breach.

[0003] The motivations behind insider attacks are complex and diverse, primarily including financial gain, retaliation, anger, stress, and organizational politics. However, insider threats are not limited to malicious employees; approximately 66% of insider threats are unintentionally caused by negligent employees. Among insider threat perpetrators, ordinary employees account for 49%, privileged network users for 59%, and contractors for 52%. Numerous factors contribute to insider attacks, such as the increasing number of users with unnecessary access privileges, the growing number of devices accessing confidential data, rapidly increasing technological complexity, insufficient user awareness and training, and the increasing volume of sensitive data. The potential losses from successful insider attacks are enormous, and they are often only discovered after significant damage has occurred. This makes the increasing trend of insider security vulnerabilities one of the most pressing challenges in the global data security field.

[0004] The rapid development of the internet is a double-edged sword for businesses and government agencies. On the one hand, it significantly improves operational efficiency; on the other hand, it opens new avenues for cyber attackers, exacerbating the problem of insider threats. Currently, various anomaly detection methods are applied to detect insider threats, mainly divided into machine learning-based methods and deep learning-based methods. Deep learning, as a currently popular research field, is applied to various security frameworks due to its significant advantages. It can be used for both supervised and unsupervised learning, and its algorithms outperform traditional machine learning algorithms in both performance and accuracy. Therefore, deep learning-based algorithms can be used to improve the effectiveness of insider threat detection, achieving results with high accuracy and lower false positive rates, helping organizations establish robust insider threat detection mechanisms.

[0005] In today's organizational environment, preventing intentional or unintentional threats from within has become a top priority. Enterprises have deployed detection systems and confidentiality measures to manage internal security risks. However, some existing detection methods have limitations. For example, in current research, insider threat detection is mainly considered a user classification problem, where the system explicitly determines whether to label an individual as a malicious user. However, this approach has two major limitations: first, detection only occurs after sufficient evidence has accumulated; second, it cannot distinguish between accidental and intentional abuse by the same user, making it ineffective against malicious employees who commit harmful acts and immediately leave the company. Some researchers have shifted from post-hoc detection to instance-based detection. For example, Li Dongyang et al. achieved AUC scores of 89.36% (instance-based) and 94.56% (user-based) using their MAITD model. Similarly, Cai Xiangrui et al. obtained AUC scores of 90.86% (instance-based) and 93.06% (post-hoc) on the CERT R4.2 dataset using a transformer + graph convolutional network, with corresponding detection rates (DR) of 82.17% and 89.34%, respectively. This suggests that instance-based detection is more challenging than post-hoc user-based detection.

[0006] In the field of internal threat detection, we use the CERT dataset for our research. However, the number of malicious samples in this dataset is extremely small, creating a huge disparity with the number of normal samples, resulting in a severe imbalance in the sample size. This problem severely interferes with model training and evaluation, making most detection models easily "misled" in this imbalanced data environment. Models often become overly biased towards the majority class (normal behavior), leading to artificially inflated detection rates during evaluation. In other words, while the model may appear to have high detection accuracy, its actual ability to identify key malicious behaviors is severely insufficient, significantly weakening the actual effectiveness of the detection system.

[0007] Currently, most related research focuses on post-event detection, using comprehensive analysis of all employee logs to determine the presence of malicious behavior. However, this detection method has significant limitations in practical applications, often failing to promptly detect and prevent ongoing threatening behaviors, exhibiting a clear lag.

[0008] To address this deficiency, instance-based detection methods have emerged, attempting to make real-time judgments on individual behavioral instances. Unfortunately, the accuracy and other evaluation metrics obtained from these studies are still unsatisfactory and cannot meet the urgent need for high-precision detection in practical applications.

[0009] The CERT dataset contains a malicious scenario where a malicious employee attempts to evade detection through prolonged malicious activity (approximately two months). This long-term, covert pattern of malicious behavior undoubtedly increases the difficulty of detection. Malicious employees deliberately disperse their malicious actions to avoid triggering the monitoring system's alert in a short period, making it difficult for detection models to capture complete and obvious malicious behavior patterns, thus easily leading to missed detections.

[0010] The behavioral patterns of insider threats are not static but highly diverse and dynamic. Different malicious employees may employ drastically different behavioral strategies to achieve their goals, and these strategies continuously adjust over time and with changing environments. This necessitates that detection models possess strong adaptability and generalization capabilities, enabling them to promptly capture subtle changes in behavioral patterns and accurately identify potential threats. This presents significant challenges to the design and optimization of such models.

[0011] In summary, current insider threat detection suffers from problems such as imbalanced datasets, challenges in detection methods and timeliness, high false negative rates for long-term malicious behavior, and the diversity and dynamism of insider behavior patterns. If a technology could overcome these shortcomings, it would greatly improve the detection rate of malicious insider behavior. Summary of the Invention

[0012] To address the challenges of detecting prolonged malicious behavior and the unsatisfactory performance of instance-based detection, this invention provides a method combining a transformer with a temporal convolutional network. By effectively fusing local features generated by the temporal convolutional network with global dependency information provided by the transformer, a more comprehensive feature representation is formed. This enhanced representation strengthens the support capabilities of subsequent classifiers and improves overall detection performance. This invention is implemented through the following technical solution, including the following steps:

[0013] Step 1: Preprocess and extract features from the CERT r4.2 dataset; the preprocessing includes: data integration, missing data filling, and data concatenation; the feature extraction includes: daily data aggregation, user information retrieval, and daily behavioral feature extraction.

[0014] Step 2: Perform data windowing according to each user's timeline; the data windowing process includes: performing dimensionality reduction on the original feature data using linear discriminant analysis; partitioning and sorting the data for each user in chronological order;

[0015] Step 3: Model Construction; The model includes a temporal convolutional network, a transformer, and a classifier; The temporal convolutional network consists of 4 temporal convolutional network blocks, the transformer consists of 2 transformer layers, and the classifier predicts and classifies the input sequence at each time step;

[0016] Step 4: Divide the data into training and testing sets; input windowed data of users into the model, using users as the unit, to train the model;

[0017] Step 5: Input the test set data into the model trained in Step 4 using windowed data, with each user as a unit, to detect abnormal behavior and analyze the performance of the detection model.

[0018] Preferably, the data integration process in step one is as follows:

[0019] (1) Iterate through each unique user identifier;

[0020] (2) Filter out records containing a specific user identifier from the five log types;

[0021] (3) Sort these records according to the time information in the date field.

[0022] Preferably, in step one, missing data is filled in, and the missing values ​​are obtained by calculating the estimated mean of the relevant features.

[0023] Preferably, the feature extraction process in step one includes the following steps:

[0024] (1) Daily data aggregation: All activity files, including devices, emails, documents, web pages, and logins, are summarized daily. This aggregation process organizes the data into daily segments and generates pickle files categorized by date.

[0025] (2) User information retrieval: obtain comprehensive user information from multiple CSV files in the LDAP directory, including relevant characteristics and malicious user tags; build a Pandas data frame containing all user characteristics, providing a comprehensive view of each employee profile;

[0026] (3) Daily behavioral feature extraction: Convert the daily behavioral data recorded in the data frame into statistical and frequency features; In addition to the provided psychological measurements and role-based attributes, extract various statistical and frequency features of employee behavioral characteristics, including: number of activities, login frequency, number of file operations, number of emails sent, number of web page requests, activity duration, email / file length, and specific category behaviors;

[0027] (4) Save the extracted features as a CSV file.

[0028] Preferably, the temporal convolutional network part of the model in step three consists of four temporal convolutional network blocks, each of which contains the following structure:

[0029] (1) One-dimensional causal convolution: Causal convolution operation only considers the input of the current and previous time steps, deliberately ignoring future data, thus ensuring the causality of the model and preserving the inherent temporal characteristics of time series data;

[0030] (2) Random discarding: Randomly set the output of some neurons to zero to prevent overfitting;

[0031] (3) Normalization: Stabilizes the distribution of activation values ​​in each layer;

[0032] (4) Residual connection: By skipping a specific layer and directly superimposing its input into the output, it effectively alleviates the gradient vanishing or exploding problem in deep neural networks, accelerates the training process, and helps the network learn identity mappings more efficiently.

[0033] Dilated convolution is achieved by stacking four temporal convolutional network blocks, and the receptive field is expanded by adjusting the convolutional kernel size or increasing the dilation factor.

[0034] Preferably, the transformer layer of the model in step three includes the following structure:

[0035] (1) Position Encoding: The output of the temporal convolutional network is fed into the position encoding module to inject positional information into the sequence. The model strengthens the temporal order by adding additional positional data. The position encoding is calculated using a sine or cosine function. For each position t and dimension m, its encoding PE(t,m) is:

[0036] Even indices use a sine function:

[0037] Odd indices use a cosine function: In the formula, d represents the feature dimension, and location information is incorporated into the input data in the following way: x pos =x output +PE;

[0038] (2) Multi-head self-attention: The input is transformed linearly to generate a query (Q), key (K), and value (V) matrix; Q = x pos W Q K = x pos W K V = x pos W V In the formula, W Q W K W V is a trainable weight matrix;

[0039] (3) Attention weight calculation: A represents the attention output, Attention(Q, K, V) represents the attention weights, and d k This represents the dimension of the K vector. The similarity between Q and each K is calculated, and after passing through softmax, it becomes a weight distribution, i.e., an attention distribution. Finally, multiplying by V means applying these attention weights to the corresponding position V to obtain the final output A.

[0040] (4) After residual connection and normalization: X attn =LayerNorm(x pos +A); where X attn This represents the output of each attention head, and LayerNorm represents performing layer normalization on the current input along the feature dimension to standardize its mean and variance.

[0041] (5) Calculate multi-head attention using multiple independent attention heads. Repeat step (4) above and record the attention output of each head as head. i (i = 1, 2, ..., h); Concatenate the attention output of all heads into MultiHead: MultiHead = Concat(head1, head2, ..., head...). h W O In the formula, h is the number of attention heads, Concat means concatenating the outputs of multiple heads, and W O It is the output projection matrix;

[0042] (6) Feedforward network layer: The feedforward layer receives the MultiHead output of the multi-head attention and performs a non-linear mapping at each time step. That is, the features are first stretched to a wider representation space, then compressed back to the original dimension after linear rectified activation; subsequently, they are randomly discarded, residual connected and layer normalized.

[0043] Preferably, the model training in step four adopts a tiered training method. In the first stage, the model learns the normal behavior patterns of normal users using segmented data from normal users. In the second stage, 80% of the segmented data from malicious users in three scenarios is used for training, and the remaining 20% ​​is used for testing.

[0044] The technical solution of this invention has the following beneficial effects:

[0045] 1) This invention employs a time-series data representation method that integrates the time dimension. This method emphasizes identifying potential malicious behavior dynamics by capturing the changing trends of user behavior over time. Specifically, the method involves stitching together data points from consecutive time periods or comparing the behavioral patterns of the same user in adjacent time periods. This strategy can effectively reveal potential behavioral changes, thereby identifying potential behavioral shifts indicating malicious activity.

[0046] 2) This invention proposes an anomaly detection model that combines a temporal convolutional network with a transformer. It detects malicious behavior by identifying operations that deviate from the preceding and following time steps within a sliding time series window, achieved by learning patterns in user actions within these windows.

[0047] 3) The model proposed in this invention performs daily checks within a window, enabling timely detection of abnormal employee behavior. This allows organizations and enterprises to detect and resolve problems promptly after they occur, offering greater practical value compared to post-event user detection. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of a data preprocessing and feature extraction system.

[0049] Figure 2 This is a diagram illustrating the data windowing process.

[0050] Figure 3 This is a diagram of the model structure.

[0051] Figure 4 The training diagram for the model in stages. Detailed Implementation

[0052] The technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this invention, not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0053] This invention provides an anomaly detection method based on temporal convolutional networks and transformers. By effectively fusing local features generated by the temporal convolutional network with global dependency information provided by the transformer, a more comprehensive feature representation is formed. This enhanced representation strengthens the support capability of subsequent classifiers and improves overall detection performance. The technical solution of this invention includes the following steps:

[0054] Step 1: Preprocess and extract features from the CERT r4.2 dataset; the preprocessing includes: data integration, missing data filling, and data concatenation; the feature extraction includes: daily data aggregation, user information retrieval, and daily behavioral feature extraction.

[0055] Step 2: Perform data windowing according to each user's timeline; the data windowing process includes: performing dimensionality reduction on the original feature data using linear discriminant analysis; partitioning and sorting the data for each user in chronological order;

[0056] Step 3: Model Construction; The model includes a temporal convolutional network, a transformer, and a classifier; The temporal convolutional network consists of 4 temporal convolutional network blocks, the transformer consists of 2 transformer layers, and the classifier predicts and classifies the input sequence at each time step;

[0057] Step 4: Divide the data into training and testing sets; input windowed data of users into the model, using users as the unit, to train the model;

[0058] Step 5: Input the test set data into the model trained in Step 4 using windowed data, with each user as a unit, to detect abnormal behavior and analyze the performance of the detection model.

[0059] Step 1: Preprocess and extract features from the CERT r4.2 dataset.

[0060] The dataset considered in this invention is the open-source CMU-CERT (Carnegie Mellon University Computer Emergency Response Team) Insider Threat r4.2 dataset. While the CMU-CERT dataset has since been updated to version r6.2, version r4.2 is the version commonly used in many studies. Version r4.2 contains activity logs from 1000 employees over 18 months, totaling 3.22 million log entries, of which 7423 were flagged as malicious activity.

[0061] The CERT dataset contains five behavioral log types, each with a large amount of data. The following process will integrate this data:

[0062] • Iterate through each unique user identifier.

[0063] • Filter records containing a specific user identifier from five log types.

[0064] Sort these records based on the time information in the date field.

[0065] In the CERT dataset, data collection primarily relies on sensors, leading to a degree of data gaps. These missing data can affect the proper functioning of the algorithms. This process addresses this issue by filling in missing values ​​and calculating estimated means of relevant features. This step is crucial for ensuring the integrity of the dataset and the effectiveness of the algorithms. This approach mitigates the missing data problem, providing a robust data foundation for subsequent internal threat detection and cybersecurity analysis.

[0066] Based on the fact that internal personnel typically exhibit similar behavioral characteristics to ordinary employees before engaging in malicious activities, this invention employs a time-series data representation method that integrates the time dimension. This method emphasizes identifying potential malicious dynamics by capturing the changing trends of user behavior over time. Specifically, the method involves stitching together data points within consecutive time periods, or comparing the behavioral patterns of the same user in adjacent time periods. This strategy effectively reveals potential behavioral changes, thereby identifying potential behavioral shifts that reflect malicious intent. For feature extraction of each employee, daily data units record the user's behavioral characteristics within a single day. Figure 1 The feature extraction process is demonstrated.

[0067] The feature extraction process includes the following steps:

[0068] • Daily Data Aggregation: All activity files (devices, emails, documents, web pages, logins) are aggregated daily. This aggregation process organizes the data into daily segments, generating pickle files categorized by date. This method facilitates the creation of time series using daily behavioral characteristics to capture user behavior patterns, thereby reducing the complexity of handling data fluctuations.

[0069] The user information retrieval process extracts comprehensive user information, including relevant features and malicious user tags, from multiple CSV files in the P directory using linear discriminant analysis. These files contain user attributes such as username, role, department, and team. After numerical processing of the psychological testing information, it is integrated into the employee database. The insiders.csv file provides detailed information on user malicious activity, including start and end times and scenario identifiers. The result is a Pandas DataFrame containing all user features, providing a comprehensive view of each employee profile.

[0070] • Daily behavioral feature extraction converts the daily behavioral data recorded in the data frame into statistical and frequency features. In addition to the provided psychometrics and role-based attributes, various statistical and frequency-based behavioral features of employees need to be extracted. These include, but are not limited to: activity quantity, login frequency, number of file operations, number of emails sent, number of web page requests, activity duration, email / file length, and specific category behaviors.

[0071] Extract features and save as a CSV file.

[0072] Step two: Window the data according to each user's timeline.

[0073] Linear discriminant analysis (LDA) was used to reduce the dimensionality of the extracted features, selecting directions that maximized the distinction between normal users and the three scenario categories. After LDA filtering, a 104-dimensional feature set excluding employee information was obtained. At this point, the feature data contained mixed data from 1,000 users. However, the model needs to learn user behavior patterns to identify malicious users who intentionally hide their activity. Therefore, a sliding window processing method was also applied to the feature data. The data for each user was partitioned and sorted chronologically. Figure 2 The time series processing is shown below: the window size is set to w = [20, 25, 30, 35, 40, 45], and the sliding step is set to s = 1. The reason for choosing the window size w is that the normal working hours of employees in the dataset correspond to a typical weekly work pattern (five days a week). In scenario 2, employees attempt to conceal malicious behavior through a long-term behavior lasting approximately two months. Since the time series cannot be too short, the sequence length of the input model is increased from the number of working days per month to approximately 20 to 25 days. The window size is increased by five working days (one week) each time, eventually reaching two months of working days, or 40 to 45 days.

[0074] Furthermore, sequential windowed inputs better capture relationships between time steps. However, extensive overlap in the data can lead to information redundancy and overfitting. We address this by integrating random dropout layers into each structural layer and using early stopping and validation strategies to prevent overfitting. NPZ files store the window set for each user, with each window containing all features across w time steps and their corresponding classification labels. The windowed data stored in the NPZ files is loaded before training begins. Data is organized by user, and the model is fed sequentially with each user's windowed data (the next user's data is only input after all windowed data for one user has been completed). Figure 2 The document demonstrates the data windowing and model input process.

[0075] Step 3: Model building.

[0076] The overall structure and process of the model are as follows: Figure 3 As shown, the model consists of two parts: a temporal convolutional network and a transformer. The temporal convolutional network consists of four temporal convolutional network blocks, and the transformer consists of two transformer layers.

[0077] The temporal convolutional network part of the model consists of four temporal convolutional network blocks, each of which contains the following structure:

[0078] One-dimensional causal convolution: Causal convolution is a convolutional neural network method specifically designed for processing time series data. Its core characteristic is that the convolution operation only considers the input at the current and previous time steps, deliberately ignoring future data. This design ensures the causality of the model, effectively avoiding the "leakage" of future information, thus preserving the inherent temporal characteristics of time series data. canv =ReLU(Conv1D(x, filters, kernel size dilation rate )), where x conv This is the result of a one-dimensional causal convolution, where ReLU represents the activation function, Conv1D is a one-dimensional causal convolution, x is the model input, filters represent the convolution kernel, and kernel... size Indicates the kernel size, dilation rate This represents the expansion factor.

[0079] Random dropout: Randomly sets the output of some neurons to zero to prevent overfitting. Z = Dropout(x conv In the formula, Z represents the result after random discarding, and Dropout represents the random discarding process.

[0080] Normalization: Stabilizes the distribution of activation values ​​in each layer. y = LayerNorm(Z), where y represents the result of normalizing the random result, and LayerNorm represents performing layer normalization on the current input along the feature dimension to standardize its mean and variance, thereby alleviating the gradient vanishing problem and improving training stability.

[0081] Residual connections: By skipping specific layers and directly superimposing their inputs onto the output, the vanishing / exploding gradient problem in deep neural networks can be effectively alleviated, accelerating the training process and helping the network learn identity mappings more efficiently. To ensure the consistency of input and output in the time dimension in residual connections, we calculate the receptive field of each residual module. R=(k-1)(∑ i d i +1)N, where R represents the final receptive field of the residual module, i.e., the length of the input time step to which the current output position depends. k is the size of the convolution kernel, d i Let represent the inflation factor of the i-th layer, for example, d = 1, 2, 4, 8, ..., and let d represent the interval sampling. ∑ i d i This represents the cumulative expansion rate across multiple layers. (k-1): This represents the contribution of each convolutional kernel to the expansion of the receptive field. +1 indicates that the current time step itself is also included in the receptive field. N represents the number of stacked layers of the TCN module (i.e., the number of residual blocks).

[0082] Dilated convolution can be achieved by stacking four temporal convolutional network blocks. This can be achieved by adjusting the kernel size k or increasing the dilation factor d. i This strategy aims to expand the receptive field. However, as network depth increases, this approach can lead to problems such as increased computational costs, gradient explosion, or vanishing gradients. In the formula, x output This represents the output value at position s after temporal convolution, where x represents the model input, x* d f represents the dilated convolution operation (d is the dilation factor, f is the convolution kernel), s is the time step index of the current output position, and f(i) represents the weight of the i-th convolution kernel. This represents the sampling point in the input sequence corresponding to the weight, and the product of the two is the i-th convolutional kernel response at the current time step s. This means summing the weighted results across all sliding windows of the convolution kernel to obtain the final output at the current time step s. In other words, it involves performing a multiplication once for each i (i.e., each bit of the convolution kernel). Then, all the results are summed to form the output of time step s. Here, k-1 represents the last index of the convolutional kernel within the scope of the current time step, which is used to control the number of summations to ensure that exactly k convolutional kernel weights are used for calculation.

[0083] The converter layer contains the following structure:

[0084] Positional encoding: The output of the temporal convolutional network is fed into the positional encoding module to inject positional information into the sequence (since convolution and transforms are themselves insensitive to position). The model reinforces the temporal order by adding additional positional data (without changing the dimensions). Positional encoding is calculated using sine / cosine functions. For each position t and dimension m, its encoding PE(t,m) is:

[0085] Even indices use a sine function:

[0086] Odd indices use a cosine function: In the formula, d represents the feature dimension. Location information is incorporated into the input data in the following way: x pos =x output +PE

[0087] Multi-head self-attention: The input undergoes a linear transformation to generate a query (Q), key (K), and value (V) matrix. Q = x pos W Q K = x pos W K V = x pos W V In the formula, W Q W K W V is a trainable weight matrix.

[0088] Attention weight calculation: In the formula, A represents the attention output, Attention(Q, K, V) represents the attention weights, and d k This represents the dimension of the K vector. The similarity between Q (query) and each K is calculated, and after passing through softmax, it becomes a weight distribution (i.e., "attention distribution"). Finally, multiplying by V means applying these attention weights to the corresponding position V to obtain the final output A.

[0089] After residual connection and normalization: X attn =LayerNorm(x pos +Z), where X attn Represents the output of each attention head

[0090] Multi-head attention is computed using multiple independent attention heads, and then the outputs of each head are concatenated together: MultiHead(Q, K, V) = Concat(head1, head2, ..., head...). h W O In the formula, h is the number of attention heads. h This represents the output of each attention head (i.e., X). attn ), W O It is the output projection matrix.

[0091] Feedforward network layer: In the converter encoder, the features of each time step are processed through two fully connected network layers: the first layer uses weights W1 and bias b1 for affine transformation followed by activation by a linear rectified function; the second layer transforms again using W2 and b2. Both the multi-head attention and feedforward network layers apply a random dropout layer followed by residual connections and layer normalization.

[0092] Classifier: Predicts and classifies the input sequence at each time step.

[0093] Step 4: Divide the training set and the test set, and input the user's windowed data into the model on a user-by-user basis.

[0094] Because the number of malicious samples is far less than that of normal samples, the experiment employs a phased training method. In the first phase, the model learns normal behavioral patterns using segmented data from normal users. In the second phase, 80% of the segmented data from malicious users across three scenarios is used for training.

[0095] The remaining 20% ​​is used as the test set. Furthermore, samples of roughly equal size to the number of normal users are randomly selected to form the normal training and test sets, thus mitigating the data imbalance problem. There are a total of 70 malicious users and 930 normal users. In the first phase of training, 860 normal users are randomly selected to learn their features. In the second phase, 80% (56 users) of the remaining 70 normal users and 70 malicious users are randomly selected as the training set, while the remaining 20% ​​(14 users) are used for testing. This method, along with the focus loss function, mitigates the impact of the data imbalance problem. The specific process is as follows... Figure 4 As shown in Table 1, after multiple experiments and optimizations, we selected the hyperparameter configurations shown in Table 1 for model training.

[0096] Step 5: Output the prediction results and test the model performance.

[0097] The model makes predictions for all time steps within each window of the input model, essentially performing malware detection for each day. The output is the predicted classification for all time steps within each window. Finally, the predicted classifications for each day are compared with the true classifications to evaluate metrics such as model accuracy and recall.

[0098] In this invention, recall and AUC for malicious users are used for evaluation. The experiment employs a phased training method, requiring the initial loading of compressed windowed data NPZ files from the local machine. Users are read by username, and the dataset is re-split by comparing it with the malicious user list recorded in insider.csv. In the first phase, the model randomly selects 860 normal users to learn their normal behavior patterns. In the second phase, 80% (56 users) of the remaining 70 normal users and 70 malicious users are randomly selected as the training set, while the remaining 20% ​​(14 users) are used for testing. The model inputs are processed user by user; after all windows for one user are entered, the next user's window is entered.

[0099] Since most research focuses on post-hoc malicious detection of users, with very few studies based on instance-based malicious detection, we compare our method with both approaches. Table 2 compares our method with instance-based methods such as gated recurrent units + graph convolutional networks, gated recurrent units + graph attention networks, long short-term memory networks + graph convolutional networks, long short-term memory networks + graph attention networks, MAITD, transformers + graph convolutional networks, and transformers + graph attention networks. Table 3 compares our method with post-hoc user detection methods.

[0100] In terms of recall, the best instance-based detection method is a Long Short-Term Memory (LSTM) network combined with a Graph Convolutional Network (GCN), achieving 93.69%. Our method improves upon this by 1.31%, and is nearly 20% better than the worst method, MAITD. Compared to post-event user detection, only the BERT+FL method achieved the same recall of 95% as our method. Other traditional LSM networks, FCVM, and gated recurrent unit variants had significantly lower recalls than our method. Regarding AUC, our method outperforms almost all instance-based methods by more than 10%, and its false positive rate is almost 0.1 lower than all methods. This indicates that our method is highly accurate in detecting malicious users. In post-event user detection, the best AUC method is a stacked convolutional neural network-attention bidirectional gated recurrent unit model, achieving 95%, which our method improves upon by 3.84%.

[0101] Regarding training time, our invention was compared with autoencoders, isolated forests, and local outlier factors. With a window size of 30, our average training time was 7.76 seconds, while autoencoders took 40 seconds, local outlier factors took 85 seconds, and isolated forests took 10 seconds. In terms of prediction time, our method required 2.07 ms, only slightly better than the local outlier factor method's 5.2 ms.

[0102] Compared to instance-based and post-event user detection, our method outperforms in recall and AUC, indicating that the temporal convolutional network + transformer model has stronger stability and can identify complex and variable user behavior patterns. Converting user logs into a sliding window sequence captures behavioral patterns that change over time and detects anomalies at each time step. This enables timely identification of malicious employees before significant harm occurs and achieves a higher detection rate. In terms of feature learning, the temporal convolutional network captures local patterns and short-term dependencies in time-series data through multiple layers of dilated convolutions. For example, some anomalous behaviors may only manifest in a few consecutive time steps, and the temporal convolutional network excels at identifying such short-term features. Simultaneously, global dependencies—the relationships between any two time steps in the sequence—are equally important. Combining both allows associating anomalous behavior with earlier events at a specific time step in the sequence. The transformer allows the model to consider information from all other time steps simultaneously while processing each time step, thus understanding the overall context. Through a self-attention mechanism, the transformer captures the relationships between any two time steps in the input sequence, enhancing the richness and expressiveness of feature representations. By effectively fusing local features generated by temporal convolutional networks with global dependency information provided by transformers, a more comprehensive feature representation can be formed. This enhanced representation strengthens the support capabilities of subsequent classifiers and improves overall detection performance.

[0103] Table 1. Model details and hyperparameters (learning rate 5×10⁻⁶) -4 Batch size is 32; loss function is focus loss (α = 0.8, γ = 2.0, reduction method = 'mean')

[0104]

[0105] Table 2. Comparison with Instance-Based Detection Evaluation Results

[0106]

[0107] Table 3. Comparison with evaluation results based on post-event user detection

[0108]

[0109]

[0110] Although some embodiments of the present invention have been shown and described, those skilled in the art will recognize that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and all such solutions fall within the scope of protection of the present invention.

Claims

1. An anomaly detection method based on temporal convolutional networks and transformers, characterized in that... The steps include the following: Step 1: Preprocess and extract features from the CERT r4.2 dataset; the preprocessing includes: data integration, missing data filling, and data concatenation; the feature extraction includes: daily data aggregation, user information retrieval, and daily behavioral feature extraction. Step 2: Perform data windowing according to each user's timeline; the data windowing process includes: performing dimensionality reduction on the original feature data using linear discriminant analysis; partitioning and sorting the data for each user in chronological order; Step 3: Model Construction; The model includes a temporal convolutional network, a transformer, and a classifier; The temporal convolutional network consists of 4 temporal convolutional network blocks, the transformer consists of 2 transformer layers, and the classifier predicts and classifies the input sequence at each time step; Step 4: Divide the data into training and testing sets; input windowed data of users into the model, using users as the unit, to train the model; Step 5: Input the test set data into the model trained in Step 4 using windowed data, with each user as a unit, to detect abnormal behavior and analyze the performance of the detection model.

2. The anomaly detection method based on temporal convolutional networks and transformers as described in claim 1, characterized in that: The data integration process in step one is as follows: (1) Iterate through each unique user identifier; (2) Filter out records containing a specific user identifier from the five log types; (3) Sort these records according to the time information in the date field.

3. The anomaly detection method based on temporal convolutional networks and transformers as described in claim 1, characterized in that: In step one, missing data is filled in. Missing values ​​are obtained by calculating the estimated mean of the relevant features.

4. The anomaly detection method based on temporal convolutional networks and transformers as described in claim 1, characterized in that: The feature extraction process in step one includes the following steps: (1) Daily data aggregation: All activity files, including devices, emails, documents, web pages, and logins, are summarized daily. This aggregation process organizes the data into daily segments and generates pickle files categorized by date. (2) User information retrieval: obtain comprehensive user information from multiple CSV files in the LDAP directory, including relevant characteristics and malicious user tags; build a Pandas data frame containing all user characteristics, providing a comprehensive view of each employee profile; (3) Daily behavioral feature extraction: Convert the daily behavioral data recorded in the data frame into statistical and frequency features; In addition to the provided psychological measurements and role-based attributes, extract various statistical and frequency features of employee behavioral characteristics, including: number of activities, login frequency, number of file operations, number of emails sent, number of web page requests, activity duration, email / file length, and specific category behaviors; (4) Save the extracted features as a CSV file.

5. The anomaly detection method based on temporal convolutional networks and transformers as described in claim 1, characterized in that: The temporal convolutional network part of the model in step three consists of four temporal convolutional network blocks, each of which contains the following structure: (1) One-dimensional causal convolution: Causal convolution operation only considers the input of the current and previous time steps, deliberately ignoring future data, thus ensuring the causality of the model and preserving the inherent temporal characteristics of time series data; (2) Random discarding: Randomly set the output of some neurons to zero to prevent overfitting; (3) Normalization: Stabilizes the distribution of activation values ​​in each layer; (4) Residual connection: By skipping a specific layer and directly superimposing its input into the output, it effectively alleviates the gradient vanishing or exploding problem in deep neural networks, accelerates the training process, and helps the network learn identity mappings more efficiently. Dilated convolution is achieved by stacking four temporal convolutional network blocks, and the receptive field is expanded by adjusting the convolutional kernel size or increasing the dilation factor.

6. The anomaly detection method based on temporal convolutional networks and transformers as described in claim 1, characterized in that: The transformer layer of the model in step three includes the following structure: (1) Position Encoding: The output of the temporal convolutional network is fed into the position encoding module to inject positional information into the sequence. The model strengthens the temporal order by adding additional positional data. The position encoding is calculated using a sine or cosine function. For each position t and dimension m, its encoding PE(t,m) is: Even indices use a sine function: Odd indices use a cosine function: In the formula, d represents the feature dimension, and location information is incorporated into the input data in the following way: x pos =x output +PE; (2) Multi-head self-attention: The input is transformed linearly to generate a query (Q), key (K), and value (V) matrix; Q = x pos W Q K = x pos W K V = x pos W V In the formula, W Q W K W V This is a trainable weight matrix; (3) Attention weight calculation: A represents the attention output, Attention(Q,K,V) represents the attention weights, and d k This represents the dimension of the K vector. The similarity between Q and each K is calculated, and after passing through softmax, it becomes a weight distribution, i.e., an attention distribution. Finally, multiplying by V means applying these attention weights to the corresponding position V to obtain the final output A. (4) After residual connection and normalization: X attn =LayerNorm(x pos +A); where X attn This represents the output of each attention head, and LayerNorm represents performing layer normalization on the current input along the feature dimension to standardize its mean and variance. (5) Calculate multi-head attention using multiple independent attention heads. Repeat step (4) above and record the attention output of each head as head. i (i = 1, 2, ..., h); Concatenate the attention output of all heads into MultiHead: MultiHead = Concat(head1, head2, ..., head h W O In the formula, h is the number of attention heads, Concat means concatenating the outputs of multiple heads, and W O It is the output projection matrix; (6) Feedforward network layer: The feedforward layer receives the MultiHead output of the multi-head attention and performs a non-linear mapping at each time step. That is, the features are first stretched to a wider representation space, then compressed back to the original dimension after linear rectified activation; subsequently, they are randomly discarded, residual connected and layer normalized.

7. The anomaly detection method based on temporal convolutional networks and transformers as described in claim 1, characterized in that: The model training in step four adopts a tiered training method. In the first stage, the model learns the normal behavior patterns of normal users using segmented data from normal users. In the second stage, 80% of the segmented data from malicious users in three scenarios is used for training, and the remaining 20% ​​is used for testing.

Images