Internal threat detection method and system based on user behavior characterization technology

Abstract

The present invention belongs to the field of network security technology, and specifically relates to an internal threat detection method and system based on user behavior characterization technology. The present invention aggregates normal user behavior data from different logs according to the user's unique ID value, extracts user behavior features from them, and organizes them into windowed normal user behavior feature time series data; constructs a user behavior characterization model composed of a clustering enhancement module and an LSTM-VAE model, uses normal user behavior feature time series data to train the user behavior characterization model and dynamically adjusts the cluster center during the training process; uses the trained user behavior characterization model to calculate the anomaly score of the user behavior data to be detected, and combines it with a defined threshold to determine whether the user behavior poses a threat. The present invention uses an unsupervised learning method to train the user behavior characterization model, which can not only effectively address data imbalance problems, but also effectively identify abnormal user behavior.

Description

Technical Field

[0001] The present invention belongs to the field of network security technology, and specifically relates to an internal threat detection method and system based on user behavior characterization technology, which is suitable for detecting abnormal user behavior in enterprise internal threat scenarios. Background Art

[0002] With the rapid development of information technology, especially the widespread application of the internet, cloud computing, big data, and artificial intelligence, enterprise information systems are becoming increasingly complex and expansive. Traditional security measures primarily focus on preventing external attacks. However, with changes in the network environment and the increasing complexity of enterprise structures, internal threats have gradually become one of the most serious security challenges facing enterprises.

[0003] Insider threats refer to attacks or misconduct initiated by personnel within an organization. Unlike external threats, insider threats are characterized by attackers possessing legitimate access rights, typically operating under an insider identity, and their activities typically occur during normal business hours. Attackers possess knowledge of an organization's infrastructure and sensitive information, enabling them to cause serious damage through the disclosure, duplication, and tampering of corporate information.

[0004] Currently, there are two main methods for detecting insider threats. One is rule-based pattern matching: This method uses a predefined set of rules or signatures to match threat behaviors and attack patterns occurring in a network or system to identify potential security threats. Because insider attackers often possess knowledge of system security mechanisms and are relatively stealthy, this method may experience missed alerts and false positives. Because the rules are based on known threat behaviors and attack patterns, this method may not detect new attack behaviors. Furthermore, with the rapid evolution of user behavior, this method requires frequent updates and maintenance of the rule base, which is cumbersome and has limitations in practical applications.

[0005] Another detection method is based on machine learning technology: it uses machine learning to learn the characteristics of normal and abnormal behavior from a large amount of historical system data for detection. This method can learn effective behavioral patterns from data, avoiding the tedious process of manually defining rules. However, in real-world insider threat detection scenarios, abnormal user behavior data is far less than normal behavior data, resulting in a serious data imbalance. Due to the lack of high-quality, annotated user behavior data for model training, traditional machine learning methods such as hidden Markov models and support vector machines often fail to effectively model user behavior, resulting in suboptimal detection results. Summary of the Invention

[0006] In order to solve the problems of severe data imbalance and low accuracy in the field of internal threat detection, the present invention provides an internal threat detection method and system based on user behavior characterization technology.

[0007] The present invention is achieved through the following technical solutions.

[0008] An internal threat detection method based on user behavior characterization technology includes the following steps:

[0009] S1. Data preprocessing: Aggregate normal user behavior data from different logs based on the user's unique ID value, extract user behavior features from it, and organize it into windowed normal user behavior feature time series data;

[0010] S2. Build a user behavior representation model consisting of a clustering enhancement module and an LSTM-VAE model. Use normal user behavior feature time series data to train the user behavior representation model and dynamically adjust the cluster center during the training process.

[0011] S3. Use the trained user behavior representation model to calculate the anomaly score of the user behavior data to be detected, and combine it with the defined threshold to determine whether the user behavior is a threat.

[0012] Further preferably, in step S2, the user behavior features are first passed to the clustering enhancement module, the clustering enhancement module uses LSTM as an encoder to convert the user behavior features into latent variables and cluster them to obtain the best initial cluster center, and then the best initial cluster center is passed to the LSTM-VAE model. The LSTM-VAE model learns and represents the user behavior features and dynamically adjusts the cluster center during the LSTM-VAE model training process.

[0013] Further preferably, the clustering process performed by the clustering enhancement module is as follows:

[0014] The input data of the clustering enhancement module is user behavior features. The clustering enhancement module includes LSTM, linear layer and K-Means algorithm clustering module;

[0015] LSTM receives input user behavior features , and convert it into a hidden state , the hidden state of the last moment of LSTM The mean μ and logarithmic variance of the latent variable are obtained through the fully connected layer , and then use the reparameterization method to get the latent variable z;

[0016] Use K-Means algorithm to cluster latent variables and select the optimal number of cluster centers K by the silhouette coefficient method best ;

[0017] Finally, the optimal number of cluster centers K is used bestRetrain the K-Means algorithm to obtain the best initial cluster center.

[0018] Further preferably, the LSTM-VAE model uses a single-layer LSTM instead of a fully connected layer as the encoder and decoder of the variational autoencoder. The input data of the LSTM-VAE model is user behavior features; the encoder maps the input data into latent variables in the latent space, and the decoder generates a reconstructed sequence by decoding the latent variables back to the input space.

[0019] Furthermore, the LSTM-VAE model is trained using a joint optimization strategy to simultaneously optimize the reconstruction loss, KL divergence loss, and clustering loss;

[0020] ;

[0021] in, is the total loss of the LSTM-VAE model, is the reconstruction loss, is the KL divergence loss, is the clustering loss, where and is a hyperparameter.

[0022] Further preferably, the method of dynamically adjusting the cluster center is as follows: after each P batch, the cluster center is recalculated based on the current latent variables, where is the momentum coefficient, which is used to control the degree of information retention of the cluster center;

[0023] ;

[0024] in, is the cluster center before adjustment, is the adjusted cluster center, where is the kth cluster, where is the number of latent variables in the k-th cluster; is the i-th latent variable, is the kth cluster center.

[0025] Further preferably, the data preprocessing includes the following sub-steps:

[0026] Step 1.1: Aggregate normal user data from different logs based on the user's unique ID value. Extract the user's behavior data within a day at a time granularity, merge it, and sort it by time.

[0027] Step 1.2: Extract user behavior features from the aggregated data and organize them into normal user behavior feature time series data;

[0028] Step 1.3: Window the time series data of normal user behavior characteristics.

[0029] The present invention also provides an internal threat detection system based on user behavior characterization technology, comprising:

[0030] Data collection module, used to collect user behavior data;

[0031] Data preprocessing module: used to aggregate normal user behavior data from different logs based on the user's unique ID value, extract user behavior features from it, and organize it into windowed normal user behavior feature time series data;

[0032] The threat detection module has a built-in user behavior characterization model consisting of a clustering enhancement module and an LSTM-VAE model. It uses normal user behavior feature time series data to train the user behavior characterization model and dynamically adjusts the cluster center during the training process. It uses the trained user behavior characterization model to calculate the anomaly score of the user behavior data to be detected, and combines it with the defined threshold to determine whether the user behavior poses a threat.

[0033] The present invention also provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory. When the computer program is executed by the processor, the internal threat detection method based on user behavior characterization technology as described above is implemented.

[0034] Beneficial effects of the present invention: The present invention first passes user behavior characteristics to the clustering enhancement module. The clustering enhancement module uses LSTM as an encoder to convert user behavior characteristics into latent variables and clusters them to obtain the optimal initial cluster center. The optimal initial cluster center is then passed to the LSTM-VAE model. The LSTM-VAE model learns and characterizes user behavior characteristics, and dynamically adjusts the cluster center during the LSTM-VAE model training process, thereby improving the accuracy of internal threat detection. The present invention uses an unsupervised learning method to train the user behavior characterization model, which can not only effectively address data imbalance problems, but also effectively identify abnormal user behavior and detect internal threats. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flowchart of the steps of the internal threat detection method based on user behavior characterization technology described in the present invention;

[0036] Figure 2 This is a structural diagram of the clustering enhancement module described in Example 2;

[0037] Figure 3 This is the LSTM-VAE model structure diagram described in Example 2. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of this application.

[0039] Example 1

[0040] This embodiment proposes an internal threat detection method based on user behavior characterization technology, aiming to solve the problems of severe data imbalance and low accuracy in the field of internal threat detection. Figure 1 As shown, the method includes the following steps:

[0041] S1. Data preprocessing: Aggregate normal user behavior data from different logs based on the user's unique ID value, extract user behavior features from it, and organize it into windowed normal user behavior feature time series data;

[0042] S2. Build a user behavior representation model consisting of a clustering enhancement module and an LSTM-VAE model. Use normal user behavior feature time series data to train the user behavior representation model and dynamically adjust the cluster center during the training process.

[0043] S3. Use the trained user behavior representation model to calculate the anomaly score of the user behavior data to be detected, and combine it with the defined threshold to determine whether the user behavior is a threat.

[0044] Example 2

[0045] The following describes in detail the internal threat detection method using user behavior characterization technology based on Example 1.

[0046] S1. Data preprocessing:

[0047] The CERT dataset was selected to validate the proposed insider threat detection method. This dataset is a multi-source dataset that includes system login and logout logs, device usage logs, email logs, network logs, human resources, and other information. Data preprocessing includes the following steps:

[0048] Step 1.1: Aggregate normal user data from different logs based on the user's unique ID value, extract the user's behavior data within a day at a time granularity, merge them, and sort them by time.

[0049] Step 1.2: Extract user behavior features from the aggregated data and organize them into normal user behavior feature time series data. This embodiment counts 23 user behavior features. These include: the number of all behaviors, the number of all behaviors during working hours, the number of all behaviors during non-working hours, the number of logins, the number of logins during non-working hours, the number of USB device uses, the number of USB device uses during non-working hours, the number of file operations, the number of file operations during non-working hours, the number of image accesses, the number of image accesses during non-working hours, the number of document accesses, the number of document accesses during non-working hours, the number of text file accesses, the number of text file accesses during non-working hours, the number of executable file accesses, the number of executable file accesses during non-working hours, the number of emails sent or received, the number of emails sent or received during non-working hours, the number of web browsing via the HTTP protocol, the number of web browsing via the HTTP protocol during working hours, and the number of web browsing via the HTTP protocol during non-working hours.

[0050] Step 1.3: Window the time series data of normal user behavior characteristics. This example uses a fixed-length sliding window method, with the window length set to 5. Each window can be used as a training sample for the model, containing the user's user behavior characteristics within 5 days.

[0051] S2. Build a user behavior representation model consisting of a clustering enhancement module and an LSTM-VAE model. Use normal user behavior feature time series data to train the user behavior representation model and dynamically adjust the cluster center during the training process.

[0052] First, the user behavior features are passed to the clustering enhancement module. The clustering enhancement module uses LSTM as an encoder to convert the user behavior features into latent variables and cluster them to obtain initial cluster centers. The initial cluster centers are then passed to the LSTM-VAE model. The LSTM-VAE model learns and represents the user behavior features and dynamically adjusts the cluster centers during the LSTM-VAE model training process. Step S2 includes the following steps:

[0053] Step 2.1: The clustering enhancement module uses LSTM as an encoder to convert user behavior features into latent variables and cluster them to obtain the best initial cluster center. The input data of this step is user behavior features , is the user behavior feature at time t, and T is the total time. The clustering enhancement module is as follows Figure 2 As shown, it includes LSTM, linear layer and K-Means algorithm clustering modules.

[0054] The task of LSTM is to encode the user behavior features and convert them into latent variables in the latent space of the variational autoencoder (VAE). LSTM receives the input user behavior features , and convert it into a hidden state , the calculation process is shown as follows:

[0055] (1);

[0056] in, is the hidden state at time t-1, is the hidden state at time t.

[0057] The hidden state of the LSTM at the last moment The mean μ and logarithmic variance of the latent variable are obtained through the fully connected layer , the calculation formula is shown in (2) and (3):

[0058] (2);

[0059] (3);

[0060] in, is the trainable bias vector of the mean generation layer, is the trainable bias vector of the variance generating layer, is the trainable weight matrix of the mean generation layer, Trainable weight matrix for the variance generation layer;

[0061] The latent variable z is obtained using the reparameterization method, and the calculation formula is shown in (4):

[0062] (4);

[0063] in, is the noise sampled from a standard normal distribution;

[0064] Use K-Means algorithm to calculate the latent variables z={z1,z2,…, ,…,z n} for clustering, is the i-th latent variable, i={1,2,…,n}, n is the number of latent variables, and the optimal number of cluster centers K is selected by the silhouette coefficient method best .

[0065] Select K data points as the initial cluster centers and assign each latent variable Assign it to the initial cluster center closest to it, that is, calculate the Euclidean distance, and the calculation formula is shown in (5):

[0066] (5);

[0067] in, is the kth cluster center, k={1,2,…,K}, K is the number of initial cluster centers;

[0068] After completing the latent variable assignment, the K-Means algorithm updates the cluster center of each cluster. The new cluster center is the mean of all latent variables in the cluster. The specific calculation process is shown in formula (6):

[0069] (6);

[0070] in, is the kth cluster, where is the number of latent variables in the k-th cluster;

[0071] The clustering process of the K-Means algorithm is an iterative process, which repeats the above steps until the cluster centers converge.

[0072] Next, the silhouette coefficient method is used to determine the optimal number of cluster centers K best , the calculation process is as follows:

[0073] Different latent variables represent different data points, and the latent variables are calculated The average distance to other latent variables in the same cluster :

[0074] (7);

[0075] in, are other latent variables within the cluster.

[0076] calculate The average distance of all latent variables to the nearest other cluster :

[0077] (8);

[0078] in, are the clusters other than the kth cluster.

[0079] Calculating latent variables The silhouette coefficient is:

[0080] (9);

[0081] For different initial cluster center numbers K, calculate the average value of the silhouette coefficients of all latent variables, and select the K that maximizes the average silhouette coefficient as the optimal number of cluster centers K. best :

[0082] (10);

[0083] in, is a hidden variable The silhouette coefficient.

[0084] Finally, the optimal number of cluster centers K is used best Retrain the K-Means algorithm to obtain the best initial cluster center:

[0085] (11);

[0086] in, is the optimal initial cluster center set, is the first best initial cluster center, For K best The best initial cluster centers.

[0087] Step 2.2: Use the LSTM-VAE model with the best initial cluster center to learn and characterize user behavior features, and dynamically adjust the cluster center during the LSTM-VAE model training process. Figure 3 As shown in the figure, the LSTM-VAE model uses a single-layer LSTM instead of a fully connected layer as the encoder and decoder of the variational autoencoder (VAE). In the encoder, a linear layer is used after the LSTM processing. LSTM can effectively capture the long-term and short-term dependencies in time series data and can effectively deal with the gradient vanishing or exploding problems of simple recurrent networks. The input data of this step is the user behavior characteristics. The encoder maps the input data to the latent variable z in the latent space in the same way as described in step 2.1. The decoder generates the reconstructed sequence by decoding the latent variable z back to the input space. , is the reconstructed feature at time t, which measures the modeling ability of the LSTM-VAE model for user behavior patterns, as shown in formulas (12) and (13):

[0088] (12);

[0089] (13);

[0090] in, is the hidden state generated by the decoder at time t, is the hidden state of the decoder at time t-1, is the trainable weight matrix of the decoder output layer, is the trainable bias vector of the decoder output layer, is the hidden variable at time t.

[0091] The LSTM-VAE model is trained using a joint optimization strategy to optimize the reconstruction loss, KL divergence loss, and clustering loss. The reconstruction loss is used to measure the difference between the sequence reconstructed by the decoder based on the latent variables and the original input sequence. The calculation is shown in formula (14):

[0092] (14);

[0093] KL divergence loss is used to control the distribution of the latent space to make it close to the standard normal distribution. The calculation is shown in formula (15):

[0094] (15);

[0095] in, is a hidden variable The square of the mean of is a hidden variable The variance of .

[0096] Clustering loss is defined as the average of the minimum Euclidean distances from the data point to the nearest cluster center. The clustering loss model can separate normal and abnormal behaviors in the latent space more clearly, thereby improving the accuracy of anomaly detection. The calculation is shown in formula (16):

[0097] (16);

[0098] By jointly optimizing the reconstruction loss, KL divergence loss, and clustering loss, we can simultaneously optimize the sequence reconstruction quality, the distribution of the latent space, and the separability of the clustering structure. The total loss is calculated as shown in formula (17), where and are hyperparameters that control the regularization strength of the KL divergence and the impact of clustering constraints on the LSTM-VAE model.

[0099] (17);

[0100] During the training of the LSTM-VAE model, the cluster center is dynamically adjusted. After each P batch, based on the current latent variable Recalculate the cluster center as shown in formula (18), where is the momentum coefficient, which is used to control the degree of information retention of the cluster center.

[0101] (18);

[0102] Based on the trained LSTM-VAE model, the anomaly score of the user behavior data to be detected is calculated, and combined with the defined threshold, it is determined whether the user behavior is threatening. The anomaly score is obtained by calculating the reconstruction error and cluster distance of the user behavior data to be detected, as shown in formula (19), where is the weight coefficient used to control the importance of cluster distance in anomaly score.

[0103] (19);

[0104] in, When the anomaly score of user behavior data exceeds the threat threshold, the user behavior is judged to be threatening.

[0105] Example 3

[0106] An internal threat detection system based on user behavior characterization technology, comprising:

[0107] Data collection module, used to collect user behavior data;

[0108] Data preprocessing module: used to aggregate normal user behavior data from different logs based on the user's unique ID value, extract user behavior features from it, and organize it into windowed normal user behavior feature time series data;

[0109] The threat detection module has a built-in user behavior characterization model consisting of a clustering enhancement module and an LSTM-VAE model. It uses normal user behavior feature time series data to train the user behavior characterization model and dynamically adjusts the cluster center during the training process. It uses the trained user behavior characterization model to calculate the anomaly score of the user behavior data to be detected, and combines it with the defined threshold to determine whether the user behavior poses a threat.

[0110] Example 4

[0111] An electronic device includes a processor, a memory, and a computer program stored in the memory. When the computer program is executed by the processor, it implements the internal threat detection method based on user behavior characterization technology as described above.

[0112] The above disclosures are merely some preferred embodiments of the present invention, and certainly cannot be used to limit the scope of the present invention. A person skilled in the art will understand that all or part of the processes of the above embodiments and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for detecting internal threats based on user behavior characterization technology, characterized in that: The following steps are involved: S1. Data preprocessing: Aggregate normal user behavior data from different logs based on the user's unique ID value, extract user behavior features from it, and organize it into windowed normal user behavior feature time series data; S2. Build a user behavior representation model consisting of a clustering enhancement module and an LSTM-VAE model, and pass the user behavior features to the clustering enhancement module. The clustering enhancement module includes LSTM, linear layer, and K-Means algorithm clustering module. LSTM receives the input user behavior features. , and convert it into a hidden state , the hidden state of the last moment of LSTM The mean μ and logarithmic variance of the latent variable are obtained through the fully connected layer , and then use the reparameterization method to obtain the latent variable z; use the K-Means algorithm to cluster the latent variables, and select the optimal number of cluster centers K by the silhouette coefficient method best ; Finally, use the optimal number of cluster centers K best Retrain the K-Means algorithm to obtain the best initial cluster center; The optimal initial cluster center is then passed to the LSTM-VAE model, which learns and characterizes user behavior characteristics and dynamically adjusts the cluster center during the LSTM-VAE model training process. The method of dynamically adjusting the cluster center is: after each P batch, the cluster center is recalculated based on the current latent variables, where is the momentum coefficient, which is used to control the degree of information retention of the cluster center; ； in, is the cluster center before adjustment, is the adjusted cluster center, where is the kth cluster, where is the number of latent variables in the k-th cluster; is the i-th latent variable, is the kth cluster center; S3. Use the trained user behavior representation model to calculate the anomaly score of the user behavior data to be detected, and combine it with the defined threshold to determine whether the user behavior is a threat.

2. The internal threat detection method based on user behavior characterization technology according to claim 1 is characterized in that: The LSTM-VAE model uses a single-layer LSTM instead of a fully connected layer as the encoder and decoder of the variational autoencoder. The input data of the LSTM-VAE model is user behavior features; the encoder maps the input data into latent variables in the latent space, and the decoder generates a reconstructed sequence by decoding the latent variables back into the input space.

3. The internal threat detection method based on user behavior characterization technology according to claim 1 is characterized in that: The LSTM-VAE model is trained using a joint optimization strategy to simultaneously optimize reconstruction loss, KL divergence loss, and clustering loss; ； in, is the total loss of the LSTM-VAE model, is the reconstruction loss, is the KL divergence loss, is the clustering loss, where and is a hyperparameter.

4. The internal threat detection method based on user behavior characterization technology according to claim 1 is characterized in that: The data preprocessing includes the following sub-steps: Step 1.1: Aggregate normal user data from different logs based on the user's unique ID value. Extract the user's behavior data within a day at a time granularity, merge it, and sort it by time. Step 1.2: Extract user behavior features from the aggregated data and organize them into normal user behavior feature time series data; Step 1.3: Window the time series data of normal user behavior characteristics.

5. A system for implementing the internal threat detection method based on user behavior characterization technology according to any one of claims 1 to 4, characterized in that: include: Data collection module, used to collect user behavior data; Data preprocessing module: used to aggregate normal user behavior data from different logs based on the user's unique ID value, extract user behavior features from it, and organize it into windowed normal user behavior feature time series data; The threat detection module has a built-in user behavior characterization model consisting of a clustering enhancement module and an LSTM-VAE model. It uses normal user behavior feature time series data to train the user behavior characterization model and dynamically adjusts the cluster center during the training process. It uses the trained user behavior characterization model to calculate the anomaly score of the user behavior data to be detected, and combines it with the defined threshold to determine whether the user behavior poses a threat.

6. An electronic device comprising a processor, a memory, and a computer program stored in the memory, wherein: When the computer program is executed by the processor, the internal threat detection method based on user behavior characterization technology described in any one of claims 1 to 4 is implemented.

Images