An insider threat detection method based on user classification and behavioral feature analysis

Abstract

The present invention discloses an internal threat detection method based on user classification and behavior feature analysis, which separates log files to be processed according to different users, and organizes each user's log file into a format that is convenient for subsequent operation processing according to time and source; identifies risks in the user's log file content, and extracts the user's behavior statistical features and user behavior serialization features; clusters all users based on the extracted user behavior statistical features and user behavior serialization features, and extracts class features for each user category; utilizes the features generated by the user's current log processing and the features of the user's category to identify potential abnormal risk behaviors and detect internal threat risks caused by the user's abnormal risk behaviors. The above method greatly reduces the deployment and maintenance costs of the detection framework, can use lower-cost detection technology for internal threat detection, and has a higher detection rate.

Description

Technical Field

[0001] The present invention relates to the technical field of user anomaly detection, and in particular to an internal threat detection method based on user classification and behavior feature analysis. Background Art

[0002] Current research on insider threat detection primarily relies on machine learning and deep learning methods to determine whether insider threats exist. Previous approaches typically employed various feature extraction techniques to process user logs and use these features to train machine learning or deep learning models to detect insider threats. Generally speaking, these methods aim to extract patterns in daily user behavior and apply them to detect anomalous user behavior.

[0003] While previous research has demonstrated some capabilities for insider threat detection, it still has some shortcomings. First, existing research focuses solely on behavioral information in user logs, ignoring the specific information contained in the logs, resulting in incomplete information. Second, existing research maintains a machine learning or deep learning model for each user, which results in prohibitively high deployment and maintenance costs in practical applications. Finally, existing research only makes a single decision for user anomaly detection. To ensure accuracy, some normal behaviors are classified as anomalies, resulting in a high false positive rate. Summary of the Invention

[0004] The purpose of this invention is to provide an internal threat detection method based on user classification and behavioral feature analysis, which greatly reduces the deployment and maintenance costs of the detection framework, can use lower-cost detection technology for internal threat detection, and has a higher detection rate.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for detecting insider threats based on user classification and behavioral feature analysis, the method comprising:

[0007] Step 1: Separate the log files to be processed by different users, and organize each user's log files by time and source into a format that is convenient for subsequent operations and processing;

[0008] Step 2: Identify risks in the user's log file content, extract the user's behavior statistical features and user behavior serialization features, and summarize the user's log file based on the user's behavior statistical features and user behavior serialization features;

[0009] Step 3: Cluster all users based on the extracted user behavior statistical features and user behavior serialization features, and extract class features for each user category;

[0010] Step 4: Utilize the features generated by the user's current log processing and the features of the user's category to identify potential abnormal risk behaviors and detect internal threat risks caused by abnormal risk behaviors of users.

[0011] It can be seen from the technical solution provided by the present invention that the above method greatly reduces the deployment and maintenance costs of the detection framework, can use lower-cost detection technology for internal threat detection, and has a higher detection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of an internal threat detection method based on user classification and behavior feature analysis provided by an embodiment of the present invention;

[0014] Figure 2 A schematic diagram of the process of user clustering according to an embodiment of the present invention;

[0015] Figure 3 This is a flowchart of an implementation example of the present invention in a large company's internal system scenario. DETAILED DESCRIPTION

[0016] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments, and do not constitute a limitation of the present invention. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0017] like Figure 1 FIG2 is a flow chart of an internal threat detection method based on user classification and behavior feature analysis according to an embodiment of the present invention. The method includes:

[0018] Step 1: Separate the log files to be processed by different users, and organize each user's log files by time and source into a format that is convenient for subsequent operations and processing;

[0019] In this step, the log file to be processed is first divided into log records of several users according to the user ID;

[0020] Split each user's log records by source (such as logon, email, http, device, file) and arrange them in chronological order;

[0021] The obtained log records are corrected for errors, redundant information is deleted, and information is filled in to facilitate subsequent operations.

[0022] Step 2: Identify risks in the user's log file content, extract the user's behavior statistical features and user behavior serialization features, and summarize the user's log file based on the user's behavior statistical features and user behavior serialization features;

[0023] In this step, risk identification of the user's log file content is performed by using a convolutional neural network model for text classification to classify the input log file content into three categories: normal, low-risk, and high-risk. Specifically:

[0024] First, the input log file content is processed, a character-to-index mapping is defined, and the log file data is cleaned and formatted to prepare its format for the embedding layer of the convolutional neural network model.

[0025] The embedding layer of the convolutional neural network model converts the previously processed log file input into a dense vector representation, mapping each input character into a high-dimensional space;

[0026] The Conv1D layer of the convolutional neural network model extracts local features of the log file through 64 filters of size 5;

[0027] The GlobalMaxPooling1D layer of the convolutional neural network model extracts the strongest signal from each convolutional feature, reducing the data dimension while retaining important information;

[0028] The Dropout layer of the convolutional neural network model randomly discards features with a probability of 15% to prevent overfitting;

[0029] The Dense layer of the convolutional neural network model is a fully connected layer that converts the output of the previous layer into the final classification prediction; the Dense layer uses softmax as the activation function and outputs the probability of each category;

[0030] Among them, the convolutional neural network model uses the Adam optimizer and categorical cross entropy as the loss function;

[0031] In the specific implementation, the user behavior statistical features are based on each user's log file as the information source, such as logon, email, http, device, and file. As shown in Table 1 below, the user behavior statistics are collected:

[0032] Table 1 User behavior statistics items

[0033]

[0034] User behavior statistics are counts of each user's behavior over a daily time period, including the number of times a user logs in or visits a website in a day. These counts are based on the different characteristics of user activity and represent user behavior habits.

[0035] The user behavior serialization feature is a vector that serializes a user's behavior within a day. The specific process of extracting the user behavior serialization feature is as follows:

[0036] First, the user's log files from different sources in a day are integrated, then all the log entries of the user in a day are arranged in chronological order, and finally the activity information is extracted for serialization.

[0037] Among them, the activity information in the form of a string is difficult to be directly used as input for subsequent operations, so the activity information needs to be encoded as shown in the following Table 2:

[0038] Table 2 Activity information coding

[0039]

[0040] The user behavior serialization feature is the string data generated by encoding the user behavior sequence, which is used to represent the sequence of a user's actions in a day.

[0041] Step 3: Cluster all users based on the extracted user behavior statistical features and user behavior serialization features, and extract class features for each user category;

[0042] In this step, if Figure 2 Figure 1 is a flow chart of user clustering according to an embodiment of the present invention. First, clustering is performed based on user behavior statistical features: the historical average feature vector is calculated based on the historical behavior statistical feature vector; then, the Euclidean distance matrix (Euclidean distance) between all users is calculated based on the pairwise comparison of the historical average feature vectors of all users; then, users are clustered using four hierarchical clustering methods: Ward method, single linkage, complete linkage, and average linkage. The mode of the four hierarchical clustering results is the clustering result C1.

[0043] Specifically, let the historical behavior statistical feature vector of user i be Then the historical average feature vector of all users is expressed as:

[0044]

[0045] Where n is the number of historical records; h ij is the feature vector of user i in the jth record;

[0046] Then calculate the Euclidean distance matrix. For any two users i and k, the Euclidean distance between them is:

[0047]

[0048] Where ||·||2 represents the L2 norm, i.e., the Euclidean distance;

[0049] Then, four hierarchical clustering methods, namely Ward method, single linkage, complete linkage and average linkage, are used to cluster users. The clustering results are expressed as follows:

[0050] C ward , C single , C complete , C average ;

[0051] Determine the final clustering result. The mode of the four hierarchical clustering results is the clustering result C1, which is expressed as:

[0052] C1=mode(C ward , C single , C complete , C average )

[0053] mode 表示用投票方法选出最终结果 ;

[0054] Then clustering is performed based on the user behavior serialization features: the Levenshtein distance matrix between all users is calculated based on the user behavior serialization features; then the users are clustered using four hierarchical clustering methods: Ward method, single linkage, complete linkage, and average linkage. The mode of the four hierarchical clustering results is the clustering result C2;

[0055] Specifically, the Levenshtein distance calculation process between two user behavior serialization feature strings A and B is as follows:

[0056]

[0057] Where m1 is the length of string A; n1 is the length of string B; D[m1][n1] is the Levenshtein distance between strings A and B; the cost function represents the cost of string conversion; i1 is the i1th character of string A; j1 is the j1th character of string B;

[0058] C2 clusters based on Levenshtein distance, and its clustering method is the same as C1;

[0059] Combining the two clustering results C1 and C2, we get the final clustering result C, which is calculated as follows:

[0060] C=C1*N+C2

[0061] Where N is the total number of user categories in the clustering result C1;

[0062] The statistical feature vectors of all users in each user category at the corresponding time are averaged to extract the statistical feature vector of each user category. The calculation formula is:

[0063]

[0064] n2 is the number of users in the class; Class Feature is the statistical feature vector of the user category to be calculated; UserFeatur i2 is the statistical feature vector of the i2th user.

[0065] Step 4: Utilize the features generated by the user's current log processing and the features of the user's category to identify potential abnormal risk behaviors and detect internal threat risks caused by abnormal risk behaviors of users.

[0066] In this step, the statistical behavior characteristics of the user's category over the previous five days are input into a two-layer LSTM network. The first layer has 100 LSTM units and uses tanh as the activation function. The second layer has 160 LSTM units and uses tanh as the activation function. The output of the second layer is fed into a dense layer using ReLU as the activation function, which outputs the predicted statistical behavior characteristics of the user in that category on the sixth day.

[0067] The predicted statistical behavior features are then compared with the features generated by the user's current log processing. The distance between the two feature vectors is used to determine whether the user's current log behavior is risky. The calculation formula for the distance D(H, C, W) is:

[0068]

[0069] Among them, H is the predicted statistical behavior feature vector; C is the feature vector generated by the current log processing of the user; W is the weight vector; i3 is the dimension of the corresponding vector; n3 is the total dimension of the vector;

[0070] The calculation result of the distance D(H, C, W) will fall into 4 non-overlapping intervals. By setting 3 risk thresholds, the user behavior is classified into corresponding levels: normal, low risk 1, low risk 2, and high risk;

[0071] For example, 3 risk thresholds can be set to 6, 9, and 12. If the calculation result of the distance D(H, C, W) ln(D) >= 12, it is determined as high risk; if 6 <= ln(D) < 9, it is determined as low risk 1; if 9 <= ln(D) < 12, it is determined as low risk 2; if 0 < ln(D) < 6, it is determined as normal. In specific implementation, other risk threshold parameters can be set according to the actual situation.

[0072] When the user behavior is determined as high risk, it is directly determined that the current log behavior of the user is abnormal;

[0073] When the user behavior is determined as low risk 1 or low risk 2, it is not temporarily determined that the current log behavior of the user is abnormal. Instead, the user, the risk level, and the current time are recorded in a risk list; the risk list is checked regularly. If it is found that a certain user has both low risk types at the same time, that is, has both low risk 1 and low risk 2, the date record of this user in the risk list is recorded, and the user behavior corresponding to these dates is marked as abnormal.

[0074] It should be noted that the content not described in detail in the embodiments of the present invention belongs to the prior art well-known to those skilled in the art.

[0075] The following uses a specific example to describe the method described in the embodiments of the present invention in detail. For example Figure 3 The following is a schematic flowchart of the implementation method of the example cited in the present invention in the scenario of an internal system of a large company. The employees within the company are used as the users in the embodiments of the present invention, and the logs generated by the employees' computers within the company are the user logs. Specifically:

[0076] The company establishes an internal threat detection platform to detect whether there are threats in the behaviors of company employees. First, the internal threat detection platform collects the historical log records of employees, and executes steps 1 and 2 to extract the statistical features of employees' behaviors and the sequential features of employees' behaviors from the historical log records of employees; then executes step 3 to classify employees using these historical features, extract the statistical features of user classes, and record the classification results and the corresponding statistical features of user classes; next, a prediction model is trained for each user class according to the statistical features of user classes. The above is the preparatory work for the internal threat detection platform to conduct detection.

[0077] When performing detection on the internal threat detection platform, first collect the current log records of all employees in the company, execute steps 1 and 2 to process the current log records to obtain the current behavior characteristics of the employees; next, find the user category to which the currently detected employee belongs based on the processing results of the previous step 3, and call out the statistical characteristics of the user category; finally, execute step 4, input the statistical characteristics of the user category to which the employee belongs into the prediction model to predict the employee's behavior, and calculate the employee's behavior deviation based on the prediction results and the employee's current behavior characteristics, that is, the distance D(H, C, W); finally, judge whether the current employee's behavior poses an abnormal threat based on the interval of the distance D(H, C, W).

[0078] In summary, the method described in the embodiment of the present invention classifies user groups and trains a deep learning model for each user group, thereby greatly reducing the cost of internal threat detection in actual deployment applications; at the same time, the solution of the present invention designs a risk detection and assessment method, and uses a risk list to make the final decision. This method based on comprehensive assessment of multiple risks greatly reduces the misjudgment rate while ensuring detection accuracy, thereby improving the performance of the internal threat detection framework.

[0079] In addition, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0080] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. An internal threat detection method based on user classification and behavior feature analysis, characterized in that: The method comprises: Step 1: Separate the log files to be processed by different users, and organize each user's log files by time and source into a format that is convenient for subsequent operations and processing; Step 2: Identify risks in the user's log file content, extract the user's behavior statistical features and user behavior serialization features, and summarize the user's log file based on the user's behavior statistical features and user behavior serialization features; Step 3: Cluster all users based on the extracted user behavior statistical features and user behavior serialization features, and extract class features for each user category; The process of step 3 is specifically as follows: First, clustering is performed based on user behavior statistical features: the historical average feature vector is calculated based on the historical behavior statistical feature vector; then the Euclidean distance matrix between all users is calculated based on the pairwise comparison of the historical average feature vectors of all users; then, four hierarchical clustering methods, namely Ward method, single linkage, complete linkage, and average linkage, are used to cluster users. The mode of the four hierarchical clustering results is the clustering result C1. Specifically: Assume that the historical behavior statistical feature vector of user i is Then the historical average feature vector of all users is expressed as: Where n is the number of historical records; h ij is the feature vector of user i in the jth record; Then calculate the Euclidean distance matrix. For any two users i and k, the Euclidean distance between them is: Where ||·||2 represents the L2 norm, i.e., the Euclidean distance; Then, four hierarchical clustering methods, namely Ward method, single linkage, complete linkage and average linkage, are used to cluster users. The clustering results are expressed as follows: C ward ,C single ,C complete ,C average ; Determine the final clustering result. The mode of the four hierarchical clustering results is the clustering result C1, which is expressed as: C1=mode(C ward ,C single ,C complete ,C average ) Mode means to use voting method to select the final result; Then clustering is performed based on the serialized features of user behavior: the Levenshtein distance matrix between all users is calculated based on the serialized features of user behavior; then the users are clustered using four hierarchical clustering methods: Ward method, single linkage, complete linkage, and average linkage. The mode of the four hierarchical clustering results is the clustering result C2. Specifically: The Levenshtein distance calculation process of two user behavior serialization feature strings A and B is as follows: Where m1 is the length of string A; n1 is the length of string B; D[m1][n1] is the Levenshtein distance between strings A and B; the cost function represents the cost of string conversion; i1 is the i1th character of string A; j1 is the j1th character of string B; C2 clusters based on Levenshtein distance, and its clustering method is the same as C1; Combining the two clustering results C1 and C2, we get the final clustering result C, which is calculated as follows: C=C1*N+C2 Where N is the total number of user categories in the clustering result C1; The statistical feature vectors of all users in each user category at the corresponding time are averaged to extract the statistical feature vector of each user category. The calculation formula is: n2 is the number of users in the class; Class Feature is the statistical feature vector of the user category to be calculated; UserFeatur i2 is the statistical feature vector of the i2th user; Step 4: Utilize the features generated by the user's current log processing and the features of the user's category to identify potential abnormal risk behaviors and detect internal threat risks caused by abnormal risk behaviors of users.

2. The internal threat detection method based on user classification and behavior feature analysis according to claim 1 is characterized in that: In step 1, the log file to be processed is divided into log records of several users according to the user ID; Split each user's log records by source and arrange them in chronological order; The obtained log records are corrected for errors, redundant information is deleted, and information is filled in to facilitate subsequent operations.

3. The internal threat detection method based on user classification and behavior feature analysis according to claim 1 is characterized in that: In step 2, risk identification of the user's log file content is performed by using a convolutional neural network model for text classification to classify the input log file content into three categories: normal, low-risk, and high-risk. Specifically: First, the input log file content is processed, a character-to-index mapping is defined, and the log file data is cleaned and formatted to prepare its format for the embedding layer of the convolutional neural network model. The embedding layer of the convolutional neural network model converts the previously processed log file input into a dense vector representation, mapping each input character into a high-dimensional space; The Conv1D layer of the convolutional neural network model extracts local features of the log file through 64 filters of size 5; The GlobalMaxPooling1D layer of the convolutional neural network model extracts the strongest signal from each convolutional feature, reducing the data dimension while retaining important information; The Dropout layer of the convolutional neural network model randomly discards features with a probability of 15% to prevent overfitting; The Dense layer of the convolutional neural network model is a fully connected layer used to convert the output of the previous layer into the final classification prediction; The Dense layer uses softmax as the activation function and outputs the probability of each category; Among them, the convolutional neural network model uses the Adam optimizer and categorical cross entropy as the loss function; User behavior statistics are based on each user's log file. They are the count characteristics of each user's behavior activities in a daily time period, including the number of times a user logs in and visits a website in a day. These count characteristics are set based on the different characteristics of user activities and represent user behavior habits. The user behavior serialization feature is a vector that serializes a user's behavior within a day. The specific process of extracting the user behavior serialization feature is as follows: First, the user's log files from different sources in a day are integrated, then all the log entries of the user in a day are arranged in chronological order, and finally the activity information is extracted for serialization representation; the user behavior serialization feature is the string data generated after encoding the user behavior sequence, which is used to represent the sequence of a user's actions in a day.

4. The internal threat detection method based on user classification and behavior feature analysis according to claim 1 is characterized in that: In step 4, the statistical behavior characteristics of the user's category over the previous five days are input into a two-layer LSTM network. The first layer has 100 LSTM units and uses tanh as the activation function. The second layer has 160 LSTM units and uses tanh as the activation function. The output of the second layer is fed into a dense layer using ReLU as the activation function, which outputs the predicted statistical behavior characteristics of the user in that category on the sixth day. The predicted statistical behavior features are then compared with the features generated by the user's current log processing. The distance between the two feature vectors is used to determine whether the user's current log behavior is risky. The calculation formula for the distance D(H, C, W) is: Where H is the predicted statistical behavior feature vector; C is the feature vector generated by the user's current log processing; W is the weight vector; i3 is the dimension of the corresponding vector; n3 is the total dimension of the vector; The calculated results of the distance D(H, C, W) fall into four non-overlapping intervals. Three risk thresholds are set to classify user behaviors into corresponding levels: normal, low risk 1, low risk 2, and high risk. When user behavior is judged to be high-risk, the user's current log behavior is directly judged to be abnormal; When user behavior is judged to be low risk 1 or low risk 2, the user's current log behavior is not temporarily determined to be abnormal. Instead, the user, risk level, and current time are recorded in a risk list. The risk list is checked regularly. If the check finds that a user has two low risk types at the same time, that is, low risk 1 and low risk 2, the user's dates in the risk list are recorded, and the user behavior corresponding to these dates is marked as abnormal.

Images