Internal threat detection method based on user classification and behavior feature analysis

Abstract

The invention discloses an internal threat detection method based on user classification and behavior feature analysis, which comprises the following steps of: separating log files to be processed according to different users, and arranging the log files of each user into a format convenient for subsequent operation processing according to time and sources; performing risk identification on the log file content of the user, and extracting behavior statistical characteristics and user behavior serialization characteristics of the user; clustering all users based on the extracted user behavior statistical features and user behavior serialization features, and extracting class features of each user class; and identifying potential abnormal risk behaviors by using the features generated by current log processing of the user and the features of the category to which the user belongs, and detecting the internal threat risk caused by the abnormal risk behaviors of the user. According to the method, the deployment and maintenance cost of the detection framework is greatly reduced, internal threat detection can be carried out by using a lower-cost detection technology, and the method has a relatively high detection rate.

Description

Technical Field

[0001] The present invention relates to the technical field of user anomaly detection, and in particular to an internal threat detection method based on user classification and behavior feature analysis. Background Art

[0002] At present, the research on internal threat detection mainly relies on machine learning and deep learning methods to determine whether there are internal threats. Past methods usually use various feature extraction techniques to process user logs and use these features to train machine learning or deep learning models to detect internal threats. In general, these methods are committed to extracting users' daily behavior patterns and applying them to the detection of abnormal user behavior.

[0003] Although some previous studies have achieved certain internal threat detection capabilities, they still have some shortcomings. First, existing studies only focus on the behavioral information in user logs, but ignore the information contained in the specific content of some log information, making the information incomplete; second, existing studies maintain a machine learning or deep learning model for each user, and the deployment cost and operation and maintenance cost are too high in actual application scenarios, making it difficult to truly implement applications; finally, existing studies only make one decision for user anomaly detection. In order to ensure the accuracy of detection, some normal behaviors are judged as abnormal, resulting in a high false positive rate. Summary of the invention

[0004] The purpose of the present invention is to provide an internal threat detection method based on user classification and behavior feature analysis, which greatly reduces the deployment and maintenance costs of the detection framework, can use lower-cost detection technology for internal threat detection, and has a higher detection rate.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] An internal threat detection method based on user classification and behavior feature analysis, the method comprising:

[0007] Step 1: Separate the log files to be processed according to different users, and organize the log files of each user according to time and source into a format that is convenient for subsequent operation and processing;

[0008] Step 2: Identify risks in the user's log file content, extract the user's behavior statistical features and user behavior serialization features, and summarize the user's log file through the user's behavior statistical features and user behavior serialization features;

[0009] Step 3: Cluster all users based on the extracted user behavior statistical features and user behavior serialization features, and extract class features for each user category;

[0010] Step 4: Use the features generated by the user's current log processing and the features of the user's category to identify potential abnormal risk behaviors and detect internal threat risks caused by the user's abnormal risk behaviors.

[0011] It can be seen from the technical solution provided by the present invention that the above method greatly reduces the deployment and maintenance costs of the detection framework, can use lower-cost detection technology for internal threat detection, and has a higher detection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0013] Figure 1 A schematic diagram of a flow chart of an internal threat detection method based on user classification and behavior feature analysis provided by an embodiment of the present invention;

[0014] Figure 2 A schematic diagram of the process of user clustering according to an embodiment of the present invention;

[0015] Figure 3 This is a flowchart of an implementation example of the present invention in a large company internal system scenario. DETAILED DESCRIPTION

[0016] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] like Figure 1 FIG. 1 is a flow chart of an internal threat detection method based on user classification and behavior feature analysis provided by an embodiment of the present invention, wherein the method comprises:

[0018] Step 1: Separate the log files to be processed according to different users, and organize the log files of each user according to time and source into a format that is convenient for subsequent operation and processing;

[0019] In this step, the log file to be processed is first divided into log records of several users according to the user ID;

[0020] Split each user's log records by source (e.g. logon, email, http, device, file) and arrange them in chronological order;

[0021] The obtained log records are processed to correct errors, delete redundant information and fill in information to facilitate subsequent operations.

[0022] Step 2: Identify risks in the user's log file content, extract the user's behavior statistical features and user behavior serialization features, and summarize the user's log file through the user's behavior statistical features and user behavior serialization features;

[0023] In this step, the risk identification of the user's log file content is carried out by using a convolutional neural network model for text classification to classify the input log file content into three categories: normal, low risk, and high risk. Specifically:

[0024] First, the input log file content is processed, a character-to-index mapping is defined, and the log file data is cleaned and formatted to prepare its format for the embedding layer of the convolutional neural network model;

[0025] The embedding layer of the convolutional neural network model converts the previously processed log file input into a dense vector representation, mapping each input character into a high-dimensional space;

[0026] The Conv1D layer of the convolutional neural network model extracts local features of the log file through 64 filters of size 5;

[0027] The GlobalMaxPooling1D layer of the convolutional neural network model extracts the strongest signal from each convolutional feature, reducing the data dimension while retaining important information;

[0028] The Dropout layer of the convolutional neural network model randomly discards features with a probability of 15% to prevent overfitting;

[0029] The Dense layer of the convolutional neural network model is a fully connected layer that converts the output of the previous layer into the final classification prediction; the Dense layer uses softmax as the activation function and the output result is the probability of each category;

[0030] Among them, the convolutional neural network model uses the Adam optimizer and categorical cross entropy as the loss function;

[0031] In the specific implementation, the user's behavior statistical features are based on each user's log file as the information source, such as logon, email, http, device, file; as shown in Table 1 below, the user's behavior is statistically analyzed:

[0032] Table 1 User behavior statistics items

[0033]

[0034] The user's behavior statistics are the counting characteristics of each user's behavior activities in a time period divided into days, including the number of times a user logs in and visits a website in a day. These counting characteristics are set according to the different characteristics of user activities and represent the user's behavior habits.

[0035] The user behavior serialization feature is a vector that serializes a user's behavior within a day. The specific process of extracting the user behavior serialization feature is as follows:

[0036] First, the log files from different sources of the user in a day are integrated, then all the log entries of the user in a day are arranged in chronological order, and finally the activity information is extracted for serialization representation;

[0037] Among them, the activity information in the form of a string is difficult to be directly used as input for subsequent operations, so the activity information needs to be encoded as shown in the following Table 2:

[0038] Table 2 Activity information coding

[0039]

[0040] The user behavior serialization feature is the string data generated by encoding the user behavior sequence, which is used to represent the sequence of a user's actions in a day.

[0041] Step 3: Cluster all users based on the extracted user behavior statistical features and user behavior serialization features, and extract class features for each user category;

[0042] In this step, if Figure 2 The figure shows a flow chart of user clustering according to an embodiment of the present invention. First, clustering is performed based on user behavior statistical features: the historical average feature vector is calculated based on the historical behavior statistical feature vector; then the Euclidean distance matrix (Euclidean distance) between all users is calculated based on the historical average feature vectors of all users; then the users are clustered using four hierarchical clustering methods, namely, Ward method, single linkage, complete linkage and average linkage, and the mode of the four hierarchical clustering results is the clustering result C1;

[0043] Specifically, let the historical behavior statistical feature vector of user i be Then the historical average feature vector of all users is expressed as:

[0044]

[0045] where n is the number of historical records; h ij is the feature vector of user i in the jth record;

[0046] Then calculate the Euclidean distance matrix. For any two users i and k, the Euclidean distance between them is:

[0047]

[0048] Where ||·||2 represents the L2 norm, i.e., the Euclidean distance;

[0049] Then, four hierarchical clustering methods, namely Ward method, single linkage, complete linkage and average linkage, are used to cluster users. The clustering results are expressed as follows:

[0050] C ward , C single , C complete , C average ;

[0051] Determine the final clustering result. The mode of the four hierarchical clustering results is the clustering result C1, which is expressed as:

[0052] C1=mode(C ward , C single , C complete , C average )

[0053] mode 表示用投票方法选出最终结果 ;

[0054] Then clustering is performed based on the user behavior serialization features: the Levenshtein distance matrix between all users is calculated based on the user behavior serialization features; the users are clustered using four hierarchical clustering methods: Ward method, single linkage, complete linkage, and average linkage. The mode of the four hierarchical clustering results is the clustering result C2;

[0055] Specifically, the Levenshtein distance calculation process of two user behavior serialization feature strings A and B is as follows:

[0056]

[0057] Where m1 is the length of string A; n1 is the length of string B; D[m1][n1] is the Levenshtein distance between string A and string B; the cost function represents the cost of string conversion; i1 is the i1th character of string A; j1 is the j1th character of string B;

[0058] C2 clusters based on Levenshtein distance, and its clustering method is the same as C1;

[0059] Combining the two clustering results C1 and C2, we get the final clustering result C, and the calculation formula is:

[0060] C=C1*N+C2

[0061] Where N is the total number of user categories in the clustering result C1;

[0062] The statistical feature vectors of all users in each user category at the corresponding time are averaged to extract the statistical feature vectors of each user category. The calculation formula is:

[0063]

[0064] n2 is the number of users in the class; Class Feature is the statistical feature vector of the user category to be calculated; UserFeatur i2 is the statistical feature vector of the i2th user.

[0065] Step 4: Use the features generated by the user's current log processing and the features of the user's category to identify potential abnormal risk behaviors and detect internal threat risks caused by the user's abnormal risk behaviors.

[0066] In this step, the statistical behavior characteristics of the category to which the user belongs in the previous five days are first input into a two-layer LSTM network. The first layer of the LSTM network has 100 LSTM units and uses tanh as the activation function. The second layer of the LSTM network has 160 LSTM units and uses tanh as the activation function. The output of the second layer of the LSTM network is sent to a dense layer, which uses ReLU as the activation function, and the output is the predicted value of the statistical behavior characteristics of the category of users on the sixth day.

[0067] Then compare the predicted statistical behavior features with the features generated by the user's current log processing, and judge whether the user's current log behavior is risky by the distance between the two feature vectors. The calculation formula of the distance D(H, C, W) is:

[0068]

[0069] Among them, H is the predicted statistical behavior feature vector; C is the feature vector generated by the current log processing of the user; W is the weight vector; i3 is the dimension of the corresponding vector; n3 is the total dimension of the vector;

[0070] The calculation result of the distance D(H, C, W) will fall into 4 non-overlapping intervals. By setting 3 risk thresholds, the user behavior is classified into corresponding levels: normal, low risk 1, low risk 2, and high risk;

[0071] For example, 3 risk thresholds can be set to 6, 9, and 12. If the calculation result of the distance D(H, C, W) ln(D) >= 12, it is determined as high risk; if 6 <= ln(D) < 9, it is determined as low risk 1; if 9 <= ln(D) < 12, it is determined as low risk 2; if 0 < ln(D) < 6, it is determined as normal. In specific implementation, other risk threshold parameters can be set according to the actual situation.

[0072] When the user behavior is determined as high risk, it is directly determined that the current log behavior of the user is abnormal;

[0073] When the user behavior is determined as low risk 1 or low risk 2, it is not temporarily determined that the current log behavior of the user is abnormal. Instead, the user, the risk level, and the current time are recorded in a risk list; the risk list is checked regularly. If it is found that a certain user has both low risk types, that is, both low risk 1 and low risk 2, the date record of this user in the risk list is recorded, and the user behavior corresponding to these dates is marked as abnormal.

[0074] It should be noted that the content not described in detail in the embodiments of the present invention belongs to the prior art well-known to those skilled in the art.

[0075] The method described in the embodiments of the present invention will be described in detail below with specific examples. For example, Figure 3 The following is a schematic flow chart of the implementation manner of the example cited in the present invention in the scenario of an internal system of a large company. The employees within the company are used as the users in the embodiments of the present invention, and the logs generated by the employees' computers within the company are the user logs. Specifically:

[0076] The company establishes an internal threat detection platform to detect whether there are threats in the behaviors of company employees. First, the internal threat detection platform collects the historical log records of employees, executes steps 1 and 2, and extracts the employee behavior statistical features and employee behavior sequence features from the historical log records of employees; then executes step 3 to classify employees using these historical features, extract the statistical features of user classes, and record the classification results and the corresponding statistical features of user classes; next, a prediction model is trained for each user class according to the statistical features of the user class. The above is the preparatory work for the internal threat detection platform to conduct detection.

[0077] When the internal threat detection platform is used for detection, the current log records of all employees of the company are first collected, and steps 1 and 2 are executed to process the current log records to obtain the current behavior characteristics of the employees; next, the user category to which the currently detected employee belongs is found according to the processing result of the previous step 3, and the statistical characteristics of the user category are retrieved; finally, step 4 is executed to input the statistical characteristics of the user category to which the employee belongs into the prediction model to predict the employee's behavior, and the employee's behavior deviation, that is, the distance D (H, C, W) is calculated based on the prediction results and the employee's current behavior characteristics; finally, the current employee's behavior is judged whether it is an abnormal threat based on the interval of the distance D (H, C, W).

[0078] In summary, the method described in the embodiment of the present invention classifies user groups and trains a deep learning model for each user group, thereby greatly reducing the cost of internal threat detection in actual deployment applications; at the same time, the solution of the present invention designs a risk detection and assessment method, and uses a risk list to make the final decision. This method based on comprehensive assessment of multiple risks greatly reduces the misjudgment rate while ensuring detection accuracy, thereby improving the performance of the internal threat detection framework.

[0079] In addition, a person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0080] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. An internal threat detection method based on user classification and behavior feature analysis, characterized in that: The method comprises: Step 1: Separate the log files to be processed according to different users, and organize the log files of each user according to time and source into a format that is convenient for subsequent operation and processing; Step 2: Identify risks in the user's log file content, extract the user's behavior statistical features and user behavior serialization features, and summarize the user's log file through the user's behavior statistical features and user behavior serialization features; Step 3: Cluster all users based on the extracted user behavior statistical features and user behavior serialization features, and extract class features for each user category; Step 4: Use the features generated by the user's current log processing and the features of the user's category to identify potential abnormal risk behaviors and detect internal threat risks caused by the user's abnormal risk behaviors.

2. The internal threat detection method based on user classification and behavior feature analysis according to claim 1 is characterized in that: In step 1, the log file to be processed is divided into log records of several users according to the user ID; Split each user's log records by source and arrange them in chronological order; The obtained log records are processed to correct errors, delete redundant information and fill in information to facilitate subsequent operations.

3. The internal threat detection method based on user classification and behavior feature analysis according to claim 1 is characterized in that: In step 2, the risk identification of the user's log file content is carried out by using a convolutional neural network model for text classification to classify the input log file content into three categories: normal, low risk, and high risk. Specifically: First, the input log file content is processed, a character-to-index mapping is defined, and the log file data is cleaned and formatted to prepare its format for the embedding layer of the convolutional neural network model; The embedding layer of the convolutional neural network model converts the previously processed log file input into a dense vector representation, mapping each input character into a high-dimensional space; The Conv1D layer of the convolutional neural network model extracts local features of the log file through 64 filters of size 5; The GlobalMaxPooling1D layer of the convolutional neural network model extracts the strongest signal from each convolutional feature, reducing the data dimension while retaining important information; The Dropout layer of the convolutional neural network model randomly discards features with a probability of 15% to prevent overfitting; The Dense layer of the convolutional neural network model is a fully connected layer that converts the output of the previous layer into the final classification prediction; The Dense layer uses softmax as the activation function and outputs the probability of each category. Among them, the convolutional neural network model uses the Adam optimizer and categorical cross entropy as the loss function; The user's behavior statistics feature uses each user's log file as the information source. The user's behavior statistics feature is the counting feature of each user's behavior activities in a time period divided into days, including the number of times a user logs in and visits a website in a day. These counting features are set according to different characteristics of user activities and represent the user's behavior habits. The user behavior serialization feature is a vector that serializes a user's behavior within a day. The specific process of extracting the user behavior serialization feature is as follows: First, the log files from different sources of the user in a day are integrated, then all the log entries of the user in a day are arranged in chronological order, and finally the activity information is extracted for serialization representation; the user behavior serialization feature is the string data generated after encoding the user behavior sequence, which is used to represent the sequence of a user's actions in a day.

4. The internal threat detection method based on user classification and behavior feature analysis according to claim 1 is characterized in that: The process of step 3 is specifically as follows: First, clustering is performed based on the statistical characteristics of user behavior: the historical average feature vector is calculated based on the historical behavior statistical feature vector; then the Euclidean distance matrix between all users is calculated based on the pairwise comparison of the historical average feature vectors of all users; Then, four hierarchical clustering methods, namely Ward method, single linkage, complete linkage and average linkage, are used to cluster users. The mode of the four hierarchical clustering results is the clustering result C1. Specifically: Assume that the historical behavior statistical feature vector of user i is Then the historical average feature vector of all users is expressed as: where n is the number of historical records; h ij is the feature vector of user i in the jth record; Then calculate the Euclidean distance matrix. For any two users i and k, the Euclidean distance between them is: Where ||·||2 represents the L2 norm, i.e., the Euclidean distance; Then, four hierarchical clustering methods, namely Ward method, single link, complete link and average link, are used to cluster users. The clustering results are expressed as follows: C ward ,C single ,C complete ,C average ; Determine the final clustering result. The mode of the four hierarchical clustering results is the clustering result C1, which is expressed as: C1=mode(C ward ,C single ,C complete ,C average ) mode means to select the final result by voting; Then clustering is performed based on the user behavior serialization features: the Levenshtein distance matrix between all users is calculated based on the user behavior serialization features; then the users are clustered using four hierarchical clustering methods: Ward method, single linkage, complete linkage, and average linkage. The mode of the four hierarchical clustering results is the clustering result C2. Specifically: The Levenshtein distance calculation process of two user behavior serialization feature strings A and B is as follows: Where m1 is the length of string A; n1 is the length of string B; D[m1][n1] is the Levenshtein distance between string A and string B; the cost function represents the cost of string conversion; i1 is the i1th character of string A; j1 is the j1th character of string B; C2 clusters based on Levenshtein distance, and its clustering method is the same as C1; Combining the two clustering results C1 and C2, we get the final clustering result C, and the calculation formula is: C=C1*N+C2 Where N is the total number of user categories in the clustering result C1; The statistical feature vectors of all users in each user category at the corresponding time are averaged to extract the statistical feature vectors of each user category. The calculation formula is: n2 is the number of users in the class; Class Feature is the statistical feature vector of the user category to be calculated; UserFeatur i2 is the statistical feature vector of the i2th user.

5. The internal threat detection method based on user classification and behavior feature analysis according to claim 1 is characterized in that: In step 4, the statistical behavior characteristics of the category to which the user belongs in the previous five days are first input into a two-layer LSTM network. The first layer of the LSTM network has 100 LSTM units and uses tanh as the activation function. The second layer of the LSTM network has 160 LSTM units and uses tanh as the activation function. The output of the second layer of the LSTM network is sent to a dense layer, which uses ReLU as the activation function, and the output is the predicted value of the statistical behavior characteristics of the category of users on the sixth day. Then compare the predicted statistical behavior features with the features generated by the user's current log processing, and judge whether the user's current log behavior is risky by the distance between the two feature vectors. The calculation formula of the distance D(H, C, W) is: Among them, H is the predicted statistical behavior feature vector; C is the feature vector generated by the user's current log processing; W is the weight vector; i3 is the dimension of the corresponding vector; n3 is the total dimension of the vector; The calculation results of the distance D(H, C, W) will fall into four non-overlapping intervals. By setting three risk thresholds, user behaviors are divided into corresponding levels: normal, low risk 1, low risk 2, and high risk; When a user's behavior is judged to be high risk, the user's current log behavior is directly judged to be abnormal; When user behavior is judged as low risk 1 or low risk 2, the user's current log behavior is not judged as abnormal for the time being. Instead, the user, risk level, and current time are recorded in a risk list. The risk list is checked regularly. If the check finds that a user has both low risk types, that is, low risk 1 and low risk 2, the user's dates in the risk list are recorded, and the user behaviors corresponding to these dates are marked as abnormal.

Images