Abnormal behavior detection method and device

Abstract

The invention provides an abnormal behavior detection method and device, and the method comprises the steps: collecting Web access log data from various sources, carrying out the preprocessing, and extracting key information; constructing a user behavior model based on the user unique identifier ID, and extracting quantized behavior features; calculating a transition probability between user behavior states based on a Markov chain model, and constructing a state transition probability matrix; learning a normal behavior pattern and an abnormal behavior pattern by using a machine learning algorithm, and classifying different types of abnormal behaviors; dynamically setting a threshold value of the abnormal behavior based on a statistical analysis method; and analyzing the real-time Web access log data, calculating an abnormal score of a user behavior, and comparing the abnormal score with a dynamic threshold to judge whether an abnormal behavior exists or not. According to the scheme, the efficient and universal feature extraction and behavior sequence detection method is provided, detection of different abnormal behavior types is supported, the false alarm rate and the missing report rate can be effectively reduced, and the generalization ability is very high.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to an abnormal behavior detection method and device. Background Art

[0002] Abnormal behavior detection methods for web access logs are of great significance in the field of network security. With the rapid development of the Internet, network attacks and malicious behaviors are increasing, posing a serious threat to network security. Abnormal behavior detection can detect and prevent potential network attacks in a timely manner, thereby protecting the security of corporate or personal data.

[0003] Traditional abnormal behavior detection methods are mainly based on statistical analysis. By extracting features and performing statistics on a large number of normal samples, a baseline model of normal behavior is established. Then, the real-time log data is compared with the baseline model. If there is a significant deviation, the behavior is considered abnormal. This type of method mainly focuses on the statistical analysis of large-scale data and has high requirements for computing resources and data storage.

[0004] Abnormal behavior detection based on statistical learning is inefficient, and the detection model has insufficient description capabilities, making it difficult to effectively detect various types of abnormal behaviors. Abnormal behavior detection based on rule bases has the disadvantages of being difficult to manage and requiring frequent updates of rule bases. In addition, statistical models are difficult to fully extract important features in user behavior, resulting in high false positive and false negative rates, poor generalization capabilities, and slow detection speeds. Summary of the invention

[0005] In view of this, the present invention aims to provide an abnormal behavior detection method and device to at least solve one problem in the background technology.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] An abnormal behavior detection method, comprising:

[0008] Collect web access log data from various sources and pre-process it to extract key information;

[0009] Build a user behavior model based on the user's unique ID and extract quantitative behavior features;

[0010] The transition probability between user behavior states is calculated based on the Markov chain model, and the state transition probability matrix is ​​constructed.

[0011] Use machine learning algorithms to learn normal and abnormal behavior patterns and classify different types of abnormal behaviors;

[0012] Dynamically set the threshold of abnormal behavior based on statistical analysis methods, taking into account individual differences of users and time factors.

[0013] Analyze real-time web access log data, calculate the abnormal score of user behavior, and compare it with the dynamic threshold to determine whether there is abnormal behavior.

[0014] Furthermore, the web access log data is collected from multiple sources and pre-processed to extract key information, including:

[0015] Extract user behavior data from the unstructured data of web access logs. First, extract the user's unique ID, source IP, target IP, target URL, access time, and access status code.

[0016] The structured log access information is made persistent, and the user behavior refers to the URLs of users accessing different target IPs.

[0017] Furthermore, the user behavior model is constructed based on the user's unique identification ID, and quantitative behavior features are extracted, including:

[0018] Extract data features by user dimension and extract key indicators, which refers to the number of times each user accesses the target IP through the target URL every day.

[0019] Furthermore, the user behavior model is constructed based on the user's unique identification ID, and quantitative behavior features are extracted, including:

[0020] Based on the user's unique ID, the user's Web access behavior sequence is aggregated to build a user behavior model. Considering the time factor, the user behavior sequence is sorted in chronological order, and features are extracted from the user behavior sequence. The feature extraction of the user behavior sequence includes:

[0021] Access frequency: the number of times a user accesses a URL or IP address per unit time;

[0022] Visit interval: the time interval between two visits by a user;

[0023] Access path: the sequence of URLs accessed by the user;

[0024] Visit depth: the depth of the level at which users visit the website;

[0025] Visit duration: the time a user stays on a page;

[0026] Access type: the type of resource accessed by the user;

[0027] Quantify user behavior characteristics.

[0028] Furthermore, the Markov chain model is used to calculate the transition probability between user behavior states and construct a state transition probability matrix, including:

[0029] Treat each action in the user action sequence as a state;

[0030] Based on the Markov chain model, the transition probability between user behavior states is calculated;

[0031] Construct a state transition probability matrix to describe the transition relationship between user behavior states;

[0032] Considering the time decay factor, different weights are assigned to historical behavior states.

[0033] Furthermore, an exponential decay function is used to assign different weights to historical behavior states, including:

[0034] Determine the time window and set the time window T to calculate the state transition probability;

[0035] Calculate the time interval. For each behavior in the user behavior sequence, calculate the time interval Δt from the current time.

[0036] Calculate the decay weight and use the exponential decay function to calculate the decay weight w(Δt) of each behavior:

[0037] w(Δt)=exp(-λ*Δt);

[0038] Among them, λ is the decay coefficient, which is used to control the decay speed. The larger the λ value, the faster the decay speed, that is, the higher the weight of recent behavior, and the lower the weight of behavior in the distant past. The λ value is adjusted according to the actual situation;

[0039] The state transition probability is calculated weightedly. When calculating the state transition probability, the decay weight of each behavior is considered. For example, assuming that the user performs behavior A at time t1 and performs behavior B at time t2, the transition probability P(A|B) from state A to state B is calculated as:

[0040] P(A|B)=(Σw(Δt)*N(A,B)) / (Σw(Δt)*N(A));

[0041] Where N(A,B) represents the number of times behavior A is followed by behavior B in time window T, N(A) represents the total number of times behavior A appears in time window T, and Δt is the time interval between each behavior A and the current time.

[0042] Furthermore, the method of dynamically setting the threshold of abnormal behavior based on the statistical analysis method includes:

[0043] Analyze real-time Web access log data, extract user behavior features, and calculate the abnormal score of user behavior. Compare the abnormal score of user behavior with the dynamic threshold to determine whether there is abnormal behavior. If there is abnormal behavior, generate an alarm message and perform corresponding processing, which includes: recording abnormal behavior logs, blocking abnormal IP addresses, restricting user access rights, and sending notifications to administrators.

[0044] Furthermore, the real-time web access log data is analyzed, user behavior features are extracted, and the abnormal score of user behavior is calculated, including:

[0045] Calculate the anomaly score based on the state transition probability:

[0046] According to the calculated state transition probability matrix, the probability value of each state transition in the user behavior sequence is obtained, and the path probability of the sequence is calculated, which is the product of all state transition probabilities in the sequence; the anomaly score is defined as the inverse of the path probability;

[0047] Calculate anomaly scores based on abnormal behavior pattern matching:

[0048] The weights are configured for the preset abnormal behavior patterns, and the abnormal score is the sum of the weights of the matched abnormal behavior patterns;

[0049] Anomaly score fusion:

[0050] The anomaly score calculated based on the state transition probability and the anomaly score calculated based on the abnormal behavior pattern matching are combined to obtain the final anomaly score, where:

[0051] The final anomaly score = α*(state transition probability anomaly score) + β*(abnormal behavior pattern matching anomaly score), where α and β are weight coefficients used to adjust the importance of the two anomaly scores.

[0052] Furthermore, the present solution discloses an electronic device, including a processor and a memory that is communicatively connected to the processor and is used to store instructions executable by the processor, wherein the processor is used to execute an abnormal behavior detection method.

[0053] Furthermore, the present solution discloses a server, comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor performs an abnormal behavior detection method.

[0054] Furthermore, the present solution discloses a computer-readable storage medium storing a computer program, wherein the computer program implements an abnormal behavior detection method when executed by a processor.

[0055] Compared with the prior art, the abnormal behavior detection method and device described in the present invention have the following advantages:

[0056] Beneficial effects:

[0057] The abnormal behavior detection method and device described in the present invention provide an efficient and universal feature extraction and behavior sequence detection method, support the detection of different abnormal behavior types, can effectively reduce the false alarm rate and the missed alarm rate, and have a high generalization ability. DETAILED DESCRIPTION

[0058] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0059] The present invention will be described in detail below with reference to embodiments.

[0060] This solution discloses an abnormal behavior detection method. In a first embodiment, this solution includes:

[0061] An abnormal behavior detection method, comprising:

[0062] Collect web access log data from various sources and pre-process it to extract key information;

[0063] Build a user behavior model based on the user's unique ID and extract quantitative behavior features;

[0064] The transition probability between user behavior states is calculated based on the Markov chain model, and the state transition probability matrix is ​​constructed.

[0065] Use machine learning algorithms to learn normal and abnormal behavior patterns and classify different types of abnormal behaviors;

[0066] Dynamically set the threshold of abnormal behavior based on statistical analysis methods, taking into account individual differences of users and time factors.

[0067] Analyze real-time web access log data, calculate the abnormal score of user behavior, and compare it with the dynamic threshold to determine whether there is abnormal behavior.

[0068] Furthermore, the web access log data is collected from multiple sources and pre-processed to extract key information, including:

[0069] Extract user behavior data from the unstructured data of web access logs. First, extract the user's unique ID, source IP, target IP, target URL, access time, and access status code.

[0070] The structured log access information is made persistent, and the user behavior refers to the URLs of users accessing different target IPs.

[0071] Furthermore, the user behavior model is constructed based on the user's unique identification ID, and quantitative behavior features are extracted, including:

[0072] Extract data features by user dimension and extract key indicators, which refers to the number of times each user accesses the target IP through the target URL every day.

[0073] Furthermore, the user behavior model is constructed based on the user's unique identification ID, and quantitative behavior features are extracted, including:

[0074] Based on the user's unique ID, the user's Web access behavior sequence is aggregated to build a user behavior model. Considering the time factor, the user behavior sequence is sorted in chronological order, and features are extracted from the user behavior sequence. The feature extraction of the user behavior sequence includes:

[0075] Access frequency: the number of times a user accesses a URL or IP address per unit time;

[0076] Visit interval: the time interval between two visits by a user;

[0077] Access path: the sequence of URLs accessed by the user;

[0078] Visit depth: the depth of the level at which users visit the website;

[0079] Visit duration: the time a user stays on a page;

[0080] Access type: the type of resource accessed by the user;

[0081] Quantify user behavior characteristics.

[0082] Furthermore, the Markov chain model is used to calculate the transition probability between user behavior states and construct a state transition probability matrix, including:

[0083] Treat each action in the user action sequence as a state;

[0084] Based on the Markov chain model, the transition probability between user behavior states is calculated;

[0085] Construct a state transition probability matrix to describe the transition relationship between user behavior states;

[0086] Considering the time decay factor, different weights are assigned to historical behavior states.

[0087] Furthermore, an exponential decay function is used to assign different weights to historical behavior states, including:

[0088] Determine the time window and set the time window T to calculate the state transition probability;

[0089] Calculate the time interval. For each behavior in the user behavior sequence, calculate the time interval Δt from the current time.

[0090] Calculate the decay weight and use the exponential decay function to calculate the decay weight w(Δt) of each behavior:

[0091] w(Δt)=exp(-λ*Δt);

[0092] Among them, λ is the decay coefficient, which is used to control the decay speed. The larger the λ value, the faster the decay speed, that is, the higher the weight of recent behavior, and the lower the weight of behavior in the distant past. The λ value is adjusted according to the actual situation;

[0093] The state transition probability is calculated weightedly. When calculating the state transition probability, the decay weight of each behavior is considered. For example, assuming that the user performs behavior A at time t1 and performs behavior B at time t2, the transition probability P(A|B) from state A to state B is calculated as:

[0094] P(A|B)=(Σw(Δt)*N(A,B)) / (Σw(Δt)*N(A));

[0095] Where N(A,B) represents the number of times behavior A is followed by behavior B in time window T, N(A) represents the total number of times behavior A appears in time window T, and Δt is the time interval between each behavior A and the current time.

[0096] Furthermore, the method of dynamically setting the threshold of abnormal behavior based on the statistical analysis method includes:

[0097] Analyze real-time Web access log data, extract user behavior features, and calculate the abnormal score of user behavior. Compare the abnormal score of user behavior with the dynamic threshold to determine whether there is abnormal behavior. If there is abnormal behavior, generate an alarm message and perform corresponding processing, which includes: recording abnormal behavior logs, blocking abnormal IP addresses, restricting user access rights, and sending notifications to administrators.

[0098] Furthermore, the real-time web access log data is analyzed, user behavior features are extracted, and the abnormal score of user behavior is calculated, including:

[0099] Calculate the anomaly score based on the state transition probability:

[0100] According to the calculated state transition probability matrix, the probability value of each state transition in the user behavior sequence is obtained, and the path probability of the sequence is calculated, which is the product of all state transition probabilities in the sequence; the anomaly score is defined as the inverse of the path probability;

[0101] Calculate anomaly scores based on abnormal behavior pattern matching:

[0102] The weights are configured for the preset abnormal behavior patterns, and the abnormal score is the sum of the weights of the matched abnormal behavior patterns;

[0103] Anomaly score fusion:

[0104] The anomaly score calculated based on the state transition probability and the anomaly score calculated based on the abnormal behavior pattern matching are combined to obtain the final anomaly score, where:

[0105] The final anomaly score = α*(state transition probability anomaly score) + β*(abnormal behavior pattern matching anomaly score), where α and β are weight coefficients used to adjust the importance of the two anomaly scores.

[0106] In another embodiment, the solution may include the following processing steps:

[0107] (1) Data preprocessing: extracting user behavior data from the unstructured data of web access logs. First, extract the user's unique ID, source IP, target IP, target URL, access time, and access status code; and make the structured log access information persistent. The user behavior refers to the URLs of users accessing different target IPs.

[0108] (2) Feature extraction: extract data features according to the user dimension and extract key indicators. This indicator refers to the number of times each user accesses the target IP through the target URL every day.

[0109] (3) Initial probability calculation: the initial probability refers to the ratio of the frequency of occurrence of different behaviors to the frequency of occurrence of all behaviors;

[0110] (4) State transition probability. The probability of changing between different states is called transition probability. The transition probability matrix is ​​calculated based on the state transition process. Assuming that the user's access behavior obeys the Markov property, the probability distribution of the next state of the behavior can only be determined by the previous state. The events before it in the time series are irrelevant. First, calculate the state pair (the previous and next two states) in the time series, and calculate the probability value of state transition based on the frequency statistics of the state pair.

[0111] (5) Window probability: Set the length of the unit window to a constant value, calculate the initial probability and the sum of the transition probabilities according to the length of the unit window, and get the window probability;

[0112] (6) Model saving and loading: Model saving and loading refers to saving the initial probability calculation results and the transition probability matrix of the behavior as parameters in a binary file and saving it as a model. When the model is tested, the model file is read and loaded to obtain the complete parameters of the model and test the web behavior data to be tested.

[0113] (7) Threshold calculation: according to the statistical distribution of the state value, the value of the state value at the 3sigma critical point is calculated;

[0114] (8) Abnormality determination: When detecting user behavior, the window probability value of the behavior is first calculated and compared with the threshold. Behavior that exceeds the threshold is determined to be abnormal behavior.

[0115] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0116] In the several embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the division of the units described above is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The above-mentioned units may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting abnormal behavior, characterized in that: include: Collect web access log data from various sources and pre-process it to extract key information; Build a user behavior model based on the user's unique ID and extract quantitative behavior features; Calculate the transition probability between user behavior states based on the Markov chain model and construct the state transition probability matrix; Use machine learning algorithms to learn normal and abnormal behavior patterns and classify different types of abnormal behaviors; Dynamically set abnormal behavior thresholds based on statistical analysis methods, taking into account individual differences among users and time factors; Analyze real-time web access log data, calculate the abnormal score of user behavior, and compare it with the dynamic threshold to determine whether there is abnormal behavior.

2. The abnormal behavior detection method according to claim 1, characterized in that: The web access log data is collected from multiple sources and preprocessed to extract key information, including: Extract user behavior data from the unstructured data of web access logs. First, extract the user's unique ID, source IP, target IP, target URL, access time, and access status code. The structured log access information is made persistent, and the user behavior refers to the URLs of different target IPs accessed by the user.

3. The abnormal behavior detection method according to claim 1, characterized in that: The user behavior model is constructed based on the user's unique ID, and quantitative behavior features are extracted, including: Extract data features by user dimension and extract key indicators, which refers to the number of times each user accesses the target IP through the target URL every day.

4. The abnormal behavior detection method according to claim 1, characterized in that: The user behavior model is constructed based on the user's unique ID, and quantitative behavior features are extracted, including: Based on the user's unique ID, the user's Web access behavior sequence is aggregated to build a user behavior model. Considering the time factor, the user behavior sequence is sorted in chronological order, and features are extracted from the user behavior sequence. The feature extraction of the user behavior sequence includes: Access frequency: the number of times a user accesses a URL or IP address per unit time; Visit interval: the time interval between two visits by a user; Access path: the sequence of URLs accessed by the user; Visit depth: the depth of the level at which users visit the website; Visit duration: the time a user stays on a page; Access type: the type of resource accessed by the user; Quantify user behavior characteristics.

5. The abnormal behavior detection method according to claim 1, characterized in that: The Markov chain model is used to calculate the transition probability between user behavior states and construct a state transition probability matrix, including: Treat each action in the user action sequence as a state; Based on the Markov chain model, the transition probability between user behavior states is calculated; Construct a state transition probability matrix to describe the transition relationship between user behavior states; Considering the time decay factor, different weights are assigned to historical behavior states.

6. The abnormal behavior detection method according to claim 5, characterized in that: Use an exponential decay function to assign different weights to historical behavior states, including: Determine the time window and set the time window T to calculate the state transition probability; calculate the time interval. For each behavior in the user behavior sequence, calculate the time interval Δt from the current time; Calculate the decay weight and use the exponential decay function to calculate the decay weight w(Δt) of each behavior: w(Δt)=exp(-λ*Δt); Among them, λ is the decay coefficient, which is used to control the decay speed. The larger the λ value, the faster the decay speed, that is, the higher the weight of recent behavior, and the lower the weight of behavior in the distant past. The λ value is adjusted according to the actual situation; The state transition probability is calculated in a weighted manner. When calculating the state transition probability, the attenuation weight of each behavior is considered.

7. The abnormal behavior detection method according to claim 1, characterized in that: The method of dynamically setting the threshold of abnormal behavior based on the statistical analysis method includes: Analyze real-time Web access log data, extract user behavior features, and calculate the abnormal score of user behavior. Compare the abnormal score of user behavior with the dynamic threshold to determine whether there is abnormal behavior. If there is abnormal behavior, generate an alarm message and perform corresponding processing, which includes: recording abnormal behavior logs, blocking abnormal IP addresses, restricting user access rights, and sending notifications to administrators.

8. The abnormal behavior detection method according to claim 7, characterized in that: The analysis of real-time web access log data, extraction of user behavior features, and calculation of abnormal scores of user behaviors include: Calculate the anomaly score based on the state transition probability: According to the calculated state transition probability matrix, the probability value of each state transition in the user behavior sequence is obtained, and the path probability of the sequence is calculated, which is the product of all state transition probabilities in the sequence; the anomaly score is defined as the inverse of the path probability; Calculate anomaly scores based on abnormal behavior pattern matching: The weights are configured for the preset abnormal behavior patterns, and the abnormal score is the sum of the weights of the matched abnormal behavior patterns; Anomaly score fusion: The anomaly score calculated based on the state transition probability and the anomaly score calculated based on the abnormal behavior pattern matching are combined to obtain the final anomaly score, where: The final anomaly score = α*(state transition probability anomaly score) + β*(abnormal behavior pattern matching anomaly score), where α and β are weight coefficients used to adjust the importance of the two anomaly scores.

9. An electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: The processor is used to execute an abnormal behavior detection method as described in any one of claims 1-8 above.

10. A server, characterized in that: It includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes an abnormal behavior detection method as described in any one of claims 1-8.