Log data classification and grading and malicious behavior analysis method and device and medium

Abstract

The invention relates to the field of log processing, and discloses a log data classification and grading and malicious behavior analysis method and device and a medium, and the method comprises the steps: obtaining Web log data, and carrying out data preprocessing to obtain log dynamic gradient features; performing clustering evaluation on the log dynamic gradient features to obtain an evaluation result; identifying potential malicious behaviors according to the evaluation results exceeding the preset value, and intercepting the potential malicious behaviors; through deep analysis of malicious behaviors in Web log data, an enterprise can identify and intercept potential malicious attacks more accurately, so that the overall security protection level is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of log processing, and in particular to a method for classifying and grading log data and analyzing malicious behavior. Background Technology

[0002] According to the relevant definition of Web log data, Web log data refers to the data recorded by the server regarding requests and responses when a user accesses the server through a browser or client. Malicious behaviors contained in this data, such as DDoS attacks, malicious URL links, SQL injection, etc., can seriously threaten the information security of network users.

[0003] In the field of web network security, classifying and grading web log data and analyzing malicious behavior within it can be considered a method for identifying and monitoring threats, such as known behavioral characteristics like IP address, request URL, request method, status code, and other related characteristics.

[0004] Traditional network security primarily focuses on the network and application layers. The network layer controls data flow by enabling web application firewalls, while application layer security measures mainly include authentication and access control. While these two security models can mitigate most network attacks to a certain extent, they still have limitations, and the ultimate effectiveness is limited by the server's ability to analyze malicious behavior in web log data. Summary of the Invention

[0005] The purpose of this invention is to propose a method, device, and medium for classifying and grading log data and analyzing malicious behavior, thereby solving the technical problems of weak server malicious behavior analysis capabilities and weak protection capabilities in traditional network security protection.

[0006] Specifically, the present invention provides a method for classifying and grading log data and analyzing malicious behavior, comprising the following steps: S1. Obtain Web log data and perform data preprocessing to obtain dynamic gradient characteristics of the logs; S2. Perform clustering evaluation on the dynamic gradient features of the logs to obtain the evaluation results; S3. Identify potential malicious behavior by evaluating results that exceed preset values ​​and intercept them.

[0007] A storage medium that stores instructions and data for implementing a log data classification and grading and malicious behavior analysis method.

[0008] A log data classification and grading and malicious behavior analysis device includes: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement a log data classification and grading and malicious behavior analysis method.

[0009] The beneficial effects provided by this invention are: by deeply analyzing malicious behavior in Web log data, enterprises can more accurately identify and block potential malicious attacks, thereby significantly improving the overall level of security protection. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the method process of the present invention; Figure 2 This is a schematic diagram of the hardware device operation according to an embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0012] Before formally describing the present invention, a general description of the solution of the present invention will be given first to facilitate understanding.

[0013] Please refer to Figure 1 The present invention provides a method for classifying and grading log data and analyzing malicious behavior, comprising: S1. Obtain Web log data and perform data preprocessing to obtain dynamic gradient characteristics of the logs; It should be noted that step S1 first classifies and grades the log data, segments it according to the time series, and identifies potential gradual change characteristics, including fusion characteristics and slow change characteristics. These characteristics will improve the accuracy and efficiency of malicious behavior analysis in log data.

[0014] Step S1 is as follows: S11. Obtain Web log data; This invention first obtains web log data generated in the background from the server in real time. In order to effectively manage and retrieve this data, Redis, a high-performance key-value database, is used for data storage.

[0015] S12. Classify the logs based on the type of Web log data to obtain the classified logs; It should be noted that, based on the type of Web log data, this invention can categorize Web log data into permission logs, access logs, and error logs. Permission logs focus on recording user login and permission data; access logs record detailed data of each user's requests, such as request time, requested resources, and other related data; error logs record all abnormal situations and error data encountered by users during server access, such as username, permission level, requested URL, HTTP status code, and other key characteristics.

[0016] S13. The classified logs are further divided into levels to obtain the logs with complete levels; It should be noted that, based on the access frequency of Web log data, this invention can categorize Web log data into three levels: red, yellow, and green. Red Web log data represents data with high access frequency; yellow Web log data represents data with normal access frequency, typically indicating normal user behavior; and green Web log data represents data with low access frequency, typically indicating less frequently accessed resources or functions.

[0017] S14. The logs that have been graded are segmented using a sliding window to obtain the segmented sub-time series. It should be noted that, for the log data that has been classified and graded as described above, it is sorted according to the level in permission logs, access logs, and error logs. Based on this, a fixed sliding window is used to perform sliding segmentation on the log time series in turn.

[0018] Time series segmentation is performed using a fixed sliding window, as shown below:

[0019] The parameters for the fixed sliding window are set as follows: This represents the i-th time series. This represents the size of the i-th time window, set to 5 minutes. This represents the i-th time window. This indicates the window's sliding step size, set to 2 minutes.

[0020] S15. Identify the trends of clustering points and gradual change points in the segmented subsequences; It should be noted that, based on each segmented sub-time series, a sliding window trend statistics strategy is used to accurately identify the trend of clustering points and gradual change points in the series. If the trend suddenly increases, it is determined to be a clustering point; if the trend is flat, it is determined to be a gradual change point. The series is divided into multiple different regions, and the number of clustering points and gradual change points in each region is monitored in real time. Clustering points usually indicate potential malicious behavior, while gradual change points usually indicate normal access behavior patterns.

[0021] S16. Combine the key feature fields of the logs to perform fine segmentation on the trend of the convergence point and the gradual change point, and obtain finely segmented sub-time series; In this invention, based on the pre-segmented sub-time series, and according to the distribution of the number of convergence points and gradual change points in each region, while also considering key feature fields in the log data (such as username, IP address, port number, etc.), a variable sliding window is used to perform more refined segmentation of the data around the convergence points and gradual change points. Subsequently, a dedicated dataset is constructed to store these segmented data results.

[0022] Use a variable sliding window to cut sub-time series, as shown below:

[0023] The variable sliding window parameters are set as follows: This represents the i-th sub-time series. This represents the size of the i-th time window, set to 1 minute. This represents the i-th time window. This indicates the sliding step size of the variable window, which can be set to 10 seconds, 15 seconds, etc., and dynamically increased sequentially until the window is cut off.

[0024] S17. Analyze the clustering points and gradual change points in the finely segmented sub-time series to extract the dynamic gradual change features of the log.

[0025] In this invention, during the in-depth analysis of the dataset, it is necessary to analyze each fusion point and gradual change point, and dynamically extract the feature information around these points. Based on the trend of data change, these features can be clearly divided into fusion features and gradual change features. Fusion features mainly reflect the drastic changes that occur in the data in a short period of time, while gradual change features reveal the pattern of gradual changes in the data over a longer period of time.

[0026] In summary, the dynamic gradient feature extraction in this invention aims to identify dynamic gradient features in Web log data, monitor the regions where gradient features appear, segment the log time series, and extract the clustering and gradual variation features of each region. Clustering features often exhibit a rapid change trend and are usually considered as potential malicious behavior, while gradual variation features show a stable and gradual change pattern, reflecting more normal user access behavior.

[0027] 1) Identifying Dynamic Gradual Changes: Gradual changes can be divided into two main categories: clustering features and gradual change features. For web log data generated by the server, time series are segmented to identify clustering points and gradual change trends within the sequence. In high-concurrency network environments, the number of clustering and gradual change points in the sequence is monitored. Clustering points typically indicate potential malicious behavior, while gradual change points typically indicate the behavior of users during normal access.

[0028] 2) Gradual Change Feature Extraction: A variable sliding window is used to further segment the sub-time series and construct a dataset to dynamically extract fusion and gradual change features around fusion points and gradual change points. Fusion features are often considered to be carried by potential malicious behavior, and features related to such behavior need to be immediately identified and blocked. Although the source of gradual change features is reliable and the access frequency is normal, the analysis results still need further verification to confirm the possibility of spoofing.

[0029] S2. Perform clustering evaluation on the dynamic gradient features of the logs to obtain the evaluation results; It should be noted that step S2 mainly involves further verifying and analyzing slowly varying features that have the potential to be disguised. Their frequency, correlation, and influence are calculated, and the results are sorted, aggregated, and added to the feature set. Based on this, a binary iterative strategy is used to perform residual segmentation of the feature set, and for each segmented sub-feature set, stepwise regression analysis is used for feature evaluation.

[0030] Specifically, step S2 includes the following steps: S21. Calculate the frequency of occurrence of slowly varying features; First, it is necessary to statistically analyze the potential, disguised, and slowly varying features that require further verification, and systematically record the frequency of these features. Then, use loops to iterate through the dataset and calculate and accumulate the number of times each feature appears.

[0031] S22. Analyze the correlation between the slowly changing characteristics and other surrounding cluster characteristics, and calculate the correlation degree; Secondly, we further analyze the interaction between this feature and other fusion features around it, and use the Pearson correlation coefficient to calculate the correlation between this feature and other fusion features. Finally, we use hash value normalization to effectively map the correlation to the numerical range of [0,1].

[0032] The correlation between this feature and other fusion features was calculated using the Pearson correlation coefficient, as shown below:

[0033] Where n is the number of elements in the set. It is the eigenvalue of the gradually changing feature at the i-th gradually changing point. It is the eigenvalue of the fusion characteristic at the i-th fusion point. It is the characteristic mean at the point of gradual change. R(G,F) is the characteristic mean at the fusion point, and R(G,F) is the correlation between this slowly varying characteristic and a certain fusion characteristic in the surrounding area.

[0034] Hash value normalization is used to effectively map the correlation to the numerical range [0,1], as shown below:

[0035] Here, RD represents the correlation degree mapped to the numerical range of [0,1]. CRC16 is a commonly used cyclic redundancy check algorithm in computer networks. Here, it is used to calculate the hash value of the correlation degree and perform modulo operation. 65535 is the maximum value of the hash value. Taking the remainder of it maps the correlation degree to the numerical range of [0,1].

[0036] S23. Combining correlation and frequency of occurrence, the slowly changing features are quickly sorted and aggregated to obtain the filtered slowly changing features and form a feature set. It should be noted that step S23 is as follows: S231. Select slowly changing features whose correlation exceeds a preset value to obtain the first slowly changing feature; In this invention, considering both frequency of association and correlation, quicksort is used to achieve accurate sorting and aggregation of all features in the dataset. Specifically, when the correlation of a behavior is higher than 1, this is usually a strong signal indicating that the behavior may indicate potential malicious activity. In such cases, it is necessary to immediately identify and effectively intercept these behaviors to prevent the risks they may pose. While behaviors with a correlation of less than 1 may be less urgent, they still warrant further analysis.

[0037] S232. Calculate the influence of the first slowly varying feature to obtain the second slowly varying feature, as shown in the following formula:

[0038] Where INF represents the magnitude of the influence of the current feature. This represents the correlation degree of the i-th feature. This represents the correlation degree of the j-th feature; This represents the frequency of occurrence of the i-th feature. This represents the frequency of occurrence of the j-th feature; First, we need to understand the influence relationships between features. Specifically, when a slow-varying feature is significantly associated with most other fusion features in the surrounding area, or is associated with only a few fusion features but these fusion features have relatively high frequencies, then the influence of that feature can be considered to be very high. Then, we can combine the frequency of occurrence and the degree of association of features to calculate the influence of each feature separately.

[0039]

[0040] Wherein, INF represents the magnitude of the influence of the current feature. This represents the correlation degree of the i-th feature. This represents the correlation degree of the j-th feature; This represents the frequency of occurrence of the i-th feature. This represents the frequency of occurrence of the j-th feature.

[0041] S233. Select the features with the highest influence in the second gradual change features as the filtered gradual change features. After calculating the influence, these features need to be sorted and aggregated to ensure that features with higher influence are ranked higher. Then, a new feature set is created, and based on the sorted and aggregated results, these features are added to this set one by one in an orderly manner.

[0042] S24. Divide the feature set into multiple sub-feature sets based on a binary iterative strategy; The resulting feature set needs to be divided into multiple relatively stable sub-feature sets. Based on a sliding window, a binary iterative strategy is used to perform residual segmentation of the feature set. After segmentation, multiple relatively stable sub-feature sets will be formed. It is necessary to ensure that their feature means are distinct and reflect different statistical patterns. Furthermore, this operation can, to some extent, eliminate the potential interference of malicious spoofing on the analysis results and mitigate the nonlinearity and non-stationarity biases introduced by the sliding window.

[0043] S25. Use regression analysis to evaluate the features in the sub-feature set and obtain the evaluation results; It should be noted that, firstly, data from the sub-feature sets are extracted sequentially, and stepwise regression analysis is used to evaluate the features, returning the evaluation results. Then, based on the evaluation results and the regression equation, the influence of the features is calculated, and the results are ranked. The definition formula of the regression equation is shown below:

[0044] y represents the evaluation result. This represents the influence of feature n after calculation. This represents the degree of correlation of feature n.

[0045] S3. Identify potential malicious behavior by evaluating results that exceed preset values ​​and intercept them.

[0046] Specifically, in this invention, based on the sorted results, behaviors with an evaluation result greater than 80% are generally considered potentially malicious behaviors that need to be immediately identified and intercepted, and then returned to the server.

[0047] Please see Figure 2 , Figure 2 This is a schematic diagram of the hardware device in operation according to an embodiment of the present invention. The hardware device specifically includes: a log data classification and grading and malicious behavior analysis device 401, a processor 402, and a storage medium 403.

[0048] A log data classification and grading and malicious behavior analysis device 401: The log data classification and grading and malicious behavior analysis device 401 implements the log data classification and grading and malicious behavior analysis method.

[0049] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the log data classification and grading and malicious behavior analysis method.

[0050] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the log data classification and grading and malicious behavior analysis method.

[0051] The beneficial effects of this invention are: by deeply analyzing malicious behavior in Web log data, enterprises can more accurately identify and block potential malicious attacks, thereby significantly improving the overall level of security protection.

[0052] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for classifying and grading log data and analyzing malicious behavior, characterized in that: Includes the following steps: S1. Obtain Web log data and perform data preprocessing to obtain dynamic gradient characteristics of the logs; S2. Perform clustering evaluation on the dynamic gradient features of the logs to obtain the evaluation results; S3. Identify potential malicious behavior by evaluating results that exceed preset values ​​and intercept them.

2. The log data classification, grading, and malicious behavior analysis method as described in claim 1, characterized in that: Step S1 is as follows: S11. Obtain Web log data; S12. Classify the logs based on the type of Web log data to obtain the classified logs; S13. The classified logs are further divided into levels to obtain the logs with complete levels; S14. The logs that have been graded are segmented using a sliding window to obtain the segmented sub-time series. S15. Identify the trends of clustering points and gradual change points in the segmented subsequences; S16. Combine the key feature fields of the logs to perform fine segmentation on the trend of the convergence point and the gradual change point, and obtain finely segmented sub-time series; S17. Analyze the clustering points and gradual change points in the finely segmented sub-time series to extract the dynamic gradual change features of the log.

3. The log data classification, grading, and malicious behavior analysis method as described in claim 1, characterized in that: The dynamic gradient characteristics of the logs include: fusion characteristics and slow-change characteristics; fusion characteristics are considered potential malicious behavior and are directly blocked.

4. The log data classification, grading, and malicious behavior analysis method as described in claim 2, characterized in that: In step S14, the sliding window is a fixed window, and the specific processing is as follows: in Represents the i-th time series. This represents the size of the i-th time window. This represents the i-th time window. This indicates the sliding step size of the window.

5. The log data classification, grading, and malicious behavior analysis method as described in claim 2, characterized in that: In step S17, the fine segmentation employs a variable sliding window, as detailed below: in This represents the i-th sub-time series. This represents the size of the i-th time window. This represents the i-th time window. This indicates the sliding step size of the variable window.

6. The log data classification and grading and malicious behavior analysis method as described in claim 1, characterized in that: Step S2 is as follows: S21. Calculate the frequency of occurrence of slowly varying features; S22. Analyze the correlation between the slowly changing characteristics and other surrounding cluster characteristics, and calculate the correlation degree; S23. Combining correlation and frequency of occurrence, the slowly changing features are quickly sorted and aggregated to obtain the filtered slowly changing features and form a feature set. S24. Divide the feature set into multiple sub-feature sets based on a binary iterative strategy; S25. Use regression analysis to evaluate the features in the sub-feature set and obtain the evaluation results.

7. The log data classification, grading, and malicious behavior analysis method as described in claim 1, characterized in that: Step S23 is as follows: S231. Select slowly changing features whose correlation exceeds a preset value to obtain the first slowly changing feature; S232. Calculate the influence of the first slowly varying feature to obtain the second slowly varying feature, as shown in the following formula: Where INF represents the magnitude of the influence of the current feature. This represents the correlation degree of the i-th feature. This represents the correlation degree of the j-th feature; This represents the frequency of occurrence of the i-th feature. This represents the frequency of occurrence of the j-th feature; S233. Select the features with the highest influence among the second gradual change features as the filtered gradual change features.

8. The log data classification and grading and malicious behavior analysis method as described in claim 7, characterized in that: In step S25, the regression analysis is specifically as follows: y represents the evaluation result. This represents the influence of feature n after calculation. This represents the degree of correlation of feature n.

9. A storage medium, characterized in that: The storage medium stores instructions and data to implement the log data classification and grading and malicious behavior analysis method according to any one of claims 1 to 8.

10. A log data classification and grading and malicious behavior analysis device, characterized in that: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the log data classification and grading and malicious behavior analysis method according to any one of claims 1 to 8.

Images