User behavior backtracking mode based on log collection

Abstract

The invention discloses a user behavior backtracking mode based on log collection, and relates to the field of information monitoring and auditing, and the method comprises the following steps: 1, capturing user behaviors through a front-end JavaScript event monitor, constructing a standard log data object, sending the standard log data object to a rear-end RESTful API interface through AJAX, and storing log data to an Ali cloud SLS by using a Java Spring Boot framework; according to the method, user behaviors are captured through the front-end JavaScript and sent to the rear end, consistency and readability of log data are ensured, consistency and validity check is carried out on log data objects, it is ensured that the data are within the format, integrity and effective range, the risk of data errors is reduced, the log data are stored and processed through the Alicloud SLS, and the user experience is improved. Abnormal operation is recognized by combining big data analysis and an isolated forest algorithm, abnormal operation in a log data object is effectively recognized, an enterprise is helped to find potential safety risks and operation errors in time, and finally a tracing report is generated.

Description

Technical Field

[0001] The present invention relates to the field of information monitoring and auditing, and in particular to a user behavior backtracking method based on log collection. Background Art

[0002] With the rapid development of computer and network technologies, the complexity of information systems and the diversity of network environments have increased significantly, bringing with them increasingly severe security threats. Frequent security incidents such as cyberattacks, data leaks, and internal violations pose significant risks to business operations and reputation. Therefore, companies urgently need to strengthen their ability to monitor and audit user behavior in order to promptly identify and respond to potential security threats.

[0003] User behavior backtracking has become a crucial security management tool for addressing potential security threats. By recording and analyzing user actions within the system, enterprises can achieve audit tracking, compliance checks, and legal evidence collection. This not only helps identify abnormal operations but also provides strong support for subsequent security incident investigations.

[0004] Existing technologies have the following shortcomings: They are static and post-processing, relying primarily on post-incident data processing. Traditional log analysis often fails to support real-time monitoring, resulting in delays in responding to security incidents and increasing potential risks. Centralized log storage and analysis systems can experience processing delays when faced with massive amounts of data, hindering timely auditing of user behavior. Logs from different systems and applications are often stored on separate platforms, lacking a unified perspective, limiting comprehensive analysis of user behavior. Over-reliance on rules and pattern matching makes it difficult to address complex anomalies, leading to potential security threats being overlooked.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0006] The purpose of the present invention is to provide a user behavior backtracking method based on log collection to solve the problems in the above background technology.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a user behavior backtracking method based on log collection, comprising the following steps:

[0008] Step 1: Capture user behavior through front-end JavaScript event listeners, construct standard log data objects, and send them to the back-end RESTful API interface via AJAX. Use the Java Spring Boot framework to store the log data in Alibaba Cloud SLS.

[0009] Step 2: Use the Alibaba Cloud SLS console to create a log project and log repository, set a log retention policy, use the Alibaba Cloud Java SDK to batch compress and send log data objects, attach an AccessKey for authentication, and configure monitoring and retry mechanisms.

[0010] Step 3: Classify and deduplicate log data objects by building a tree structure. Analyze invalid log features using a decision tree. Check data consistency and validity. Divide into topic partitions and fill in missing values. Use a big data processing framework to count user behavior, extract active time periods and frequently visited pages, and identify high-value users and churned users.

[0011] Step 4: Deploy ClickHouse through Alibaba Cloud, create a database and table structure to store user behavior log data, use the batch insert function to import log data objects in CSV format, and use query verification and audit purpose classification. Combined with the isolation forest algorithm, identify abnormal operations and compile them into a traceability report.

[0012] Preferably, a JavaScript event listener is introduced through the front-end, and the addEventListener method is used to capture the user's front-end behavior. According to the event trigger of the user's front-end behavior, a standard log data object containing the user behavior is constructed, including the user ID, operation type, page path and trigger timestamp. The log data object is forwarded to the designated server endpoint of the back-end gateway through AJAX asynchronous communication, and a RESTful API interface is developed through the back-end and the Java Spring Boot framework is used to receive the log data object and store it in Alibaba Cloud SLS.

[0013] Preferably, create a log project through the Alibaba Cloud SLS console to organize and manage log data objects, create multiple log repositories to store different types of log data, set the retention policy of the log repository, set the retention event of the log data object to 7 days, use the Java SDK provided by Alibaba Cloud to directly send the log data object to Alibaba Cloud SLS, attach AccessKey for authentication, compress the log data object through Gzip and implement a batch sending mechanism, merge multiple log data objects into one request for sending, set monitoring and alarm rules and implement a retry mechanism, and trigger the retry mechanism when the log data object drops sharply or the request fails to send.

[0014] Preferably, a tree structure is constructed, the branch nodes inside the tree structure represent the invalid log features of the log data object, the leaf nodes inside the tree structure represent the invalid removal results, multiple decision trees are constructed by randomly selecting invalid object features and log data objects, the invalid object features of the branch nodes are randomly selected according to the leaf nodes for optimal splitting, independent classification prediction is performed on each decision tree, the classification prediction that appears most frequently in all decision trees is selected as the invalid object classification, invalid log data objects are judged and removed, duplicate log data objects are checked and deleted, the format of the log data objects is ensured to be consistent by processing the trigger timestamp, the log data objects are checked for null values and missing values, and the log data objects are checked for Within the valid range, the log data objects are checked for anomalies and duplicate records through the constraints of uniqueness and consistency. For different existence situations, themes are determined and divided into different theme partitions. The corresponding log data objects are deleted from each partition, and the log data objects are filled accordingly using the mean filling method. Group statistics are performed according to user ID, operation type and trigger timestamp. The frequency of distributed computing operations, user activity and access time are calculated through the big data processing framework, and the user's active time period, most frequently visited pages, and frequent operation behaviors are extracted. High-activity time periods and high-access page frequencies are classified as high-value users, and low-activity time periods and low-access page frequencies are classified as lost users.

[0015] Preferably, ClickHouse is deployed through Alibaba Cloud, the server is connected to and a database is created through the ClickHouse client, the database structure table is defined with user behavior as the column type, the log data object is converted to CSV format, the log data object is inserted into the table using the batch insert function, query verification is performed through the SELECT*FROM command, the audit purpose is divided into user behavior tracking, abnormal operation identification and compliance checking, and the log data object keywords are marked, query conditions are constructed according to the audit purpose, the constructed query is executed through the log query service console, the abnormal operation records contained in the log data object are identified in combination with the isolated forest, the abnormal operation records contained in the log data object are organized into a traceability report, and the timeline view is used to provide a detailed description of tracing user behavior.

[0016] Preferably, for the selected feature, a value "val" is randomly selected as the split value, and the log data object feature and the split value are used as the root node of the isolation tree. For each log data object, starting from the root node of the isolation tree, move downward along the branch until reaching the leaf node of the isolation tree, define the depth normalization value of each isolation tree, divide the log data object into two isolated tree subsets, left and right, according to the log data object feature and the split value, iterate the random selection and division steps for the left and right isolated tree subsets respectively, and construct isolated subtrees on the isolated tree subsets until the stop condition is triggered, and combine the isolated subtrees to form an isolation forest. The abnormal operation records contained in the log data object are identified through the anomaly analysis of the isolation forest. The specific formula is:

[0017]

[0018] Among them, S(x,n) represents the abnormal score of the log data object on n isolated subtrees, E(h(x)) represents the expected value of the depth of the log data object on all isolated subtrees, and c(n) represents the average depth of the log data object in all isolated subtrees in the isolation forest. When S(x,n)=1, it means that there are abnormal operation records contained in the log data object.

[0019] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0020] Through front-end event monitoring, users' operational behaviors on the website are captured in real time. Standard log data objects are built to ensure the consistency and readability of log data. Asynchronous communication using AJAX will not affect the user's browsing experience and reduce page freezes or delays. RESTful API interfaces are developed through the Java Spring Boot framework to conveniently receive and process log data from the front-end and store it in Alibaba Cloud SLS to achieve persistent data storage. Log projects and log libraries are created through the Alibaba Cloud SLS console to organize and manage different types of log data in an orderly manner. Retention policies for log libraries are set to help enterprises effectively manage storage space and prevent unlimited growth of log data, thereby reducing storage costs. AccessKey is used for authentication to ensure that only authorized users can access and send log data, thereby improving security. The Java SDK provided by Alibaba Cloud is used to conveniently send log data objects to Alibaba Cloud SLS. Batch compression and sending are supported to reduce the burden of network transmission and improve data transmission efficiency.

[0021] Compressing log data with Gzip can significantly reduce data size, shorten transmission time and bandwidth consumption, and improve overall performance. Combining multiple log data objects into one request for sending can effectively reduce the number of requests. Effective log management and transmission mechanisms ensure that high-quality log data is stored and analyzed in a timely manner. Monitoring and retry mechanisms ensure that log data collection and storage will not be interrupted in the event of network or system failures, thereby supporting business continuity and stability.

[0022] Constructing a tree structure to classify and deduplicate log data objects can effectively identify and delete invalid logs, improve data accuracy and effectiveness, and ensure that data is in the format, completeness and valid range by checking the consistency and validity of log data objects, thereby reducing the risk of data errors. By processing missing values through the mean filling method, data integrity is ensured and analysis bias caused by missing values is avoided. By analyzing user activity and access frequency, high-value users can be identified, helping companies to develop more accurate marketing strategies and user maintenance measures. By analyzing low-activity time periods and low access frequencies, lost users can be identified, providing companies with a basis for improving user experience and increasing user stickiness. According to reports, by creating database and table structures through ClickHouse, user behavior log data can be stored efficiently. Using the batch insert function to import log data objects in CSV format can greatly improve data import efficiency and reduce the time and errors of manual operations. Combined with the isolation forest algorithm, it can effectively identify abnormal operations in log data objects, help companies to promptly discover potential security risks and operational errors, and organize abnormal operation records into traceability reports to help companies quickly respond to and handle abnormal situations. Through the division of audit purposes, user behavior can be comprehensively tracked. The isolation forest algorithm effectively identifies abnormal operations by constructing isolation trees and isolation forests, with high accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction to the drawings required for use in the embodiments will be given below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0024] Figure 1 This is a flow chart of a method for tracing user behavior based on log collection in the present invention. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0026] The present invention provides Figure 1 The user behavior backtracking method based on log collection shown includes the following steps:

[0027] Step 1: Capture user behavior through front-end JavaScript event listeners, construct standard log data objects, and send them to the back-end RESTful API interface via AJAX. Use the Java Spring Boot framework to store the log data in Alibaba Cloud SLS.

[0028] A JavaScript event listener is introduced on the front end, and the addEventListener method is used to capture user front-end behaviors. Based on the events triggered by the user front-end behaviors, a standard log data object containing the user behaviors is constructed, including the user ID, operation type, page path, and trigger timestamp. The log data object is forwarded to the designated server endpoint of the back-end gateway through AJAX asynchronous communication. A RESTful API interface is developed on the back end, and the Java Spring Boot framework is used to receive the log data object and store it in Alibaba Cloud SLS.

[0029] Step 2: Use the Alibaba Cloud SLS console to create a log project and log repository, set a log retention policy, use the Alibaba Cloud Java SDK to batch compress and send log data objects, attach an AccessKey for authentication, and configure monitoring and retry mechanisms.

[0030] Create a log project through the Alibaba Cloud SLS console to organize and manage log data objects. Create multiple log repositories to store different types of log data. Set the log repository retention policy and set the retention period for log data objects to 7 days. Use the Java SDK provided by Alibaba Cloud to directly send log data objects to Alibaba Cloud SLS, attaching an AccessKey for authentication. Compress log data objects using Gzip and implement a batch sending mechanism. Combine multiple log data objects into a single request for sending. Set monitoring and alert rules and implement a retry mechanism. The retry mechanism is triggered when the number of log data objects decreases sharply or the request fails to send.

[0031] Step 3: Classify and deduplicate log data objects by building a tree structure. Analyze invalid log features using a decision tree. Check data consistency and validity. Divide into topic partitions and fill in missing values. Use a big data processing framework to count user behavior, extract active time periods and frequently visited pages, and identify high-value users and churned users.

[0032] Construct a tree structure, and let the branch nodes inside the tree structure represent the invalid log features of the log data object, and the leaf nodes inside the tree structure represent the invalid removal results. Construct multiple decision trees by randomly selecting invalid object features and log data objects, and randomly select invalid object features of branch nodes according to leaf nodes for optimal splitting. Make independent classification predictions for each decision tree, and select the classification prediction that appears most frequently in all decision trees as the invalid object classification. Judge and remove invalid log data objects, check and delete duplicate log data objects, ensure the consistency of the format of log data objects by processing trigger timestamps, check whether there are null values and missing values in log data objects, and check whether log data objects are in Within the effective range, the log data objects are checked for anomalies and duplicate records through the constraints of uniqueness and consistency. For different existence situations, themes are determined and divided into different theme partitions. The corresponding log data objects are deleted from each partition, and the log data objects are filled in accordingly using the mean filling method. Group statistics are performed according to user ID, operation type and trigger timestamp. The frequency of distributed computing operations, user activity and access time are calculated through the big data processing framework, and the user's active time period, most frequently visited pages, and frequent operation behaviors are extracted. The high-activity time period and high-access page frequency users are classified as high-value users, and the low-activity time period and low-access page frequency are classified as lost users.

[0033] Step 4: Deploy ClickHouse on Alibaba Cloud, create a database and table structure to store user behavior log data, use the batch insert function to import CSV format log data objects, perform query verification and audit purpose classification, combine the isolation forest algorithm to identify abnormal operations, and compile them into a traceability report;

[0034] Deploy ClickHouse through Alibaba Cloud, connect to the server and create a database through the ClickHouse client, define the database structure table and use user behavior as the column type, convert log data objects into CSV format, use the batch insert function to insert log data objects into the table, perform query verification through the SELECT * FROM command, divide the audit objectives into user behavior tracking, abnormal operation identification and compliance checking, and mark the log data object keywords, construct query conditions according to the audit objectives, execute the constructed query through the log query service console, combine the isolated forest to identify the abnormal operation records contained in the log data objects, organize the abnormal operation records contained in the log data objects into a traceability report, and use the timeline view to provide detailed descriptions of tracing user behavior.

[0035] For the selected features, randomly select a value "val" as the split value, use the log data object features and the split value as the root node of the isolation tree, and for each log data object, start from the root node of the isolation tree and move down along the branch until reaching the leaf node of the isolation tree. Define the depth normalization value of each isolation tree, and divide the log data object into two isolated tree subsets, left and right, according to the log data object features and the split value. Iterate the random selection and division steps for the left and right isolated tree subsets respectively to construct isolated subtrees on the isolated tree subsets until the stop condition is triggered. The isolated subtrees are combined to form an isolation forest. The abnormal operation records contained in the log data object are identified through the anomaly analysis of the isolation forest. The specific formula is:

[0036]

[0037] Among them, S(x,n) represents the abnormal score of the log data object on n isolated subtrees, E(h(x)) represents the expected value of the depth of the log data object on all isolated subtrees, and c(n) represents the average depth of the log data object in all isolated subtrees in the isolation forest. When S(x,n)=1, it means that there are abnormal operation records contained in the log data object.

[0038] Through front-end event monitoring, users' operational behaviors on the website are captured in real time. Standard log data objects are built to ensure the consistency and readability of log data. Asynchronous communication using AJAX will not affect the user's browsing experience and reduce page freezes or delays. RESTful API interfaces are developed through the Java Spring Boot framework to conveniently receive and process log data from the front-end and store it in Alibaba Cloud SLS to achieve persistent data storage. Log projects and log libraries are created through the Alibaba Cloud SLS console to organize and manage different types of log data in an orderly manner. Retention policies for log libraries are set to help enterprises effectively manage storage space and prevent unlimited growth of log data, thereby reducing storage costs. AccessKey is used for authentication to ensure that only authorized users can access and send log data, thereby improving security. The Java SDK provided by Alibaba Cloud is used to conveniently send log data objects to Alibaba Cloud SLS. Batch compression and sending are supported to reduce the burden of network transmission and improve data transmission efficiency.

[0039] Compressing log data with Gzip can significantly reduce data size, shorten transmission time and bandwidth consumption, and improve overall performance. Combining multiple log data objects into one request for sending can effectively reduce the number of requests. Effective log management and transmission mechanisms ensure that high-quality log data is stored and analyzed in a timely manner. Monitoring and retry mechanisms ensure that log data collection and storage will not be interrupted in the event of network or system failures, thereby supporting business continuity and stability.

[0040] Constructing a tree structure to classify and deduplicate log data objects can effectively identify and delete invalid logs, improve data accuracy and effectiveness, and ensure that data is in the format, completeness and valid range by checking the consistency and validity of log data objects, thereby reducing the risk of data errors. By processing missing values through the mean filling method, data integrity is ensured and analysis bias caused by missing values is avoided. By analyzing user activity and access frequency, high-value users can be identified, helping companies to develop more accurate marketing strategies and user maintenance measures. By analyzing low-activity time periods and low access frequencies, lost users can be identified, providing companies with a basis for improving user experience and increasing user stickiness. According to reports, by creating database and table structures through ClickHouse, user behavior log data can be stored efficiently. Using the batch insert function to import log data objects in CSV format can greatly improve data import efficiency and reduce the time and errors of manual operations. Combined with the isolation forest algorithm, it can effectively identify abnormal operations in log data objects, help companies to promptly discover potential security risks and operational errors, and organize abnormal operation records into traceability reports to help companies quickly respond to and handle abnormal situations. Through the division of audit purposes, user behavior can be comprehensively tracked. The isolation forest algorithm effectively identifies abnormal operations by constructing isolation trees and isolation forests, with high accuracy and robustness.

[0041] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.

Claims

1. A user behavior backtracking method based on log collection, characterized in that: The following steps are involved: Step 1: Capture user behavior through front-end JavaScript event listeners, construct standard log data objects, send them to the back-end RESTful API via AJAX, and use the Java Spring Boot framework to store the log data in Alibaba Cloud SLS. Step 2: Use the Alibaba Cloud SLS console to create a log project and log repository, set a log retention policy, use the Alibaba Cloud Java SDK to batch compress and send log data objects, attach an AccessKey for authentication, and configure monitoring and retry mechanisms. Step 3: Classify and deduplicate log data objects by building a tree structure. Analyze invalid log features using a decision tree. Check data consistency and validity. Divide into topic partitions and fill in missing values. Use a big data processing framework to count user behavior, extract active time periods and frequently visited pages, and identify high-value users and churned users. Step 4: Deploy ClickHouse through Alibaba Cloud, create a database and table structure to store user behavior log data, use the batch insert function to import log data objects in CSV format, and use query verification and audit purpose classification. Combined with the isolation forest algorithm, identify abnormal operations and compile them into a traceability report.

2. The user behavior backtracking method based on log collection according to claim 1 is characterized by: In step 1, a JavaScript event listener is introduced through the front-end, and the addEventListener method is used to capture the user's front-end behavior. According to the event trigger of the user's front-end behavior, a standard log data object containing the user behavior is constructed, including the user ID, operation type, page path and trigger timestamp. The log data object is forwarded to the designated server endpoint of the back-end gateway through AJAX asynchronous communication. A RESTful API interface is developed through the back-end and the Java Spring Boot framework is used to receive the log data object and store it in Alibaba Cloud SLS.

3. The user behavior backtracking method based on log collection according to claim 1 is characterized by: In the step 2, a log project is created through the Alibaba Cloud SLS console to organize and manage log data objects, multiple log repositories are created to store different types of log data, a retention policy for the log repositories is set, and the retention event of the log data object is set to 7 days. The Java SDK provided by Alibaba Cloud is used to directly send the log data object to Alibaba Cloud SLS, with an AccessKey for authentication. The log data object is compressed using Gzip and a batch sending mechanism is implemented. Multiple log data objects are merged into one request for sending. Monitoring and alarm rules are set and a retry mechanism is implemented. When the number of log data objects decreases sharply or the request fails to be sent, the retry mechanism is triggered.

4. The user behavior backtracking method based on log collection according to claim 1 is characterized by: In the step three, a tree structure is constructed, invalid log data objects are judged and removed, duplicate log data objects are checked and deleted, the format of the log data objects is ensured to be consistent by processing the trigger timestamp, the log data objects are checked for null values and missing values, the log data objects are checked for being within the valid range, the log data objects are checked for anomalies and duplicate records by the constraints of uniqueness and consistency, the topics are determined for different existence situations and divided into different topic partitions, the corresponding log data objects are deleted from each partition, and the log data objects are filled in accordingly using the mean filling method, group statistics are performed according to user ID, operation type and trigger timestamp, the frequency of distributed computing operations, user activity and access time are calculated through the big data processing framework, and the user's active time period, the most frequently visited pages, and frequent operation behaviors are extracted, and the high-activity time period and high-access page frequency are divided into high-value users, and the low-activity time period and low-access page frequency are divided into lost users.

5. The user behavior backtracking method based on log collection according to claim 4 is characterized by: The specific steps of constructing the tree structure are: the branch nodes inside the tree structure represent the invalid log features of the log data object, the leaf nodes inside the tree structure represent the invalid removal results, and multiple decision trees are constructed by randomly selecting invalid object features and log data objects. The invalid object features of the branch nodes are randomly selected according to the leaf nodes for optimal splitting. Each decision tree is independently classified and predicted, and the classification prediction that appears most frequently in all decision trees is selected as the invalid object classification.

6. The user behavior backtracking method based on log collection according to claim 1 is characterized by: In the step 4, ClickHouse is deployed through Alibaba Cloud, the server is connected to and a database is created through the ClickHous e client, the database structure table is defined and user behavior is used as the column type, the log data object is converted to CSV format, the batch insert function is used to insert the log data object into the table, query verification is performed through the SELECT * FROM command, the audit purpose is divided into user behavior tracking, abnormal operation identification and compliance checking, and the log data object keywords are marked, query conditions are constructed according to the audit purpose, the constructed query is executed through the log query service console, the abnormal operation records contained in the log data object are identified in combination with the isolated forest, the abnormal operation records contained in the log data object are organized into a traceability report, and the timeline view is used to provide a detailed description of tracing user behavior.

7. The user behavior backtracking method based on log collection according to claim 1 is characterized by: In the fourth step, for the selected feature, a value "val" is randomly selected as the split value, and the log data object feature and the split value are used as the root node of the isolation tree. For each log data object, starting from the root node of the isolation tree, move downward along the branch until reaching the leaf node of the isolation tree, define the depth normalization value of each isolation tree, divide the log data object into two isolated tree subsets, left and right, according to the log data object feature and the split value, iterate the random selection and division steps for the left and right isolated tree subsets respectively, and construct isolated subtrees on the isolated tree subsets until the stop condition is triggered, and combine the isolated subtrees to form an isolation forest, and identify abnormal operation records contained in the log data object through the anomaly analysis of the isolation forest.

8. The user behavior backtracking method based on log collection according to claim 7 is characterized by: The specific formula for anomaly identification of the isolation forest is: Among them, S(x,n) represents the abnormal score of the log data object on n isolated subtrees, E(h(x)) represents the expected value of the depth of the log data object on all isolated subtrees, and c(n) represents the average depth of the log data object in all isolated subtrees in the isolation forest. When S(x,n)=1, it means that there are abnormal operation records contained in the log data object.

Images