A Web log-based intrusion analysis method and system

Abstract

The present invention discloses a method and system for intrusion analysis based on web logs. The analysis method includes: cutting the original log into fields according to the default delimiter and the default quote character; automatically identifying the field format, performing rapid matching of the field format, parsing and storing in a database; reading the stored logs in batches and performing log matching rule detection; updating the logs and writing them back to the database; aggregating the logs to generate web attack events and determine whether the intruder's attack was successful; generating and submitting a protection and processing report based on the web attack time and the analysis results and the impact range of the intrusion confirmed manually. The present invention realizes intrusion analysis of the attacker's IP address by cutting the web logs, uniformly parsing, rule matching detection and log aggregation. The present invention adopts a method combining automatic detection and manual confirmation, which has the advantages of high efficiency, high result accuracy, high log parsing strength and convenience for secondary analysis, thereby greatly improving the efficiency of web analysis.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a Web log-based intrusion analysis method and system. Background Art

[0002] As one of the most important applications on the internet, the web provides a convenient mechanism for publishing and retrieving documents and has gradually become a hub for a wide range of information resources. The abundance and diversity of this information has lured hackers, leading to an increasing number of attacks on web applications, particularly those targeting web servers. These attacks are diverse and potentially devastating, sometimes leaking company user information or, worse still, paralyzing servers.

[0003] When users access web services through clients, web servers generate web access logs based on their requests. Web access logs record various raw information, including the web server's receipt and processing of requests and runtime errors. Web logs can be used to analyze and trace user behavior, such as whether attacks occurred, most frequently visited pages, address distribution, and access device distribution. Security analysis of web logs can not only help locate attackers but also retrace attack paths, identify website security vulnerabilities, and address them accordingly.

[0004] Web services need to be hosted on various web servers. Common web servers include Nginx, Apache, Tomcat, and IIS. Logs from these servers carry similar information, including records of user access behavior, such as access time, client IP address, request method, request URL, protocol version, user agent, cookies, referrer address, server address, response code, request byte count, and response byte count. However, log formats vary from server to server. Existing web log analysis relies primarily on manual parsing and analysis, resulting in significant workload and low efficiency, hindering intrusion prevention and response.

[0005] In view of this, there is an urgent need for an intrusion analysis method and system based on Web logs to solve the problem of low analysis efficiency caused by the non-uniform format of Web log types. Summary of the Invention

[0006] The technical problem to be solved by the present invention is that the web log types and formats provided by different types of existing web servers are not uniform, resulting in low efficiency in web log analysis.

[0007] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0008] A web log-based intrusion analysis method includes the following steps:

[0009] Split the original log into fields based on the default delimiter and default quote character.

[0010] Automatically identify field formats, quickly match field formats, parse and process them uniformly, and store them in the web log library;

[0011] Read the stored logs in batches and perform log matching rule detection; the log matching rule detection includes: determining whether the visitor IP of the web log is malicious based on the threat intelligence rules, adding a malicious tag to the web log if the visitor IP is malicious; determining the IP reputation of the visitor IP of the web log based on the IP reputation database; determining the access device of the visitor IP of the web log based on the keyword rules; determining the attack type of the web log based on the security detection rules; and detecting whether there is login behavior in the web log based on the login detection rules;

[0012] Update the logs that have completed the matching rule detection and write them back to the web log library;

[0013] Aggregate web logs with malicious tags based on the attacker's IP and account information in the web logs to generate web attack events. Then, determine whether the intruder's attack was successful based on the attack, login, and download behaviors of the IP address.

[0014] Based on the generated web attack time and analysis results, and the manually confirmed impact range of the intrusion, a protection and processing report is generated and submitted.

[0015] In the above solution, preferably, the quick matching of the field format further includes:

[0016] For the first log processed in the same batch, recommended fields are given based on the identified field format and sent to the administrator for manual confirmation.

[0017] In the above solution, preferably, the IP information in the IP reputation database includes IP source and IP threat type.

[0018] In the above solution, preferably, the IP source is a domestic home user, a domestic IDC, an external IDC or an external home user; and the IP threat type is that the IP has historically carried out attacks, including spam or web attacks.

[0019] In the above solution, preferably, the user's access device is crawler access, mobile device access or PC device access.

[0020] In the above solution, preferably, the attack type of the log is SQL injection attack, XSS, PHP injection or JAVA vulnerability injection.

[0021] In the above solution, preferably, the step of cutting the original log according to the default delimiter and the default quote character, and before cutting the original log into fields, further includes:

[0022] Set the default delimiter and default quote characters; the default delimiter characters are commas and spaces, and the default quote characters are double quotes, brackets, and single quotes.

[0023] A web log-based intrusion analysis and judgment system, comprising:

[0024] The parsing module is used to split the original log into fields based on the default delimiter and default quote character, automatically identify the field format, quickly match the field format, parse and process the fields uniformly, and store them in the database.

[0025] Log matching rule engine, used to perform rule matching judgment on the logs parsed and stored according to the rules;

[0026] The attack aggregation engine is used to aggregate web logs that have completed rule matching based on the attacker's IP address in the web logs, generate web log events, and determine whether the attack is successful;

[0027] Storage module, used to store logs and IP reputation database.

[0028] In the above solution, preferably, the log matching rule engine includes:

[0029] Threat intelligence rule judgment module is used to analyze and judge the logs stored in the database according to the threat intelligence rule judgment module.

[0030] The IP reputation determination module is used to determine the reputation of the visitor IP in the web log according to the IP reputation database;

[0031] A keyword rule determination module is used to determine the user's access device through the user agent field of the log according to the keyword rule;

[0032] The security detection rule module is used to determine the attack type of the web log based on the request details of the web log;

[0033] The login detection rule module is used to detect whether there is login behavior in the web log.

[0034] Compared with the prior art, the intrusion analysis and judgment method based on Web logs provided by the present invention cuts the logs by separators and quotes and automatically identifies the format of the cut fields to achieve unified parsing of the logs; performs threat intelligence rule judgment, IP reputation library judgment, keyword rule judgment, security detection rule judgment, and login rule judgment on the parsed logs to achieve rule matching detection of the logs; aggregates the logs that have undergone rule matching detection according to the attacker's IP and account information to generate web attack events, and achieves intrusion analysis and judgment of the attacker's IP. The intrusion analysis and judgment method based on Web logs provided by the present invention adopts a combination of automatic detection and manual confirmation, and has the advantages of high efficiency, high result accuracy, high log parsing strength, and convenience for secondary analysis, greatly improving the efficiency of web analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the structure of the intrusion analysis and judgment system based on Web logs in the present invention;

[0036] Figure 2 Schematic diagram of the principle of the intrusion analysis method based on Web logs in the present invention;

[0037] Figure 3 The flowchart of the intrusion analysis method based on Web logs in the present invention. DETAILED DESCRIPTION

[0038] The present invention provides a web log-based intrusion analysis method. By segmenting web logs, performing unified parsing, rule matching detection, and log aggregation, it enables intrusion analysis of attacker IP addresses, effectively solving the problems of low efficiency and high workload in manual web log analysis. The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] like Figure 1 As shown, the intrusion analysis and judgment system based on web logs provided by the present invention includes a parsing module 1, a rule matching engine 2, an aggregation engine 3 and a storage module 4. The parsing module 13 is used to cut the original log into fields according to the default delimiter and the default quote character, and automatically identify the format of the field, quickly match the format of the field, and parse and process it uniformly into the database. The log matching rule engine 2 is used to perform rule matching judgment on the parsed and stored logs according to the rules. The attack aggregation engine 3 is used to aggregate the web logs that have completed the rule matching judgment according to the attacker IP of the web log, generate web log events, and determine whether the attack is successful. The storage module 4 is used to store logs and IP reputation databases.

[0040] The log matching rule engine 2 includes a threat intelligence rule determination module 21, an IP reputation determination module 22, a keyword rule determination module 23, a security detection rule module 24, and a login detection rule module 25. The threat intelligence rule determination module 21 is used to analyze and store logs according to the threat intelligence rule determination module. The IP reputation determination module 22 is used to perform reputation determination on the visitor IP address of the web log based on the IP reputation database. The keyword rule determination module 23 is used to determine the user's access device based on the user agent field of the log according to the keyword rule. The security detection rule module 24 is used to determine the attack type of the web log based on the request details of the web log. The login detection rule module 25 is used to detect whether there is any login behavior in the web log.

[0041] like Figure 2 and Figure 3 As shown, the present invention provides an intrusion analysis method based on Web logs, comprising the following steps:

[0042] S1. Split the original log into fields based on the default delimiter and default quote character.

[0043] First, the administrator needs to set the default delimiter and default quote characters. The default delimiters are commas and spaces, and the default quote characters are double quotes, brackets, and single quotes.

[0044] Then, the parsing module 1 parses the original log using a CSV-like method based on the default delimiter and default quote character, and cuts the original log into fields.

[0045] S2. Automatically identify the field format, quickly match the field format, parse and process it uniformly, and store it in the web log library;

[0046] Parsing module 1 automatically identifies field formats, such as IP, time, request content, and response code.

[0047] For the first log processed in a batch, recommended fields are given based on the identified field format and sent to the administrator for manual confirmation. The recommended fields are shown in Table 1.

[0048] Table 1

[0049] Field Type Field Value Reference Properties IP Legal IP Client IP, Server IP positive integer [100-600) Response Code positive integer Less than 100, greater than 600 Response / Request Bytes time time Request time Decimals Decimals Request duration String URL Request URL, response address String Mozilla / 5.0* User Agent String GET, PUT, etc. Request content

[0050] The parsing module 1 sends the web logs after the parsing and unified processing to the web log library in the storage module 4 for storage.

[0051] S3. Read the stored logs in batches and perform log matching rule detection.

[0052] After the web logs are stored in the database, the log matching rule engine 2 reads the web logs in batches and performs matching rule detection on the web logs.

[0053] Specifically, matching rule detection includes threat intelligence rule determination, IP reputation determination, keyword rule determination, security detection rules, and login detection rules.

[0054] The threat intelligence rule determination module 21 determines whether the visitor IP of the web log is malicious according to the threat intelligence rule; after determining that the visitor IP of the web log is malicious, a malicious tag is added to the web log.

[0055] The IP reputation determination module 22 determines the IP reputation of the visitor IP in the web log according to the IP reputation database;

[0056] The IP reputation database is stored in the storage module 4. The IP information in the IP reputation database includes the IP source and IP threat type. The IP source can be a domestic home user, a domestic IDC, an international IDC, or an international home user. The IP threat type can be a history of attacks, spam, or web attacks.

[0057] The keyword rule determination module 23 uses the keyword rules to determine the access device of the visitor's IP address in the web log using the log's UserAgent field. Different devices have different userAgents. For example, Google Chrome and Firefox have different userAgents, and Android and iPhones have different userAgents. Some malicious software (such as crawlers) also has its own unique userAgent. The access device can be used to initially determine whether the request is malicious, thereby assisting manual analysis. The types of user access devices include crawler access, mobile device access, and PC device access.

[0058] The security detection rule module 24 detects the request details in the web log according to the security detection rules and determines the attack type in the web log. Request details are the detailed content of the request, including the request URL and request parameters. Malicious attack behavior can be determined based on the request parameters and request URL. The attack type in the log can be SQL injection, XSS, PHP injection, or Java vulnerability injection.

[0059] Login detection rule module 25 checks web logs for login activity based on login detection rules. Since logs represent login activity, and if the visitor's IP address in the web log has been previously attacked and has performed some privileged operations, the host may have been compromised, allowing the attacker to log into the system. Login activity can also be used to identify subsequent brute force attacks.

[0060] The above matching rule detection can be performed simultaneously or in a certain order.

[0061] S4. Update the log that completes the matching rule detection and write it back to the web log library;

[0062] S5. Aggregate web logs with malicious tags based on the attacker's IP and account information in the web logs to generate web attack events. Then, based on the attack, login, and download behaviors of the IP, determine whether the intruder's attack was successful.

[0063] Aggregation engine 3 aggregates the attacker's IP and account information in the web log to generate a web attack event, and determines whether the intruder's attack is successful based on the subsequent behavior of the IP, such as attack, login and download.

[0064] Aggregation of IP, time, and behavior involves analyzing logs from the same IP address over the same period of time. If the same visitor IP address exhibits both attack and login activity, as well as download activity, it can be determined that the attack was successful, the host was compromised, and files were stolen.

[0065] For different attack types, if the same visitor IP has a large number of login behaviors, first fails to log in, and then succeeds, the attack type is brute force cracking, the password is guessed by the attacker, and the host is compromised.

[0066] If the same visitor IP has a large number of access behaviors, with some accesses succeeding at first and then failing, then the attack type is a DDOS attack, and the attacker's attack is successful, affecting normal business.

[0067] S6. Based on the generated web attack time and analysis results, and the manually confirmed impact range of the intrusion, generate and submit a protection and processing report.

[0068] The aggregation engine 3 automatically generates a judgment result on whether the intrusion of the visitor IP is successful. The administrator manually confirms the scope of the intrusion based on the web attack time and judgment result generated by the aggregation engine 3, and performs protection and processing, and submits a protection and processing report.

[0069] Compared with the existing technology, the Web log-based intrusion analysis method and system provided by the present invention has the following advantages:

[0070] 1. Logs are segmented using delimiters and quotes, and the format of the segmented fields is automatically identified, achieving unified log parsing and unified processing of logs of different formats, greatly improving log processing efficiency. The processed logs have high parsing granularity, making it easier for administrators to conduct secondary analysis of web logs.

[0071] 2. Using a combination of automatic detection and manual confirmation, we automatically analyze and judge intrusions on web logs through rule matching detection and log aggregation. We also manually confirm the scope of intrusion impact, further confirming the analysis results and securing the evidence, greatly improving the accuracy of intrusion analysis.

[0072] 3. Based on multi-line record calculations, combined with the IP reputation database and subsequent attacks, logins, and downloads of the same visitor IP, it has a high accuracy rate.

[0073] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone should be aware that any structural changes made under the inspiration of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the scope of protection of the present invention.

Claims

1. A method for intrusion analysis based on Web logs, characterized in that: The following steps are involved: Split the original log into fields according to the default delimiter and default quote characters, where the default delimiter characters are commas and spaces, and the default quote characters are double quotes, brackets, and single quotes. Automatically identify field formats, quickly match field formats, parse and process them uniformly, and store them in the web log library; Read the stored logs in batches and perform log matching rule detection; The log matching rule detection includes: determining whether the visitor IP of the web log is malicious according to the threat intelligence rule, and adding a malicious tag to the web log if the visitor IP is malicious; determining the IP reputation of the visitor IP of the web log according to the IP reputation library; determining the access device of the visitor IP of the web log according to the keyword rule, including: determining whether the access device of the visitor IP of the web log has crawler access with malware through the userAgent field; determining the attack type of the web log according to the security detection rule; and detecting whether there is login behavior in the web log according to the login detection rule. Update the logs that have completed the matching rule detection and write them back to the web log library; Aggregate web logs with malicious tags based on the attacker's IP and account information in the web logs to generate web attack events. Then, determine whether the intruder's attack was successful based on the attack, login, and download behaviors of the same IP within the same time period. Based on the generated web attack time and analysis results, and according to the manually confirmed impact range of the intrusion, a protection and processing report is generated and submitted; the web attack time is generated by the aggregation engine when an attack occurs.

2. The intrusion analysis method based on Web logs according to claim 1 is characterized in that: The quick matching of the field format also includes: For the first log processed in the same batch, recommended fields are given based on the identified field format and sent to the administrator for manual confirmation.

3. The intrusion analysis method based on Web logs according to claim 1 is characterized in that: The IP information in the IP reputation database includes IP sources and IP threat types.

4. The intrusion analysis method based on Web logs according to claim 3 is characterized in that: The IP source is a domestic home user, a domestic IDC, an external IDC or an external home user; the IP threat type is that the IP has historically carried out attacks, including spam or web attacks.

5. The intrusion analysis method based on Web logs according to claim 1 is characterized in that: The access device is crawler access, mobile device access or PC device access.

6. The intrusion analysis method based on Web logs according to claim 1 is characterized in that: The attack type in the log is SQL injection attack, XSS, PHP injection or JAVA vulnerability injection.

7. An intrusion analysis and judgment system based on Web logs, characterized in that: include: The parsing module is used to cut the original log into fields according to the default delimiter and default quote characters, automatically identify the field format, quickly match the field format, parse and process the fields uniformly, and store them in the database. The default delimiters are commas and spaces, and the default quote characters are double quotes, brackets, and single quotes. Log matching rule engine, used to perform rule matching judgment on the logs parsed and stored according to the rules; The attack aggregation engine is used to aggregate web logs that have completed rule matching based on the attacker's IP address in the web logs, generate web log events, and determine whether the attack is successful; The storage module is used to store logs and IP reputation database. The IP reputation database is used to determine the IP reputation of visitor IPs in web logs. The log matching rule engine includes: Threat intelligence rule judgment module, used to analyze the stored logs according to the threat intelligence rule judgment module; The IP reputation determination module is used to determine the reputation of the visitor IP in the web log according to the IP reputation database; The keyword rule determination module is used to determine the user's access device through the user agent field of the log according to the keyword rule, including: determining whether the access device of the visitor IP in the web log has crawler access with malware through the userAgent field; The security detection rule module is used to determine the attack type of the web log based on the request details of the web log; The login detection rule module is used to detect whether there is login behavior in the web log.

Images