Method and system for transaction volume monitoring based on nginx log traffic analysis

Abstract

The application relates to a method and system for transaction volume monitoring based on nginx log flow analysis, belongs to the fields of internet business monitoring and software development and testing, and solves the problem of low speed and large time and energy consumption caused by the statistics and arrangement of business transaction volumes of various systems due to different system rules. The application arranges the transaction key nodes of key businesses in internet transactions, combs the uri_path corresponding to the key nodes in nginx historical logs, counts the transaction conditions of the source ip of the uri_path in the historical logs of nginx in a specific time period, records the statistical results in a database, analyzes the transaction volumes of various key businesses in the last one hour through nginx real-time log analysis, carries out deviation analysis on the transaction volumes in the last one hour and the average historical transaction volumes recorded in the database, counts the abnormal access source ip based on the obtained deviation value, and pushes the abnormal access source ip to a security device. The security device limits the source ip according to rules. The application is used for transaction volume monitoring.

Description

Technical Field

[0001] A method and system for monitoring transaction volume based on nginx log traffic analysis is presented, which falls under the fields of internet business monitoring and software development and testing. Background Technology

[0002] Currently, the monitoring of online business transaction volumes, such as mobile banking app user logins, personal consumption bandwidth credit application volume, and the number of Class II accounts opened, is mostly implemented at the application layer. This involves querying application database tables and application system logs, which consumes application resources. Querying the database consumes database memory and CPU resources, while querying application system logs consumes application server memory, CPU, and disk I / O resources, which can impact application system performance. Furthermore, the statistical data is delayed. If the application system itself malfunctions, the statistical data often fails to reflect the true transaction volume, potentially leading to production accidents.

[0003] Among existing business transaction volume monitoring solutions, the most common approach is to query the database to statistically analyze the transaction volume. This approach requires the application to record transaction data into the database, and the monitoring script or program to query the database to obtain transaction volume information. This approach has the following shortcomings:

[0004] 1) Due to the different rules of each system, it is necessary to collect and organize the business transaction volume for each system, which results in a lot of time and effort being spent and the speed being too low;

[0005] 2) Monitoring scripts and database query statements need to be written manually, and this part of the work has a low priority and is often overlooked;

[0006] 3) Database queries require the application to record transaction data into the database. If the database or the application malfunctions, the data cannot be stored in the database, resulting in inaccurate or non-existent transaction volume statistics in the business system. Summary of the Invention

[0007] To address the problems mentioned above, the present invention aims to provide a method and system for monitoring transaction volume based on nginx log traffic analysis. This solves the problem that existing technologies require statistical analysis and organization of business transaction volume for each system due to different system rules, resulting in significant time and effort expenditure and low speed.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A method for monitoring transaction volume based on nginx log traffic analysis includes the following steps:

[0010] Step 1: For critical internet services, identify the trancodes corresponding to key nodes;

[0011] Step 2: Combine the nginx historical logs to organize the uri_path in the nginx historical logs corresponding to the application system's interface code trancode. The nginx historical logs include logs from the past month. The uri_path is used to identify a resource on the Internet, indicating a transaction zone.

[0012] Step 3: Based on the uri_path in the nginx historical logs, calculate the hourly transaction volume of each source IP in the application system over the past month and record it in the database. The hourly transaction volume of each source IP refers to the number of times different source IPs access uri_path within each hourly time period.

[0013] Step 4: Collect nginx real-time logs, analyze the transaction volume of critical business access URI_path in the application system within one hour based on nginx real-time logs, perform deviation analysis with the average transaction volume of the corresponding time period in the past month, and judge normal source IPs and abnormal source IPs based on the results of deviation analysis, and compile a list of abnormal source IPs. The nginx real-time logs are the logs of one hour.

[0014] Step 5: The security device performs security verification on the list of abnormal source IPs. After verification, it records the normal source IPs as historical data and takes corresponding restrictive actions against the abnormal source IPs.

[0015] Furthermore, the key internet services in step 1 include user registration, electronic account opening, loan application, and loan withdrawal application.

[0016] Key nodes include obtaining SMS verification codes in user registration, bank card OCR recognition in electronic account opening, signing credit authorization letters in loan application, and facial recognition in loan withdrawal application.

[0017] Furthermore, the application systems in step 2 include mobile banking apps and credit systems.

[0018] Furthermore, the specific steps of step 2 are as follows:

[0019] Step 2.1: Obtain the application system's logs from the past month to get the Nginx historical logs;

[0020] Step 2.2: Based on the application system's interface code trancode, retrieve the application system's nginx historical logs to obtain the correspondence between the interface code trancode and the uri_path in the nginx historical logs.

[0021] Furthermore, the specific steps of step 3 are as follows:

[0022] Step 3.1: Obtain the application system's logs from the past month to get the Nginx historical logs;

[0023] Step 3.2: Analyze and organize the nginx historical logs through the unified log platform to calculate the transaction volume data of each URI_path in hours every day, that is, to count the number of times each source IP accesses the URI_path in an hourly time period.

[0024] Step 3.3: Record the transaction volume data into the database.

[0025] Furthermore, the specific steps of step 4 are as follows:

[0026] Step 4.1: Obtain the Nginx real-time logs for one hour. Analyze the Nginx real-time logs through a unified log platform to count the number of times each key business accesses the URI_path within one hour, that is, count the number of times different source IPs access the URI_path.

[0027] Step 4.2: Perform a deviation analysis between the number of visits counted in Step 4.1 and the average number of visits recorded in Step 3.2 for the corresponding time period. The average number of visits refers to the average transaction volume.

[0028] Step 4.3: If the results of the deviation analysis meet the given conditions, it indicates that the transaction volume is normal. The transaction volume appearing in the key business access uri_path is recorded in the database as historical data. Otherwise, it indicates that the transaction volume is abnormal. It is necessary to continue to compile a list of abnormal source IPs and proceed to step 5.

[0029] A system for monitoring transaction volume based on nginx log traffic analysis includes:

[0030] Analysis Unit: For critical internet businesses, analyze the trancode of the corresponding key nodes;

[0031] Historical data processing unit: Based on the nginx historical logs, process the uri_path in the nginx historical logs corresponding to the application system's interface code trancode. The nginx historical logs include logs from the past month. The uri_path is used to identify a resource on the Internet, indicating a transaction zone.

[0032] Historical statistics unit: Based on the uri_path in the nginx historical logs, the system counts the transaction volume of each source IP in the past month by hour and records it in the database. The transaction volume of each source IP in the past hour refers to the number of times different source IPs access uri_path in each time period of one hour.

[0033] Judgment Unit: Collects nginx real-time logs, analyzes the transaction volume of critical business access URI_path in the application system within one hour based on nginx real-time logs, performs deviation analysis with the average transaction volume of the corresponding time period in the past month, and judges normal source IPs and abnormal source IPs based on the results of deviation analysis, and compiles a list of abnormal source IPs. The nginx real-time logs are the logs of one hour.

[0034] Security device: Used to perform security verification on the list of abnormal source IPs. After verification, it records the normal source IPs as historical data and takes corresponding restrictive actions against the abnormal source IPs.

[0035] Furthermore, the key internet services in the sorting unit include user registration, electronic account opening, loan application, and loan withdrawal application.

[0036] Key nodes include obtaining SMS verification codes in user registration, bank card OCR recognition in electronic account opening, signing credit authorization letters in loan application, and facial recognition in loan withdrawal application.

[0037] Furthermore, the application systems in the historical data processing unit include mobile banking apps and credit systems;

[0038] The specific implementation steps of the historical organization unit are as follows:

[0039] Step 2.1: Obtain the application system's logs from the past month to get the Nginx historical logs;

[0040] Step 2.2: Based on the application system's interface code trancode, retrieve the application system's nginx historical logs to obtain the correspondence between the interface code trancode and the uri_path in the nginx historical logs.

[0041] Furthermore, the specific implementation steps of the historical statistics unit are as follows:

[0042] Step 3.1: Obtain the application system's logs from the past month to get the Nginx historical logs;

[0043] Step 3.2: Analyze and organize the nginx historical logs through the unified log platform to calculate the transaction volume data of each URI_path in hours every day, that is, to count the number of times each source IP accesses the URI_path in an hourly time period.

[0044] Step 3.3: Record the transaction volume data into the database;

[0045] The specific implementation steps of the judgment unit are as follows:

[0046] Step 4.1: Obtain the Nginx real-time logs for one hour. Analyze the Nginx real-time logs through a unified log platform to count the number of times each key business accesses the URI_path within one hour, that is, count the number of times different source IPs access the URI_path.

[0047] Step 4.2: Perform a deviation analysis between the number of visits counted in Step 4.1 and the average number of visits recorded in Step 3.2 for the corresponding time period. The average number of visits refers to the average transaction volume.

[0048] Step 4.3: If the results of the deviation analysis meet the given conditions, it indicates that the transaction volume is normal. The transaction volume appearing in the key business access uri_path is recorded in the database as historical data. Otherwise, it indicates that the transaction volume is abnormal. It is necessary to continue to compile a list of abnormal source IPs and proceed to step 5.

[0049] Compared with the prior art, the beneficial effects of this invention are as follows:

[0050] This invention greatly improves the timeliness and accuracy of transaction volume statistics in application systems, while adapting to different application systems. The statistical operation does not affect application performance, and the statistical results are unaffected by the application service status. Specifically:

[0051] 1) This invention avoids the need to spend a lot of time and effort on the statistics and organization of business transaction volumes for each system due to different system rules, and greatly improves the speed of statistics and organization.

[0052] 2) This invention eliminates the need to write monitoring scripts and database query statements, thus avoiding the problem of work priorities being ignored;

[0053] 3) This invention addresses the issue that data cannot be properly stored in the database or application due to database or application malfunctions, resulting in inaccurate or non-existent transaction volume statistics in the business system. Attached Figure Description

[0054] none Detailed Implementation

[0055] Step 1: For critical internet services, identify the trancodes corresponding to key nodes;

[0056] Key internet services include user registration, electronic account opening, loan application, and loan withdrawal application. Key nodes include obtaining SMS verification codes during user registration, bank card OCR recognition during electronic account opening, signing credit authorization letters during loan application, and facial recognition during loan withdrawal application. For example, the direct bank login interface code is trancode: userLogin.

[0057] Step 2: Based on the Nginx historical logs, organize the URI_path from the Nginx historical logs corresponding to the application system's trancode. The Nginx historical logs include logs from the past month. The URI_path is used to identify a resource on the internet, representing a transaction. Application systems include mobile banking apps and credit systems.

[0058] The specific steps are as follows:

[0059] Step 2.1: Obtain the application system's logs from the past month to get the Nginx historical logs;

[0060] Step 2.2: Based on the application system's interface code (trancode), retrieve the Nginx historical logs of the application system to obtain the correspondence between the interface code (trancode) and the URI_path in the Nginx historical logs. For example, find that the URI_path for the direct bank login interface in the Nginx historical logs is / portal / api / userLogin.do;

[0061] Step 3: Based on the uri_path in the nginx historical logs, calculate the hourly transaction volume of each source IP in the application system over the past month and record it in the database. The hourly transaction volume of each source IP refers to the number of times different source IPs access uri_path within each hourly time period.

[0062] The specific steps are as follows:

[0063] Step 3.1: Obtain the application system's logs from the past month to get the Nginx historical logs;

[0064] Step 3.2: Analyze and organize Nginx historical logs through a unified log platform to calculate the daily hourly transaction volume data for each URI_path. Specifically, calculate the number of times each source IP accesses the URI_path within a one-hour time period. Analyze and statistically analyze Nginx historical log statements such as: `index = zhixiao uri_path = " / portal / api / userLogin.do" group by date, hour, sre_ip last one month`.

[0065] Step 3.3: Record the transaction volume data into the database.

[0066] Step 4: Collect nginx real-time logs, analyze the transaction volume of critical business access URI_path in the application system within one hour based on nginx real-time logs, perform deviation analysis with the average transaction volume of the corresponding time period in the past month, and judge normal source IPs and abnormal source IPs based on the results of deviation analysis, and compile a list of abnormal source IPs. The nginx real-time logs are the logs of one hour.

[0067] The specific steps are as follows:

[0068] Step 4.1: Obtain the Nginx real-time logs for one hour. Analyze the Nginx real-time logs through a unified log platform to count the number of times each key business accesses the URI_path within one hour, that is, count the number of times different source IPs access the URI_path.

[0069] Step 4.2: Perform a deviation analysis between the number of visits counted in Step 4.1 and the average number of visits recorded in Step 3.2 for the corresponding time period. The average number of visits refers to the average transaction volume. For example, if the average transaction volume of the URI_path of the user registration and verification code node during the 9:00-10:00 time period over a month is x times, and the average transaction volume during the 9:00-10:00 time period on the current day is y times, calculate ((yx) / x)*100% to obtain the deviation value of this transaction for the current time period.

[0070] Step 4.3: If the deviation analysis results meet the given conditions, it indicates that the transaction volume is normal. Record the transaction volume appearing at the critical business access URI_path in the database as historical data. Otherwise, it indicates that the transaction volume is abnormal, and a list of abnormal source IPs needs to be compiled. Proceed to Step 5. If the obtained deviation value is greater than 50% or less than -50%, it indicates that the number of accesses during this time period for the current transaction is abnormal. The abnormal source IPs should be compiled. Otherwise, if the given conditions are met, record the transaction volume appearing at the critical business access URI_path in the database as historical data.

[0071] Analyzing and statistically analyzing nginx real-time logs using statements such as: index = zhixiao uri_path = " / portal / api / userLogin.do" date = currentdate() group by src-ip last one hour

[0072] Step 5: The security device performs security verification on the list of abnormal source IPs. After verification, it records the normal source IPs as historical data and takes corresponding restrictive actions against the abnormal source IPs.

[0073] In summary, this invention monitors current transaction volume based on historical transaction volume information. Through this analysis and alarm mechanism, the purpose is to monitor whether there are any abnormalities in internet transaction volume, for security protection and to prevent arbitrage activities by professional groups.

[0074] The above are merely representative embodiments among the many specific applications of this invention, and do not constitute any limitation on the scope of protection of this invention. All technical solutions formed by transformation or equivalent substitution fall within the scope of protection of this invention.

Claims

1. A method for monitoring transaction volume based on nginx log traffic analysis, characterized in that, Includes the following steps: Step 1: For critical internet services, identify the trancodes corresponding to key nodes; Step 2: Combine the nginx historical logs to organize the uri_path in the nginx historical logs corresponding to the application system's interface code trancode. The nginx historical logs include logs from the past month. The uri_path is used to identify a resource on the Internet, indicating a transaction zone. Step 3: Based on the uri_path in the nginx historical logs, calculate the hourly transaction volume of each source IP in the application system over the past month and record it in the database. The hourly transaction volume of each source IP refers to the number of times different source IPs access uri_path within each hourly time period. Step 4: Collect nginx real-time logs, analyze the transaction volume of critical business access URI_path in the application system within one hour based on nginx real-time logs, perform deviation analysis with the average transaction volume of the corresponding time period in the past month, and judge normal source IPs and abnormal source IPs based on the results of deviation analysis, and compile a list of abnormal source IPs. The nginx real-time logs are the logs of one hour. Step 5: The security device performs security verification on the list of abnormal source IPs. After verification, it records the normal source IPs as historical data and takes corresponding restrictive actions against the abnormal source IPs.

2. The method for monitoring transaction volume based on nginx log traffic analysis according to claim 1, characterized in that: The key internet services in step 1 include user registration, electronic account opening, loan application, and loan withdrawal application. Key nodes include obtaining SMS verification codes in user registration, bank card OCR recognition in electronic account opening, signing credit authorization letters in loan application, and facial recognition in loan withdrawal application.

3. The method for monitoring transaction volume based on nginx log traffic analysis according to claim 2, characterized in that: The application systems in step 2 include mobile banking apps and credit systems.

4. The method for monitoring transaction volume based on nginx log traffic analysis according to claim 3, characterized in that, The specific steps of step 2 are as follows: Step 2.1: Obtain the application system's logs from the past month to get the Nginx historical logs; Step 2.2: Based on the application system's interface code trancode, retrieve the application system's nginx historical logs to obtain the correspondence between the interface code trancode and the uri_path in the nginx historical logs.

5. The method for monitoring transaction volume based on nginx log traffic analysis according to claim 4, characterized in that step 3 specifically comprises: Step 3.1: Obtain the application system's logs from the past month to get the Nginx historical logs; Step 3.2: Analyze and organize the nginx historical logs through the unified log platform to calculate the transaction volume data of each URI_path in hours every day, that is, to count the number of times each source IP accesses the URI_path in an hourly time period. Step 3.3: Record the transaction volume data into the database.

6. The method for monitoring transaction volume based on nginx log traffic analysis according to claim 5, characterized in that, The specific steps of step 4 are as follows: Step 4.1: Obtain the Nginx real-time logs for one hour. Analyze the Nginx real-time logs through a unified log platform to count the number of times each key business accesses the URI_path within one hour, that is, count the number of times different source IPs access the URI_path. Step 4.2: Perform a deviation analysis between the number of visits counted in Step 4.1 and the average number of visits recorded in Step 3.2 for the corresponding time period. The average number of visits refers to the average transaction volume. Step 4.3: If the results of the deviation analysis meet the given conditions, it indicates that the transaction volume is normal. The transaction volume appearing in the key business access uri_path is recorded in the database as historical data. Otherwise, it indicates that the transaction volume is abnormal. It is necessary to continue to compile a list of abnormal source IPs and proceed to step 5.

7. A system for monitoring transaction volume based on nginx log traffic analysis, characterized in that, include: Analysis Unit: For critical internet businesses, analyze the trancode of the corresponding key nodes; Historical data processing unit: Based on the nginx historical logs, process the uri_path in the nginx historical logs corresponding to the application system's interface code trancode. The nginx historical logs include logs from the past month. The uri_path is used to identify a resource on the Internet, indicating a transaction zone. Historical statistics unit: Based on the uri_path in the nginx historical logs, the system counts the transaction volume of each source IP in the past month by hour and records it in the database. The transaction volume of each source IP in the past hour refers to the number of times different source IPs access uri_path in each time period of one hour. Judgment Unit: Collects nginx real-time logs, analyzes the transaction volume of critical business access URI_path in the application system within one hour based on nginx real-time logs, performs deviation analysis with the average transaction volume of the corresponding time period in the past month, and judges normal source IPs and abnormal source IPs based on the results of deviation analysis, and compiles a list of abnormal source IPs. The nginx real-time logs are the logs of one hour. Security device: Used to perform security verification on the list of abnormal source IPs. After verification, it records the normal source IPs as historical data and takes corresponding restrictive actions against the abnormal source IPs.

8. A system for monitoring transaction volume based on nginx log traffic analysis according to claim 7, characterized in that, The key internet services in the sorting unit include user registration, electronic account opening, loan application, and loan withdrawal application. Key nodes include obtaining SMS verification codes in user registration, bank card OCR recognition in electronic account opening, signing credit authorization letters in loan application, and facial recognition in loan withdrawal application.

9. A system for monitoring transaction volume based on nginx log traffic analysis according to claim 8, characterized in that, The application systems in the historical data processing unit include mobile banking apps and credit systems; The specific implementation steps of the historical organization unit are as follows: Step 2.1: Obtain the application system's logs from the past month to get the Nginx historical logs; Step 2.2: Based on the application system's interface code trancode, retrieve the application system's nginx historical logs to obtain the correspondence between the interface code trancode and the uri_path in the nginx historical logs.

10. A system for monitoring transaction volume based on nginx log traffic analysis according to claim 9, characterized in that, The specific implementation steps of the historical statistics unit are as follows: Step 3.1: Obtain the application system's logs from the past month to get the Nginx historical logs; Step 3.2: Analyze and organize the nginx historical logs through the unified log platform to calculate the transaction volume data of each URI_path in hours every day, that is, to count the number of times each source IP accesses the URI_path in an hourly time period. Step 3.3: Record the transaction volume data into the database; The specific implementation steps of the judgment unit are as follows: Step 4.1: Obtain the Nginx real-time logs for one hour. Analyze the Nginx real-time logs through a unified log platform to count the number of times each key business accesses the URI_path within one hour, that is, count the number of times different source IPs access the URI_path. Step 4.2: Perform a deviation analysis between the number of visits counted in Step 4.1 and the average number of visits recorded in Step 3.2 for the corresponding time period. The average number of visits refers to the average transaction volume. Step 4.3: If the results of the deviation analysis meet the given conditions, it indicates that the transaction volume is normal. The transaction volume appearing in the key business access uri_path is recorded in the database as historical data. Otherwise, it indicates that the transaction volume is abnormal. It is necessary to continue to compile a list of abnormal source IPs and proceed to step 5.