A method, device, and readable storage medium for Nginx log compression and analysis.

Abstract

This application discloses an Nginx log compression and analysis method, device, and readable storage medium. The method includes reading access log files under a single Nginx instance; determining compression segments based on access time and a preset compression duration; compressing the access log files within a single compression segment into a first compressed record, the first compressed record including at least one first field and a corresponding first attribute value; compressing access log files with the same access URL into a second compressed record, the second compressed record including at least one second field and a corresponding second attribute value; and sending the first and second compressed records to a database. This application addresses the needs of log analysis from project and function (URL) dimensions by compressing logs and storing them in an Elasticsearch index. It provides a series of metrics in three directions—performance, errors, and client—to reduce the impact of compression and highlight the value of logs.

Description

Technical Field

[0001] This invention belongs to the field of data processing, specifically relating to an Nginx log compression and analysis method, device, and readable storage medium. Background Technology

[0002] Nginx logs can comprehensively reflect changes in important indicators such as access volume, success rate, and performance of an internet service, which helps to ensure and continuously improve the quality of related service operations. The industry currently widely uses ELK to collect, store, and analyze this data.

[0003] Due to the massive volume of logs—some business operations generate billions of logs daily, requiring tens of terabytes or even more of storage—enterprises still need to periodically expand service capacity after investing in a large number of servers. Otherwise, log servers are prone to slow response or failure. Common solutions include storing only the most recent small amount of data and using different clusters for different business operations. Therefore, it is often impossible to fully grasp the value of all the data, hindering comprehensive analysis and comparison of different projects, and making it difficult to promptly identify and respond to some problems. How to efficiently process this data with limited IT resources in operation and maintenance work is one of the important challenges that urgently needs to be addressed in this field. Summary of the Invention

[0004] Based on this, the present invention aims to provide an Nginx log compression and analysis method, system, device and readable storage medium to compress logs from different business dimensions, explore the value of Nginx logs, and overcome the shortcomings of the prior art.

[0005] In a first aspect, the present invention provides an Nginx log compression and analysis method, comprising:

[0006] Read the access log file of a single Nginx instance;

[0007] The compression segment is determined based on the access time and the preset compression duration. The access log file in a single compression segment is compressed into a first compressed record. The first compressed record includes at least one first field and a corresponding first attribute value.

[0008] The access log files with the same access URL are compressed into a second compressed record. The second compressed record includes at least one second field and a corresponding second attribute value.

[0009] Send the first and second compressed records to the database.

[0010] Preferably, the above log compression and analysis method further includes:

[0011] Based on preset event types, the access log files corresponding to the same event are compressed into event compression records. The preset event types include at least one of the following: the log status code is 500 or above, the response time exceeds the set threshold, and the log request and response contain abnormal content.

[0012] Preferably, determining the compression segment based on the access time and preset compression duration includes:

[0013] Parse the access time of the log file according to the Nginx log format, let u = access time L / preset compression duration T, and round it down. The interval [u*T, (u+1)*T) is a compression segment. Log files whose access time is within [u*T, (u+1)*T) are compressed into the first compressed record.

[0014] Preferably, the process of generating the first compressed record includes:

[0015] Based on the first field, perform statistics and / or queries on the access log files within the compressed section to obtain the corresponding first attribute value.

[0016] and / or

[0017] Parse the log file, perform statistics and / or queries on the access log file in the compressed section based on the first field and log variables, and obtain the corresponding first attribute value;

[0018] At least one first field and its corresponding first attribute value constitute the first compressed record.

[0019] Preferably, the process of generating the second compressed record includes:

[0020] Based on the second field, perform statistics and / or queries on the access log files with the same access URL to obtain the corresponding second attribute value;

[0021] At least one second field and its corresponding second indicator value constitute the second compressed record.

[0022] Preferably, the first field includes at least performance values ​​TP90, TP95, and TP99. When the access volume p within a single compressed segment is less than 10000, the calculation of TP90 includes:

[0023] Generate a first performance data list, which includes the performance data corresponding to each log file;

[0024] Randomly select a time-consuming data point with index i, and recursively query the following sub-table to ensure that it is greater than any time-consuming data point between index [1, i-1] and less than any time-consuming data point between index [i+1, p]:

[0025] If i < 0.9p, recursively search for the 0.9p-th number in the sub-table [i+1, p-1]. If i > 0.9p, recursively search for the 0.9pi-th number in the sub-table [0, i-1]. When i-1 ≤ 0 or i+1 ≥ p-1, exit the recursion and return the first element of the recursive sub-table. The element determined by the recursive sub-table search process is TP90.

[0026] Sort the data after TP90 in the first performance data list in ascending order to obtain the second performance data list, which includes c elements.

[0027] Based on the proportional relationship between TP95, TP99, and TP90 in terms of serial number, TP95 and TP99 are determined in the second performance data list according to the serial number position of TP90. Specifically, TP95 is the [number missing]th [item missing] in the second performance data list. The data point, TP99, is the second data point in the second performance data list. Data, i dx This indicates the TP90's sequence number in the first performance data list.

[0028] Preferably, when a single project has at least two Nginx instances, the above method further includes:

[0029] Read the first and second compressed records of each Nginx instance;

[0030] Compress the first compressed record into the third compressed record;

[0031] Compress the second compressed record with the same access URL into a fourth compressed record;

[0032] Send the third and fourth compressed records to the database.

[0033] Preferably, the above method further includes adaptively recommending a preset compression duration T, including:

[0034] Step S1. Calculate the compression ratio r of a single Nginx instance based on the number of Nginx instances ns and the overall business compression ratio c, r = c / ns;

[0035] Step S2. Determine the minimum duration M of at least r log files;

[0036] Step S3. Determine the minimum duration M of at least r log files, in milliseconds;

[0037] Step S3. Calculate z = M * N / (24 * 3600 * 1000), where N represents the total number of time periods with no less than r log records in all durations M on that day. If z exceeds the set value, T = M, and T is uploaded and saved; otherwise, proceed to step S4.

[0038] Step S4. Set M = 1.1M and round up, then repeat step S3.

[0039] Secondly, the present invention provides an Nginx log compression and analysis system, including a server and a compression terminal. The server stores Nginx instances to be compressed and analyzed, and the compression terminal is used to implement the relevant steps in the Nginx log compression and analysis method provided in the first aspect.

[0040] Thirdly, the present invention provides an Nginx log compression and analysis device, including a memory and a processor;

[0041] The memory is used to store programs;

[0042] The processor is used to execute the program to implement the various steps of the Nginx log compression and analysis method provided in the first aspect.

[0043] Fourthly, the present invention also provides a readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the various steps of the Nginx log compression and analysis method provided in the first aspect.

[0044] As can be seen from the above technical solutions, the Nginx log compression and analysis method provided in this embodiment of the invention has the following beneficial effects:

[0045] 1. This feature compresses and extracts value from Nginx logs across three business dimensions: project, function, and critical events. Beyond standard analysis of access volume, traffic, performance, and error trends, the compressed results provide a series of specialized metrics focusing on client-side, error, and performance aspects to deeply and intuitively demonstrate the value and detailed distribution of Nginx logs. It enables simple and rapid analysis of Nginx logs from more projects on an Elasticsearch cluster with the same configuration, providing more analytical results without missing important information or causing data bloat. This helps standardize Nginx log analysis and fully uncover more value.

[0046] 2. Supports custom compression ratio, automatic recommendation of log compression duration, and automatic discovery and management of Nginx instance information, which helps to automatically identify Nginx configuration problems and facilitates automatic parsing of Nginx logs;

[0047] 3. The architecture consisting of a custom compression module, Elasticsearch (ES), and Kibana is easier to develop, maintain, and expand than a Hadoop-based big data architecture. It also offers higher cluster analysis and query performance with the same configuration and allows for flexible adjustments to these three components using Logstash, ClickHouse, and Grafana. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 This is a flowchart of the Nginx log compression and analysis method provided in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the Nginx log compression and analysis system provided in an embodiment of the present invention;

[0051] Figure 3 This is a business entity design architecture diagram for Nginx log compression and analysis provided in an embodiment of the present invention;

[0052] Figure 4 This is a structural block diagram of the Nginx log compression and analysis device provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] The reason for wasted storage in Nginx access logs is the large amount of duplicate data. The basis for compression can be determined by analyzing the repetition of values ​​in each field of each log entry. These fields mainly include: remote_addr and port, access time and timezone (time_local), request URI (URL for HTTP / HTTPS protocols), server_protocol, request_method, status (indicating whether an error occurred and its type), upstream_addr, URL redirection source, http_user_agent, request and response duration and bytes, SSL protocol, request parameters, and server hostname (server_name). Among these, server_name, server_protocol, and SSL protocol values ​​are fixed; URI, request_method, upstream_addr, URL redirection source, and status have a fixed set of optional values; http_user_agent shows a large number of duplicates after removing the version number; and remote_addr also has a series of identical values. The only truly variable and meaningful fields are time_local, client port, request and response duration and bytes, and request parameters.

[0055] The value of Nginx logs lies primarily in trends such as access volume (log volume), performance, errors, and traffic, as well as some URLs. Logs with low access volume, normal performance, and no errors do not require detailed review; users typically only examine them in detail when trends are about to worsen, hoping to quickly identify the problem. While alarm and fault event logs account for a small percentage, detailed logs are indeed necessary when problems occur. Therefore, this invention improves the existing log compression process from three dimensions: project, function, and important events, through architectural design and compression algorithms. From the Nginx instance to the overall project and its key functions, the original logs are compressed in two stages at an appropriate time granularity to form trend data. That is, all log records within a certain time period (aggregated time period, such as 1 second, 5 seconds, etc.) are compressed into one record, for example, at least 100 logs are compressed into one result. Then, logs for deduplication, errors, and important events such as chronic performance issues are compressed. This allows an Elasticsearch cluster to store and analyze massive amounts of logs from multiple businesses, enabling users to analyze data over a larger time range while retaining details of the main events and URLs before compression, making the original logs negligible.

[0056] See Figure 1 This embodiment provides an Nginx log compression and analysis method, including the following:

[0057] Read the access log file of a single Nginx instance;

[0058] The compression segment is determined based on the access time and the preset compression duration. The access log file in a single compression segment is compressed into a first compressed record. The first compressed record includes at least one first field and a corresponding first attribute value.

[0059] The access log files with the same access URL are compressed into a second compressed record. The second compressed record includes at least one second field and a corresponding second attribute value.

[0060] Send the first and second compressed records to the database.

[0061] The first compression record reflects the compression process at the project level. It can store the compressed results of all access logs of an Nginx instance within a certain time range, such as compression start time, compression duration, access volume, average and maximum performance, the time and URL of the maximum performance occurrence, request performance metrics including percentile performance TP90, TP95, TP99 (other percentile performance can be added as needed, such as TP99.9), access volume for each performance range, and input and output traffic, as well as backend errors in the format "status code 1 - error volume | status code 2 - error volume...". The distribution of client errors, categorized by common backend error status codes, includes possible error types and their quantities. It also includes the distribution of access problems (Nginx configuration errors, security risk issues, etc.) determined based on log content and request parameters, saved in the format "Type 1 code - number of problems | Type 2 code - number of problems...".

[0062] The second compression record reflects the compression processing at the functional level. It can save the compression results of user-specified project functions (HTTP / HTTPS protocol projects are URLs, hereinafter referred to as URLs). In addition to the fields that are the same as the first compression record, it also includes 4XX and 5XX error quantities and error distribution, output, and output traffic in the format of "status code 1 - error quantity | status code 2 - error quantity...". It can save the standard deviation of response performance for more specific characteristics of the technical solution of the function than the project, which can help judge the stability of the implementation effect. It can also expand more fields according to the needs of functional log analysis.

[0063] Preferably, the above log compression and analysis method further includes:

[0064] Based on preset event types, the access log files corresponding to the same event are compressed into event compression records. The preset event types include at least one of the following: the log status code is 500 or above, the response time exceeds the set threshold, and the log request and response contain abnormal content.

[0065] Event compression logs primarily preserve the main content of logs related to backend errors, performance limits, and security risks. This includes the URLs related to the issue, the project, the time of occurrence, the event type, the protocol, the agent, and the addresses of the client and backend services in the original logs.

[0066] In a further embodiment, determining the compression segment based on the access time and a preset compression duration includes:

[0067] Parse the access time of the log file according to the Nginx log format, let u = access time L / preset compression duration T, and round it down. The interval [u*T, (u+1)*T) is a compression segment. Log files whose access time is within [u*T, (u+1)*T) are compressed into the first compressed record.

[0068] In a further embodiment, the process of generating the first compressed record includes:

[0069] Based on the first field, perform statistics and / or queries on the access log files within the compressed section to obtain the corresponding first attribute value.

[0070] and / or

[0071] Parse the log file, perform statistics and / or queries on the access log file in the compressed section based on the first field and log variables, and obtain the corresponding first attribute value;

[0072] At least one first field and its corresponding first attribute value constitute the first compressed record.

[0073] The first field that needs to be statistically analyzed mainly includes access volume, input traffic, output traffic, the number of requests within each performance statistical interval configured, the maximum performance and its occurrence time and corresponding URL, and the number of logs for different log status codes.

[0074] After parsing the log file, the statistics based on the first field and log variables mainly include the access volume, error log volume, and error distribution for each client type.

[0075] In a further embodiment, the process of generating the second compressed record includes:

[0076] Based on the second field, perform statistics and / or queries on the access log files with the same access URL to obtain the corresponding second attribute value.

[0077] At least one second field and its corresponding second indicator value constitute the second compressed record.

[0078] In a further embodiment, the first field includes at least performance values ​​TP90, TP95, and TP99. When the access volume p within a single compressed segment is less than 10000, the calculation of TP90 includes:

[0079] Generate a first performance data list, which includes the performance data corresponding to each log file;

[0080] Randomly select a time-consuming data point with index i, and recursively query the following sub-table to ensure that it is greater than any time-consuming data point between index [1, i-1] and less than any time-consuming data point between index [i+1, p]:

[0081] If i < 0.9p, recursively search for the 0.9p-th number in the sub-table [i+1, p-1]. If i > 0.9p, recursively search for the 0.9pi-th number in the sub-table [0, i-1]. When i-1 ≤ 0 or i+1 ≥ p-1, exit the recursion and return the first element of the recursive sub-table. The element determined by the recursive sub-table search process is TP90.

[0082] Sort the data after TP90 in the first performance data list in ascending order to obtain the second performance data list, which includes c elements.

[0083] Based on the proportional relationship between TP95, TP99, and TP90 in terms of serial number, TP95 and TP99 are determined in the second performance data list according to the serial number position of TP90. Specifically, TP95 is the [number missing]th [item missing] in the second performance data list. The data point, TP99, is the second data point in the second performance data list. Data, i dx This indicates the TP90's sequence number in the first performance data list.

[0084] In a further embodiment, when a single project has at least two Nginx instances, the above method further includes:

[0085] Read the first and second compressed records of each Nginx instance;

[0086] Compress the first compressed record into the third compressed record;

[0087] Compress the second compressed record with the same access URL into a fourth compressed record;

[0088] Send the third and fourth compressed records to the database.

[0089] In a further embodiment, the method also includes adaptively recommending a preset compression duration T, comprising the following steps:

[0090] Step S1. Calculate the compression ratio r of a single Nginx instance according to the number of Nginx instances ns and the overall business compression ratio c, r = c / ns;

[0091] Step S2. Determine the shortest duration M of no less than r log files, in milliseconds;

[0092] Step S3. Calculate z = M*N / (24*3600*1000), where N represents the total number of time periods when the number of log records in all durations M of the day is no less than r. When z exceeds the set value, T = M, and upload and save T, otherwise go to Step S4;

[0093] Step S4. Let M = 1.1M and round up, and repeat Step S3.

[0094] Refer to Figure 2 and Figure 3 , the following embodiments further describe the compression analysis method provided by the present invention through a specific entity design.

[0095] Figure 2 Shows one of the Nginx log compression analysis systems, which consists of two parts: a client and a server. The server is responsible for the discovery and management of Nginx instance information in units of projects, the analysis and display of compression results. The client is suitable for development using go or Rust, and the compression results are stored in ES. The analysis effect sharing functions of Kibana and Grafana can be used to make up for their deficiencies. Based on the compression results, some custom analysis functions can be further developed and integrated with the analysis of the two into a UI to achieve the linkage between relevant business entity data.

[0096] The automatic discovery of Nginx instance information helps users quickly discover possible inconsistencies in the Nginx configurations of the same project, facilitates the parsing and processing of Nginx access logs, and timely processes the discovered problems. It mainly includes the following steps:

[0097] a) If there is a running Nginx process on the current server, execute the corresponding Nginx with the specified option -t and then parse the output to obtain the configuration file it is using, and extract configurations such as log_format, backend service instances, and weights from this file;

[0098] b) If there is no running Nginx process, first parse the operating system history to obtain the command for operating Nginx. If it cannot be found, exit. If the command is in the default location, use the configuration file in the default location of Nginx, otherwise try to parse the most recently used configuration file from the system history Nginx startup command or startup script;

[0099] c) After finding the configuration file, parse log_format, backend service instance and weight configuration in the same way as the previous step, then parse the server IP and report it to the server. If the configuration file cannot be found, exit and prompt the user to enter the location of the Nginx configuration file.

[0100] d) After receiving the reported Nginx instance records, the server parses all backend service instance addresses, the access volume weight that may be assigned to each backend instance, log_format, and Nginx instance IP, and then saves the records to the Nginx instance table.

[0101] e) The server periodically checks the saved Nginx instance data for different records with the same or non-empty intersection of backend service instance sets. If a record is found, the corresponding Nginx instances are considered to belong to the same project. Then, a project number is automatically generated for these records. This number can be modified by the user to a straightforward value, as long as it remains unique in the table. Next, a new project record is generated in the project table using the obtained project number and the record count as the Nginx instance count. The user can then supplement more information through relevant management functions.

[0102] f) The server can further check for differences in the log_format and other configurations of various Nginx instances within the same project and notify the user to confirm the differences.

[0103] Figure 3 This embodiment illustrates the entity design provided. Under this entity architecture, Nginx instance information can be managed using a relational database. The database stores information on how clients automatically discover and manage Nginx instances, facilitating the parsing of Nginx access logs. It includes an Nginx instance table: storing the IP address of each Nginx instance, its log_format configuration, the set of backend service addresses, and possible load balancing ratios. More Nginx instance information can be stored as needed. The system can further identify all Nginx instance records belonging to the same project based on the Nginx instance information reported by the client and automatically generate and save the project number and the number of Nginx instances in the project table. Users can configure more attributes such as the product to which the project belongs, the project type used for analysis, and the project's chronic performance threshold through management functions. The project URL table stores which URLs (project functions) the user needs to analyze and their configurations. It can also save data for analyzing URL-based aggregation results as needed. The system provides relevant management functions to support users in adjusting the data in these tables as required.

[0104] Log compression is mainly divided into compression of access logs for a single Nginx instance and compression of all Nginx instances within the project.

[0105] 1. Compression processing of access logs for a single Nginx instance.

[0106] The system resolves the IP address of the server hosting the current Nginx instance. Using this IP, it queries the server for the Nginx log format configuration (log_format), the project number, the total number of Nginx instances in the project (nc), the aggregation duration (T), and the performance segmentation interval configuration (e.g., [0,0.2], [0.2,1], [1,3]). If not specified, the default value is used. The system reads and sorts the filenames of all files in the specified log directory. It checks if a previously saved processing history file exists in the specified directory. The file content includes the successfully reported filenames and their last processed line numbers. If it exists, it loads and compares the file to determine if it is the currently being processed file. If so, it jumps to the next line after the last processed line number and continues processing each line of log data.

[0107] The specific compression process is as follows:

[0108] S101. Based on buffered concurrent reading of the log file, parse the record corresponding to each line according to log_format, convert the time_local field in the log file to millisecond form L, let u = L / T and round it, and compress all logs with time_local belonging to the interval [u*T, (u+1)*T) into a single record with u*T as the starting time.

[0109] S102. For all log records with status codes of 500 or higher, response performance exceeding a certain threshold, or requests and responses containing abnormal content, assemble them into corresponding records based on a one-to-one index of important events and submit them to the server for storage. This can be extended to support the detection and analysis of more types of events. Within the compression time period containing these events, a smaller compression cycle can be used, supporting the display of data changes at a finer time granularity.

[0110] S103. Perform the following compression process on the single compressed segment obtained in step S101 to obtain the compressed record in the item dimension of the compressed segment.

[0111] The cumulative number of visits (p), input and output traffic, the number of requests within each performance statistics interval, the maximum performance and its occurrence time, and the corresponding URL are configured.

[0112] For logs with status codes 400 or higher, accumulate the error count corresponding to each status code.

[0113] Parse the access log files and obtain the client type (browser or access library) corresponding to each log file based on the http_user_agent field. Accumulate the access volume, the number of 4XX status code errors, and the error distribution for each client type.

[0114] S104. For all log files to be compressed, classify them by project number, URL, and start time, accumulate the access volume (url_pv) of log files with the same URL, the access volume of logs with status codes 4XX and 5XX, input and output traffic, error distribution, and record a list of all response performance data for the current time period to obtain the compression record in the functional dimension of the compression section.

[0115] The calculation of the performance fields TP90, TP95, and TP99 in the compressed record follows the procedure below.

[0116] When the number of visits p < 10000, the performance field is calculated as follows:

[0117] Find the smallest percentile performance among the several percentile performances to be calculated. Suppose we want to calculate TP90, TP95 and TP99, and the smallest is TP90.

[0118] Add all the performance data to be calculated to a list L. Randomly select a data point from the list as the sorting criterion, such that after sorting, all data to the left of this data point are smaller than it, and all data to the right of this data point are larger than it. The index of this data point in the list is i.

[0119] If i < 0.9p, recursively search for the 0.9p-th number in the sub-table [i+1, p-1]. If i > 0.9p, recursively search for the 0.9pi-th number in the sub-table [0, i-1]. When i-1 ≤ 0 or i+1 ≥ p-1, exit the recursion and return the first element of the recursive sub-table. The element determined by the recursive sub-table search process is TP90.

[0120] Sort the data in list L after TP90 in ascending order to obtain list L1. Calculate the ratio of TP95 and TP99 to TP90. Specifically, TP95 is the number of rows in list L1 that are in the correct order. The data point, TP99, is the first data point in list L1. Data, i dx This indicates the index of TP90 in list L.

[0121] When the number of visits p > 10000, it means that there are still many logs to be processed. At this time, a performance data summary object is created first, which is compressed by the default compression ratio c = 100, but can be adjusted. Then, the request performance data of all logs in the current time period is continuously added to the summary object with the default weight of 1. After that, the summary object can be called to approximate the performance percentile value.

[0122] The creation of the summary object includes:

[0123] Add the performance data d of each log file to the summary object. First, accumulate the weight of d to the total weight of the summary object, and then find the cluster whose average value of the added data clusters is closest to d. For each cluster, set ptil = (weight of the current cluster / 2 + total weight of all previous clusters) / total weight, and the upper weight limit wl of the cluster = 4 * c * ptil * (1 - ptil) * total number of clusters. If not found, generate a new cluster according to the data to be added; if a cluster is found and its weight < wl, it means there is still space in this cluster, return the serial number of this cluster, otherwise also generate a new cluster.

[0124] If multiple clusters are found, there are four cases: if the average values of all clusters are less than d, find the cluster with the largest number and return the serial number of this cluster when there is space in this cluster; if the average values of all clusters are greater than d, find the cluster with the smallest number and return the serial number of this cluster when there is space in this cluster; if the average values of all clusters are equal to d, randomly find a cluster with space and then return its serial number; if only the average values of some clusters are greater than d, find two clusters whose average values include d, and randomly return the serial number of a cluster with space; if there is no space, generate a new cluster. Each time a new cluster is generated, determine the position serial number of the new cluster according to the average value of the elements in the cluster, so that the average values of all clusters are in ascending order. If no new cluster needs to be added, add d to the found cluster. At this time, if the sum of the weights of the cluster after adding d does not exceed wl, directly accumulate the weight of d to the weight of the cluster, and the average value of the cluster = original average value + weight of d * (d - original average value) / new weight of the cluster, otherwise fill the space of the cluster and calculate the average value of the cluster after adding elements using the remaining space:

[0125] After creating the performance data summary object through the above process, use this summary object to calculate the percentile value of performance. For example, when calculating TP90, set tp = 0.9. First, calculate q = tp * total weight, and then start to find the i-th cluster, so that the sum of the weights of the first i - 1 clusters + half of the weight of the i-th cluster > q, but the sum of all the first i - 2 clusters does not meet this condition. Next, calculate the slope s of these two clusters = (average value of the i-th cluster - average value of the previous cluster) / sum of half of their weight values, and finally take the average value of the i-th cluster + s * (q - (weight of the i-th cluster / 2 + sum of the weights of all previous clusters)) as the required percentile value.

[0126] When compressing the URL log, similar to executing the above step S104, if nc = 1, when url_pv <10000, directly calculate the percentile value, standard deviation, average, and maximum performance of each performance according to the response performance list, url_pv >When url_pv = 10000, estimate the performance of each percentile according to the above summary object; if nc > 1, when url_pv >= 10000 / nc, calculate the summary object of the response performance of each Nginx instance according to step S104 above, and then combine the fields accumulated in step S103 and send them to Kafka; when url_pv < 10000 / nc, directly send the accumulated metrics in step d) to Kafka. When it is necessary to calculate the standard deviation, send url_pv, the sum of the performance data of the current instance, and the IP of Nginx to Kafka, accumulate the data of each instance of the project, calculate the average value, and then send the average value and the total performance data volume back to the client here through the queue. Then calculate the variance of each instance, accumulate them, and take the square root to obtain the standard deviation.

[0127] S105. Update the compression processing history, recording the current file and the line numbers of completed processes.

[0128] S106. Continue processing all logs in the next compressed section according to steps S101 to S105 until all logs have been processed;

[0129] The real-time generated Nginx access logs can also be compressed directly in memory before being sent to the Kafka cluster. The main aggregation logic is the same as the compression process described above: First, the Nginx module `ngx_http_log_module` needs to be modified, then Nginx needs to be recompiled, the updated module added, and the actual Nginx instance updated to take effect. The real-time log aggregation function can be enabled or disabled through Nginx configuration. This approach avoids unnecessary I / O during log aggregation and offers better real-time performance.

[0130] 2. Compression process for projects with multiple Nginx instances.

[0131] When a project has multiple Nginx instances, the following section involves further processing the compression results of all instances to generate the final record and save it. This part can be controlled as part of the client through configuration files, resulting in a different operating mode than described above. Alternatively, the client can send the input data here to the server via protocols such as RPC instead of Kafka, and then compress and store it according to the following steps. However, such variations should still be within the scope of this patent. The execution of this process includes the following steps:

[0132] S201. Read the current project's aggregation duration T, the list of URLs to be analyzed, and the total number of Nginx instances nc from the server interface.

[0133] S202. Read the compression records of a single Nginx instance from the Kafka queue. Accumulate the access volume, performance distribution, 5XX and 4XX error volume, error distribution, input and output traffic of each URL according to the compression duration of the project. Calculate the average performance and maximum performance of each URL and calculate the standard deviation. Calculate the percentile performance of each URL according to the compression process of the single Nginx instance's functional dimensions. Then, assemble the fields into a record according to the composition design of the functional log compression history and submit it to the server for storage.

[0134] S203. Read data from the Kafka queue that temporarily stores the compression results of Nginx instances, accumulate the values ​​of various metrics for each Nginx instance compression record with the same project number, including access volume, performance distribution, error volume and distribution of each error type, access problem distribution, input and output traffic, merge all records with the same start time into one record, compare the maximum performance of each Nginx instance, and use the maximum value, related URL and occurrence time as the maximum performance, occurrence time and related URL of the project in the corresponding compression section;

[0135] For performance percentile calculation in multi-Nginx instance projects: When nc > 1, if the merged access volume p >= 10000, read the Kafka queue of the data digest objects and merge all digest objects. During merging, first randomly shuffle the clusters of one digest object, then add each cluster to another digest object one by one. When adding each cluster (let its weight be c), gradually increase the weight to a certain number of times (adjustable) to find the maximum allowed weight a without exceeding the limit. If a > c, finally add the cluster with weight c; otherwise, add it with weight ca to prevent overload after addition. After all the digest objects corresponding to each Nginx instance in a compression period have been merged, calculate the performance percentile values ​​using the merged digest objects according to the compression process disclosure of a single Nginx instance. If p < 10000, read and parse the Kafka queue for calculating the full performance and error volume, and accurately calculate the performance data for each percentile.

[0136] S204. Based on the composition design of the project log compression history, assemble each field and submit it to the server for storage.

[0137] Furthermore, this embodiment also supports adaptive recommendation of Nginx instance compression duration, specifically through the following steps:

[0138] Step S301. Calculate the compression ratio r of a single Nginx instance based on the number of Nginx instances ns and the overall business compression ratio c, r = c / ns;

[0139] Step S302. Determine the minimum duration M of at least r log files, in milliseconds;

[0140] Step S3. Calculate z = M * N / (24 * 3600 * 1000), where N represents the total number of time periods with no less than r log records in all durations M on that day. If z exceeds the set value, T = M, and T is uploaded and saved; otherwise, proceed to step S4.

[0141] Step S304. Set M = 1.1M and round up, then repeat step S303.

[0142] Because the number of records after compression can be significantly reduced compared to the original logs (by default, 100 original logs are compressed into one, which can be adjusted), an ES cluster can store the compressed results of Nginx logs from many projects. At the same time, it can comprehensively analyze and compare all indicators based on compressed historical data from three dimensions: project and its products, project functions, and important events. The query and analysis performance is good, while reducing storage usage. Based on the targeted compression result composition scheme, it is also unnecessary to view the original logs.

[0143] Kibana / Grafana allows for convenient and rapid analysis of trends in metrics such as access volume, traffic, average performance, and maximum performance based on compressed history. This meets various analytical needs for raw Nginx logs in practical work and provides further analysis capabilities, including but not limited to: ① Organizing a series of specific analytical charts into a dashboard to asynchronously obtain trends in access volume, performance, and / or error metrics for projects and functions over time, displaying details of important events and maximum performance; ② Comparing error volume, error rate, chronic energy, and chronic energy ratio of different functions in different projects or technical solutions using tabular leaderboards to identify key projects and functions with optimization potential; ③ Calculating and merging trends in access volume and 4XX error volume for various error types and client types using the Painless script; ④ Comparing metrics across different projects and trends of the same project over different time periods based on Kibana's timelion; ⑤ Utilizing ELK's watcher mechanism to trigger real-time alerts via email or other channels when any metric or trend deteriorates beyond a certain limit; ⑥ Analyzing detailed data on important events such as errors, performance, and security, as well as specific changes in access volume, error volume, traffic, and performance metrics of related systems around the time of the event.

[0144] The access logs of Kubernetes ingress and Apache HTTP Server can also be compressed using this method. Compared to directly using ELK to process the raw logs, both methods can achieve similar results by processing more logs with less ES storage space and providing more and more intuitive analysis.

[0145] This embodiment is designed to showcase the value of Nginx logs from three dimensions: project, function, and important events. It avoids record bloat caused by differences in fields of the compression result. It can also conveniently obtain the data of the three parts in parallel on the UI through statistical graphs, tables, metrics, etc., and then assemble them into a timed refresh display interface to show the changes of key indicators, related impacts, and important analysis results, and support viewing related details. The time granularity during analysis can be selected as needed, any granularity greater than the compression duration.

[0146] This invention also provides an Nginx log compression and analysis system, including a server and a compression terminal. The server stores Nginx instances to be compressed and analyzed, and the compression terminal is used to implement the relevant steps in the Nginx log compression and analysis methods provided in the foregoing embodiments.

[0147] like Figure 4 As shown, this embodiment of the invention provides an Nginx log compression and analysis device, including: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0148] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0149] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0150] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0151] The memory stores a program, which the processor can call. The program is used to implement the various steps of the Nginx log compression and analysis method described in the foregoing embodiments.

[0152] This invention also provides a readable storage medium storing a computer program thereon, which, when executed by a processor, implements the various steps of the Nginx log compression and analysis method provided in the foregoing embodiments.

[0153] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for Nginx log compression analysis, characterized in that, The method comprises: reading access log files under a single Nginx instance; determining a compression section according to access time and a preset compression duration, and compressing the access log files in the single compression section into a first compressed record, the first compressed record comprising at least one first field and a corresponding first attribute value; compressing the access log files of the same access URL into a second compressed record, the second compressed record comprising at least one second field and a corresponding second attribute value; sending the first compressed record and the second compressed record to a database; the first field at least comprises performance values TP90, TP95 and TP99, and when the access volume p in the single compression section is less than 10000, the calculation of TP90 comprises: generating a first performance data list, the list comprising performance data corresponding to each log file; randomly selecting a time-consuming data with a serial number i, and making it greater than any time-consuming data between serial numbers [1, i-1] and less than any time-consuming data between serial numbers [i+1, p] through the following recursive query process of a sub-list: if i < 0.9p, recursively finding the 0.9pth number in the sub-list [i+1, p-1], if i > 0.9p, recursively finding the 0.9p-i th number in the sub-list [0, i-1], and when i-1≤0 or i+1≥p-1, exiting the recursion and returning the first element of the recursive sub-list, and the element determined by the recursive sub-list query process is TP90; sorting the data after TP90 in the first performance data list in ascending order to obtain a second performance data list comprising c elements. The proportional relationship of TP95, TP99 and TP90 in the sequence number is used to determine TP95 and TP99 according to the sequence number position of TP90 in the second performance data list. Specifically, TP95 is the first data in the second performance data list, and TP99 is the second data in the second performance data list. TP90 represents the sequence number of TP90 in the first performance data list.​​ 2. The Nginx log compression analysis method of claim 1, wherein, The method further comprises: compressing the access log files corresponding to the same event into an event compressed record according to a preset event type, the preset event type at least comprising at least one of the following: a log status code being 500 or above, a response time-consuming exceeding a set threshold, and a log request and response having abnormal content.

3. The Nginx log compression analysis method of claim 1, wherein, The determination of the compression section according to the access time and the preset compression duration comprises: parsing the access time of the log file according to the Nginx log format, letting u = access time L / preset compression duration T, and taking the integer, and the interval [u*T, (u+1)*T) being a compression section, and the log files with the access time belonging to the interval [u*T, (u+1)*T) being compressed into the first compressed record.

4. The Nginx log compression analysis method of claim 1, wherein, The generation process of the first compressed record comprises: statistically and / or inquiring the access log files in the compression section according to the first field to obtain the corresponding first attribute value, and / or parsing the log file, and statistically and / or inquiring the access log files in the compression section according to the first field and log variables to obtain the corresponding first attribute value; at least one first field and the corresponding first attribute value constitute the first compressed record.

5. The Nginx log compression analysis method of claim 1, wherein, The generation process of the second compressed record comprises: statistically and / or inquiring the access log files of the same access URL according to the second field to obtain the corresponding second attribute value; at least one second field and the corresponding second attribute value constitute the second compressed record.

6. The Nginx log compression analysis method of claim 1, wherein, When a single project has at least two Nginx instances, the method further comprises: reading the first compressed record and the second compressed record of each Nginx instance; compressing the first compressed record into a third compressed record; compressing the second compressed record with the same access URL into a fourth compressed record; sending the third compressed record and the fourth compressed record to the database.

7. The Nginx log compression analysis method of claim 1 or 6, wherein, The method further comprises adaptively recommending a preset compression duration T, and specifically comprises: Step S1. Calculating a compression ratio r of a single Nginx instance according to the number ns of Nginx instances and the overall compression ratio c of the business, r = c / ns; Step S2. Determining a minimum duration M of no less than r log files, in milliseconds; Step S3. Calculating z = M*N / (24*3600*1000), N representing the total number of time periods in which the number of log records in all durations M of the day is no less than r, and when z exceeds a set value, T = M, uploading and saving T, otherwise, proceeding to Step S4; Step S4. Letting M = 1.1M and rounding up, and repeating Step S3.

8. An Nginx log compression analysis system, characterized by, The server and the compression end, the server stores the Nginx instance to be compressed and analyzed, and the compression end is used to realize the Nginx log compression analysis method according to any one of claims 1-7.

9. An Nginx log compression analysis device, characterized by, The memory and the processor; The memory is used to store programs; The processor is used to execute the programs, and realize the Nginx log compression analysis method according to any one of claims 1-7.

10. A readable storage medium, characterized by, The computer program is stored thereon, and when executed by the processor, realizes the Nginx log compression analysis method according to any one of claims 1-7.

Images