Distributed Link Error and Slow Log Fast Collection and Filtering Method, System and Medium

Abstract

The present invention provides a method, system and medium for fast collection and filtering of distributed link error and slow logs, including: extracting log data within a specified time range; storing the log data in a circular buffer, with the write pointer pointing to the buffer block; judging each log data item one by one to determine if the status is not 200, and recording it as a log with a status other than 200; storing the field traceId of the log with a status other than 200 in the error and slow log traceId set S1; reading the log data in the block pointed to by the read pointer in the circular buffer and the log data in the blocks before and after the pointer, and reading the problem traceId data D1 of the set S1; if the traceIds of the data in S1 and D1 match, it is marked as problem link log data; taking out the problem link log data and storing it in the error and slow log set L1; using the merge sort method to sort the traceId field id in ascending order for the data in the error and slow log set L1 to obtain the set R1; using the merge sort method to sort the time field time from earliest to latest for the set R1 to obtain the set R2; reporting the error and slow log links in the set R2.

Description

Technical Field

[0001] The present invention relates to the field of distributed link technology, and in particular to a method, system and medium for quickly collecting and filtering distributed link error and slowness logs. Background Art

[0002] To cope with diverse and complex businesses, system architectures have evolved from single-machine, large-scale software to microservices. Microservices are built on diverse software modules, which may be developed by different teams, implemented using different programming languages, and deployed across multiple servers. Therefore, a single service failure can cause anomalies in dozens of services. Distributed tracing systems can record information within a request, from the moment a user clicks on a page to the entire processing phase of that request, including the number of services involved, the duration of each service, and any anomalies encountered.

[0003] The distributed link system records information within the scope of the request. The more link data collected, the higher the cost. When any node encounters a slowdown log, all link data for that request must be collected, even if other link data comes before or after the node data for that link, or even if other link data exists at multiple nodes. Currently, the industry's most common sampling methods follow this approach, such as fixed-ratio sampling, reservoir sampling, and mixed sampling.

[0004] The above sampling method can ensure the integrity of the entire call chain, but it also brings two problems:

[0005] (1) Some abnormalities and slow requests are not collected due to sampling methods. However, these error and slow log link data are important to the business, and the error and slow log completeness rate does not meet the high-quality business requirements. How to improve the error and slow log collection rate, and extract the full link node log information involved in any node with error and slow characteristics;

[0006] (2) Error and slow log collection is a stream processing scenario. Logs are continuously and uninterruptedly generated in large quantities. Conventional methods for filtering error and slow logs are inefficient, seriously affecting the efficiency and quality of operation and maintenance. How can we quickly and cost-effectively filter out logs with error and slow log characteristics from massive logs? Summary of the Invention

[0007] In response to the deficiencies in the prior art, the present invention provides a method, system, and medium for quickly collecting and filtering distributed link error and slowness logs.

[0008] According to the present invention, a method, system, and medium for quickly collecting and filtering distributed link error and slowness logs are provided. The solution is as follows:

[0009] In a first aspect, a method for quickly collecting and filtering distributed link error and slowness logs is provided, the method comprising:

[0010] Step 1: Extract log data within a specified time range from log files of multiple servers;

[0011] Step 2: storing the log data in a ring buffer, wherein the write pointer in the ring buffer points to the buffer block;

[0012] Step 3: Determine whether the status of each log data item is non-200; if so, record it as a status non-200 log;

[0013] Step 4: Store the traceId field of the non-200 status log in the error log traceId set S1;

[0014] Step 5: Read the block pointed to by the read pointer in the ring buffer and the log data of the blocks before and after the pointer, and read the problem traceId data D1 of the set S1; compare the traceIds of the S1 and D1 data to see if they match; if so, mark them as problem link log data;

[0015] Step 6: Take out the problem link log data and store it in the error and slowness log set L1;

[0016] Step 7: Use the merge sort method to sort the traceId field ID of the error slow log set L1 from small to large to obtain set R1; use the merge sort method to sort the time field time of the set R1 from early to late to obtain set R2;

[0017] Step 8: Report the set R2 to the error and slow log link.

[0018] Preferably, the step 1 comprises:

[0019] Step 1.1: Collect historical request data and obtain the longest request duration T1 of a single request;

[0020] Step 1.2: Determine the longest request duration T1*1.2 as the tumbling moving window time width T2;

[0021] Step 1.3: From the log files of multiple servers, simultaneously extract the log data within the time range of (n-1)T2-n*T2 and record it as the nth batch of log data.

[0022] Preferably, the step 2 comprises:

[0023] Step 2.1: Configure the ring buffer;

[0024] Step 2.2: Initialize the write and read pointers for the circular buffer log data;

[0025] Step 2.3: Sequentially store the log data in the block pointed to by the write pointer, and then move the write pointer to the next block.

[0026] Preferably, the said Step 5 includes:

[0027] Step 5.1: Read the block log data of the block before the read pointer, the block pointed to by the read pointer, and the block after the read pointer;

[0028] Step 5.2: Read the problem traceId data D1 of the error and slow log traceId set S1;

[0029] Step 5.3: Use a comparison program to compare the error and slow log traceId set S1 in the buffer with the problem traceId data D1. If they match, mark it as the problem link log data.

[0030] In the second aspect, a distributed link error and slow log fast acquisition and filtering system is provided. The system includes:

[0031] Module 1: Extract the log data within a specified time range from the log files of multiple servers;

[0032] Module 2: Store the log data in a circular buffer, and the write pointer in the circular buffer points to the buffer block;

[0033] Module 3: Judging one by one whether the status of the log data is non-200; if so, record it as the log with status non-200;

[0034] Module 4: Store the traceId field of the log with status non-200 in the error and slow log traceId set S1;

[0035] Module 5: Read the log data of the block pointed to by the read pointer in the circular buffer and the blocks before and after the pointer, and read the problem traceId data D1 of the set S1; compare whether the traceIds of the S1 and D1 data match; if they match, mark it as the problem link log data;

[0036] Module 6: Take out the problem link log data and store it in the error and slow log set L1;

[0037] Module 7: Use the merge sort method to sort the traceId field id of the error and slow log set L1 data from small to large to obtain the set R1; use the merge sort method to sort the time field time of the set R1 from first to last to obtain the set R2;

[0038] Module 8: Report the set R2 on the error and slow log link.

[0039] Preferably, the module 1 includes:

[0040] Module 1.1: Statistically analyze historical request data to obtain the longest request duration T1 for a single request;

[0041] Module 1.2: Determine the tumbling moving window time width T2 as T1 * 1.2;

[0042] Module 1.3: Simultaneously extract log data within the time range of (n - 1)T2 - n*T2 from the log files of multiple servers, and record it as the nth batch of log data.

[0043] Preferably, the module 2 includes:

[0044] Module 2.1: Configure a circular buffer;

[0045] Module 2.2: Initialize the write and read pointers for the circular buffer log data;

[0046] Module 2.3: Sequentially store the log data in the block pointed to by the write pointer, and then the write pointer points to the next block.

[0047] Preferably, the module 5 includes:

[0048] Module 5.1: Read the block log data of the block before the read pointer, the block pointed to by the read pointer, and the block after the read pointer;

[0049] Module 5.2: Read the problem traceId data D1 of the error and slow log traceId set S1;

[0050] Module 5.3: Use a comparison program to compare the error and slow log traceId set S1 in the buffer with the problem traceId data D1. If there is a match, it is marked as problem link log data.

[0051] In a third aspect, a computer-readable storage medium storing a computer program is provided, characterized in that when the computer program is executed by a processor, the steps of the method are implemented.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. When collecting error and slow log data, the present invention uses the current buffer plus the data of the front and back buffers as the statistical data for a time window. This method ensures that all node data of the links involved in the requests falling within this time window are covered, and achieves the purpose of the smallest statistical data set, reducing the amount of data statistics and improving efficiency;

[0054] 2. The log data of different time windows in the present invention is stored in different blocks of a circular buffer. Write and read pointers are used to point to the corresponding buffer, and data is written and read in a planned manner. Moreover, the memory buffer improves the data reading and writing efficiency, as well as the collection and filtering efficiency of error and slow logs.

[0055] 3. The present invention performs double sorting on the reported data in terms of link and time, enabling operation and maintenance personnel to clearly see the error and slow link information at a glance, and improving the operation and maintenance efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0057] Figure 1 It is a schematic diagram of business processing;

[0058] Figure 2 It is a schematic diagram of the call chain;

[0059] Figure 3 It is a schematic diagram of the call chain;

[0060] Figure 4 It is a schematic diagram of the call chain;

[0061] Figure 5 It is a schematic diagram of the circular cache;

[0062] Figure 6 It is a total flow chart of the full process of error and slow log filtering;

[0063] Figure 7 It is a schematic diagram of step 1 of the present invention;

[0064] Figure 8 It is a schematic diagram of step 2 of the present invention;

[0065] Figure 9 It is a schematic diagram of step 5 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0066] The following will describe the present invention in detail with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those of ordinary skill in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0067] The embodiment of the present invention provides a method for quickly collecting and filtering error and slow logs in a distributed link. Among them, the terms involved in the present invention include the following:

[0068] Distributed system: It refers to a system formed by connecting multiple decentralized computers through an interconnected network. The processing and control functions of the system are distributed on each computer. Distributed deployment means storing the application dispersedly on multiple independent machine devices, deploying the same application on multiple machines simultaneously, and using multiple storage servers for load balancing.

[0069] Link tracing: It provides complete request call link restoration, call request volume statistics, link topology, and application analysis tools for distributed applications, which can help developers quickly analyze and diagnose the request call path under the distributed application architecture and improve the development diagnosis efficiency in the microservices era. As Figure 1 shown, each colored call path is a link, passing through the application programs and methods under those IP servers; the request may fall on different instances with the same function. Example: The same application is deployed with n replicas, and each replica is an instance.

[0070] Link tracing traceId, spanId:

[0071] Refer to Figure 2 shown. Inside an interface, multiple services may be called. If you want to know which link an interface has a problem in, you need to know which services the interface calls and the call order. If these services are strung together, it looks like a chain, which is called a call chain.

[0072] Refer to Figure 3 shown. To implement a call chain, an identifier needs to be made for each call, and then the services are arranged according to the size of the identifier, so that the call order can be seen more clearly. This identifier is named spanId.

[0073] Refer to Figure 4 shown. In the actual scenario, to know the situation of a certain request call, only having spanId is not enough. A unique identifier needs to be made for each request, so that all services called by this request can be found according to the identifier. This identifier is named traceId.

[0074] Error and slow logs: Logs of abnormal error reports and slow requests;

[0075] Circular buffer: A circular buffer is a first-in-first-out, fixed-size, circular buffer with the head and tail connected. The read pointer points to the data that can be read in the circular buffer, and the write pointer points to the data that can be written in the circular buffer, which is suitable for caching data streams. When the buffer is used up and new data needs to be stored, the oldest data is discarded and the latest data is saved. Refer to Figure 5 shown.

[0076] A method for quickly collecting and filtering error and slow logs in a distributed link provided by the present invention, refer to Figure 6As shown below, the specific content is as follows:

[0077] Step 1: Extract log data within a specified time range from the log files of multiple servers.

[0078] Refer to Figure 7 As shown, this step specifically includes:

[0079] 1) Statistic historical request data to obtain the longest request duration T1 for a single request.

[0080] 2) Determine the tumbling moving window time width T2 as T1 * 1.2.

[0081] 3) From the log files of multiple servers, simultaneously extract log data within the time range of (n - 1)T2 - n*T2, and record it as the nth batch of log data.

[0082] Step 2: Store the log data in a circular buffer, and the write pointer in the circular buffer points to the buffer block.

[0083] Refer to Figure 8 As shown, this step specifically includes:

[0084] 1) Configure the circular buffer. Configure the circular buffer memory size to 6GB, divide the circular buffer memory evenly into 6 blocks, each block is 1GB, and name them the 0th block (Q0), the 1st block (Q1)... the 6th block (Q5).

[0085] 2) Initialize the write and read pointers for the circular buffer log data. The write pointer initially points to the address Q0 of the 0th block, and after writing data once, the write pointer points to the next block; the read pointer initially points to the 3rd block Q3, and after reading data once, the read pointer points to the next block.

[0086] 3) Store the log data sequentially in the block pointed to by the write pointer (that is, the log data of the Nth time window. If N <= 6, store it in the Nth block; if N > 6, store it in the block with the remainder of N divided by 6 to overwrite the original data), and then the write pointer points to the next block.

[0087] Step 3: Use the error and slow log judgment program to judge whether the status status of each log data is non - 200 one by one; if so, record it as a log with status non - 200.

[0088] Step 4: Store the field traceId of the log with status non - 200 in the error and slow log traceId set S1.

[0089] Step 5: Read the log data of the block pointed to by the read pointer in the circular buffer and the blocks before and after the pointer, and read the problem traceId data D1 of set S1; compare whether the traceIds of S1 and D1 data match; if they match, mark it as problem link log data.

[0090] Refer to Figure 9 As shown, this step specifically includes:

[0091] 1) Read the block log data of the block before the read pointer (Q N -1), the block pointed to by the read pointer (Q N ), and the block after the read pointer (Q N +1) (to ensure that when the entire request time may fall within the block pointed to by the read pointer, there are log data of other nodes in the link in the blocks before and after the block pointed to by the read pointer). Specifically, as Figure 9 shown, when reading N data, it is necessary to read the data of the previous and subsequent blocks at the same time. The reason is that when processing the request link data falling within block N, the start time of the request may be in block N - 1, or the end time of the request may be in block N + 1; in order to ensure that all the full-link data of the request falling within block N is complete, the data of the previous and subsequent blocks need to be included in the statistics when counting the data of block N.

[0092] 2) Read the problem traceId data D1 of the wrong and slow log traceId set S1.

[0093] 3) Use a comparison program to compare the wrong and slow log traceId set S1 in the buffer and the problem traceId data D1. If they match, mark it as problem link log data.

[0094] Step 6: Take out the problem link log data and store it in the wrong and slow log set L1.

[0095] Step 7: Use the merge sort method to sort the data in the wrong and slow log set L1 in ascending order of the traceId field id to obtain set R1; use the merge sort method to sort set R1 in chronological order of the time field to obtain set R2.

[0096] Step 8: Report the wrong and slow log link of set R2.

[0097] Next, it will be further described in combination with the implementation scenario.

[0098] Usage scenario: Full-link log data collection of wrong and slow logs in a distributed system link.

[0099] As a professional network management system for integrating multiple department's business support systems, the centralized BOMC system itself has a vast amount of business support system log data. The storage log volume of a single province exceeds 300T. Currently, the centralized BOMC has centralized management of the vast amount of logs and stores the logs in a distributed link manner. The distributed link can record the log information within the request scope, including a user's click on a page to send a request and all the processing processes of this request, such as how many services and instances it passes through, on which machines it is executed, the time consumption and exception situations of each instance.

[0100] After centralized management of the logs, the key issue for operation and maintenance is how to quickly filter out slow error logs during the statistics and retrieval of the logs. During the peak period of business processing, such as common business processing like top - up payment and package change, a large amount of logs will be generated. The huge log stream data will continue to accumulate, resulting in the superposition of the filtering and extraction delay of slow error logs, and it is completely impossible to meet the requirement of real - time filtering of slow error logs during the business peak period.

[0101] When a system exception occurs during the business peak period, because the filtering rate of slow error logs is much lower than the log generation rate, operation and maintenance personnel cannot obtain the slow error logs of the entire link in a timely manner, which seriously affects the efficiency of operation and maintenance personnel in locating problems. It will consume a lot of energy and time in troubleshooting the error logs of the production system, and is very likely to increase the fault duration, resulting in assessment deductions and seriously affecting the customer perception.

[0102] For example, if there are 2 servers storing logs, if the logs of n servers are collected and slow error logs are filtered simultaneously, the fast collection and filtering architecture shown in the present invention Figure 6 is required.

[0103] The embodiment of the present invention provides a method, system and medium for fast collection and filtering of slow error logs in a distributed link. As long as any node in the link trace data of the request has link data that meets the sampling conditions, all the link data of this request will be collected, whether other link data is generated before or after the link node data of this link, or even other link data is generated on hundreds of machines in the distributed system. It will be quickly collected and filtered from the vast amount of logs, and the data will be sorted according to the link and time.

[0104] Log data is continuously extracted through a rolling moving window and stored in a corresponding buffer. The data in the current buffer and its front and rear areas is cleverly counted simultaneously to ensure that all node data involved in the link where the current buffer request is located is included. The rolling moving window continuously extracts non-duplicate data, the buffer improves the data reading rate, and the minimum data statistics interval improves the filtering efficiency. Finally, the filtered and sorted error and slow log information is reported. In this way, the acquisition cost is significantly reduced, and the acquisition rate and filtering efficiency of distributed link error and slow logs are improved, which can maximize the operation and maintenance troubleshooting efficiency and operation and maintenance quality, and ultimately improve customer satisfaction.

[0105] Those skilled in the art know that in addition to implementing the system and its various devices, modules, and units provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system and its various devices, modules, and units provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc. to achieve the same function. Therefore, the system and its various devices, modules, and units provided by the present invention can be regarded as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be regarded as the structure within the hardware component; the devices, modules, and units for implementing various functions can also be regarded as either software modules for implementing the method or the structure within the hardware component.

[0106] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined arbitrarily with each other.

Claims

1. A method for fast collection and filtering of distributed link error and slow logs, characterized in that, include: Step 1: Extract log data within a specified time range from log files of multiple servers; Step 2: storing the log data in a ring buffer, wherein a write pointer in the ring buffer points to a buffer block; Step 3: Determine whether the status of each log data item is non-200; if so, record it as a status non-200 log; Step 4: Store the traceId field of the non-200 status log in the error log traceId set S1; Step 5: Read the block pointed to by the read pointer in the ring buffer and the log data of the blocks before and after the pointer, and read the problem traceId data D1 of the set S1; compare the traceIds of the S1 and D1 data to see if they match; if so, mark them as problem link log data; Step 6: Take out the problem link log data and store it in the error and slowness log set L1; Step 7: Use the merge sort method to sort the traceId field ID of the error slow log set L1 from small to large to obtain set R1; use the merge sort method to sort the time field time of the set R1 from early to late to obtain set R2; Step 8: Report the set R2 to the error and slow log link.

2. The rapid acquisition and filtering method for distributed link error and slow logs according to claim 1, characterized in that, The step 1 comprises: Step 1.1: Collect historical request data and obtain the longest request duration T1 of a single request; Step 1.2: Determine the longest request duration T1*1.2 as the tumbling moving window time width T2; Step 1.3: From the log files of multiple servers, simultaneously extract the log data within the time range of (n-1)T2-n*T2 and record it as the nth batch of log data.

3. The method for fast acquisition and filtering of distributed link error and slow logs according to claim 1, characterized in that, The step 2 includes: Step 2.1: Configure the ring buffer; Step 2.2: Initialize the ring buffer log data write and read pointers; Step 2.3: The log data is sequentially stored in the block pointed to by the write pointer, and then the write pointer points to the next block.

4. The method for quickly collecting and filtering distributed link error and slow logs according to claim 1, wherein The step 5 comprises: Step 5.1: Read the block log data of the block before the read pointer, the block pointed to by the read pointer, and the block after the read pointer; Step 5.2: Read the problem traceId data D1 of the error log traceId set S1; Step 5.3: Use a comparison program to compare the error log traceId set S1 in the buffer with the problem traceId data D1. If they match, they are marked as problem link log data.

5. A distributed link error and slow log fast acquisition and filtering system, characterized in that include: Module 1: Extract log data within a specified time range from log files of multiple servers; Module 2: storing the log data in a ring buffer, wherein a write pointer in the ring buffer points to a buffer block; Module 3: Determine whether the status of each log data is non-200; if so, record it as a status non-200 log; Module 4: storing the traceId field of the non-200 status log in the error log traceId set S1; Module 5: Read the block log data pointed to by the read pointer in the circular buffer and the block log data before and after the pointer, and read the problem traceId data D1 of the set S1; Compare whether the traceIds of the S1 and D1 data match; If they match, mark it as problem link log data; Module 6: Take out the problem link log data and store it in the error and slow log set L1; Module 7: Use the merge sort method to sort the traceId field id in ascending order for the data in the error and slow log set L1 to obtain the set R1; Use the merge sort method to sort the time field time in ascending order for the set R1 to obtain the set R2; Module 8: Report the set R2 on the error and slow log link.

6. The distributed link error and slow log fast acquisition and filtering system according to claim 5, characterized in that [[ID=...]]The module 1 includes: Module 1.1: Statistic the historical request data and obtain the longest request duration T1 for a single request; Module 1.2: Determine the tumbling moving window time width T2 as 1.2 * the longest request duration T1; Module 1.3: Extract the log data with a time range of (n - 1)T2 - n*T2 from the log files of multiple servers, and record it as the nth batch of log data.

7. The distributed link error and slow log fast acquisition and filtering system according to claim 5, characterized in that The module 2 includes: Module 2.1: Configure the circular buffer; Module 2.2: Initialize the write and read pointers of the circular buffer log data; Module 2.3: Sequentially store the log data in the block pointed to by the write pointer, and then the write pointer points to the next block.

8. The distributed link error and slow log fast acquisition and filtering system according to claim 5, characterized in that The module 5 includes: Module 5.1: Read the block log data of the block before the read pointer, the block pointed to by the read pointer, and the block after the read pointer; Module 5.2: Read the problem traceId data D1 of the error and slow log traceId set S1; Module 5.3: Use a comparison program to compare the error and slow log traceId set S1 in the buffer and the problem traceId data D1. If they match, mark it as problem link log data.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 4.

Images