Distributed tracking data downsampling processing method and system

Abstract

The invention belongs to the technical field of data processing, and particularly relates to a distributed tracking data down-sampling processing method and system, and the method comprises the steps: servers in different regional intranets report tracking data, abstract information extraction and index construction are carried out on the tracking data, a central control end reads and processes tracking abstract information, and a sampling collection decision is made; and according to the sampled and collected decision data, indexing to the corresponding tracking data fragment from the local, and reporting to a tracking data back-end management platform. The system comprises a data acquisition end, a central control end and a tracking data back-end management platform. The central control end can collect error transactions and slow transactions with time consumption higher than a threshold value for tracking data of intranets in different areas and collect normal transactions according to a certain proportion, so that the comprehensiveness and reliability of down-sampling collection of the tracking data can be ensured, and the data collection scale can be greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a method and system for downsampling distributed tracing data. Background Art

[0002] Distributed tracing is a technical means that helps monitor and quickly locate problems by capturing data of function calls or service calls within an application.

[0003] The quality of distributed tracing depends on the supply of rich-dimensional data, and its data volume is positively correlated with the following factors: the scale of sampling objects, such as the number of microservice processes, the number of servers, etc.; the sampling depth, such as the function instrumentation level; the sampling frequency, such as tracing an API interface (Application Programming Interface), when the QPS (Queries Per Second) of the API interface increases, correspondingly, the number of traced data will increase linearly.

[0004] Moreover, the business is globally deployed and there are cross-region calls. To restore the real link state, it is necessary to collect tracing data generated in various regions of the world.

[0005] Among these huge link data, normal link data even occupies more than 95%. However, the value generated by this part of normal tracing data is almost the same as that of 1% of normal data, and they are all only used to draw the service traffic topology map. Therefore, it is necessary to adopt a downsampling method to reduce the data scale.

[0006] Currently, there are the following downsampling schemes: 1. Proportional sampling: A low sampling ratio will cause the loss of Error transactions (i.e., error transactions) and slow transactions, and a high sampling ratio cannot meet the requirement of reducing the data scale. 2. Sampling based on attribute selection: It cannot handle the collection of Error transactions and slow transactions well because they cannot be predicted in advance, and there is no pre-set attribute to force their collection. 3. Tail sampling: This method requires pre-collecting data in one place. After collecting the complete link data of a transaction, it decides whether to collect according to the status of the transaction. This method can handle Error transactions and slow transactions well, and can also customize the proportional sampling of normal transactions. But its premise is at the cost of huge data transmission and storage, and it does not solve the problem.

[0007] It should be noted that the information disclosed in the above background art section is only used to strengthen the understanding of the background of the present invention, and therefore may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0008] The object of the present invention is to solve the technical problems existing in the background art. To this end, a distributed trace data downsampling processing method and system are provided.

[0009] To achieve the above object, the technical solutions adopted by the present invention are as follows: A distributed trace data downsampling processing method, comprising the following steps: Step S1: Servers in the intranet of different regions report trace data to the file exporter; Step S2: The file exporter receives the reported trace data and persists the trace data to the disk; Step S3: The trace proxy collector continuously reads the trace data from the disk, asynchronously extracts summary information from the trace data and constructs an index, and at the same time persists it to the disk; Step S4: The trace proxy collector reports the trace summary information to the collection server of the central control end, and the collection server receives the reported trace summary information and writes it into the data storage; Step S5: The control server of the central control end reads and processes the trace summary information from the data storage, makes a sampling collection decision, and then writes it back into the data storage; Step S6: The collection server reads the sampling collection decision data from the data storage and distributes the sampling collection decision data to the trace proxy collector; Step S7: The trace proxy collector receives the sampling collection decision data of the central control end, finds the corresponding trace data segment from the local index according to the sampling collection decision data, and reports it to the trace data back-end management platform.

[0010] The following is a further limited technical solution of the method in the present invention. In step S3, when extracting summary information from the trace data, it includes: Across the full text, perform field standardization on the trace data; Based on the HTTP status code and custom error markers, determine error transactions; Based on a dynamic threshold, determine slow transactions; Based on an adaptive proportional sampling algorithm, perform hash sampling; Generate trace summary information, where the trace summary information includes the trace data ID and its corresponding HTTP status code, error marker hit, transaction duration, or hash value.

[0011] The following is a further limited technical solution of the method in the present invention. In step S3, when constructing an index for the trace data, it includes: parsing and obtaining the trace data ID from the trace summary information, constructing a one-to-many mapping relationship from the trace data ID to the file where the trace data is located and the file line; based on the index, after the trace proxy collector receives the sampling collection decision data, find the corresponding trace data segment according to the trace data ID.

[0012] The following are the technical solutions for further limiting the method in the present invention. The tracing proxy collector sequentially writes the tracing summary information into the local disk in the form of text lines, writes the index information into the local disk through the local KV storage engine, and performs IO exclusive isolation on the tracing data, tracing summary information, and index information.

[0013] The following are the technical solutions for further limiting the method in the present invention. The summary information with the same tracing data ID is stored in the directory named after the value of the tracing data ID, and the control server reads all the corresponding summary information according to the tracing data ID.

[0014] The following are the technical solutions for further limiting the method in the present invention. In step S5, the control server reads and processes the tracing summary information from the data storage, and makes a sampling collection decision, including: Step S51: Determine whether the data in the previous two minutes of the current time has been decided; If so, mark the decided data and end. If not, read all the tracing summary information corresponding to the time from the data storage; Step S52: Determine whether all the tracing summary information read from the data storage corresponding to the time is empty; If so, mark the empty data and end. If not, query the summary information directory data in the data storage using the tracing data ID; Step S53: Determine whether the summary information directory data can be queried in the data storage using the tracing data ID; If it cannot be queried, mark the corresponding data and end. If it can be queried, query the HTTP status code and error flag hit in the summary information directory data; Step S54: Determine whether the HTTP status code is 0 and whether there is an error flag hit; If either the HTTP status code is 0 or there is an error flag hit, it means that an error transaction is included. Determine whether a sampling task has been created for the corresponding tracing data ID. If so, mark the corresponding data and end. If not, create a sampling task for the tracing data ID in the data storage; If neither the HTTP status code is 0 nor there is an error flag hit, query the transaction duration in the summary information directory data; Step S55: Determine whether the transaction duration exceeds the dynamic threshold; If it exceeds, it means that a slow transaction is included. Determine whether a sampling task has been created for the corresponding tracing data ID. If so, mark the corresponding data and end. If not, create a sampling task for the tracing data ID in the data storage; If it does not exceed, it indicates that the tracking data status is normal, and the hash value in the query summary information directory data is retrieved. Step S56: Determine whether the hash value is hit. If it is hit, determine whether a sampling task has been created for the corresponding tracking data ID. If so, mark the corresponding data and end. If not, create a sampling task for the tracking data ID in the data storage. If it is not hit, mark the corresponding data and end.

[0015] The following is a further limited technical solution of the method in the present invention. In the data storage, the data survival time is set to 5 minutes, and the control server initiates the decision-making process by scanning every minute.

[0016] The distributed tracking data downsampling processing system is used to implement the above-mentioned distributed tracking data downsampling processing method, and includes a data collection end, a central control end, and a tracking data back-end management platform. The data collection end is deployed in each regional intranet and is used to collect tracking data, its summary information, and indexes, send the tracking summary information to the central control end, receive the sampling collection decision data, and extract the corresponding tracking data segments based on the sampling collection decision data and indexes and send them to the tracking data back-end management platform. The central control end makes a sampling collection decision based on the tracking summary information and sends the sampling collection decision to the data collection end. The tracking data back-end management platform is used for tracking data storage and display.

[0017] The following is a further limited technical solution of the system in the present invention. The data collection end includes a file exporter and a tracking agent collector. The file exporter is responsible for receiving the reported tracking data and persisting it to the disk. The tracking agent collector is responsible for extracting summary information and constructing indexes for the tracking data, persisting them to the disk at the same time, sending the tracking summary information to the central control end, receiving the sampling collection decision from the central control end, indexing the corresponding tracking data segments from the local area, and reporting them to the tracking data back-end management platform.

[0018] The following is a further limited technical solution of the system in the present invention. The central control end includes a collection server, a data storage, and a control server. The collection server is responsible for establishing two-way communication with the tracking agent collector, receiving the reported tracking summary information and writing it into the data storage, and sending the sampling collection decision data to the tracking agent collector. The control server is responsible for reading the tracking summary information from the data storage, making a sampling collection decision, and then writing it back into the data storage.

[0019] Compared with the prior art, the present invention has the following technical effects: The central control end of the present invention can collect error transactions of tracking data, slow transactions with a collection time higher than the threshold, and normal transactions collected at a certain ratio for different intranets, which can not only ensure the comprehensiveness and reliability of the downsampling collection of tracking data, but also greatly reduce the data collection scale.

[0020] The present invention will be further described below in conjunction with the drawings and embodiments. Description of the Drawings

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0022] Figure 1 is a system connection block diagram of the present invention; Figure 2 is a system connection block diagram of the present invention for different intranets; Figure 3 is a schematic flow chart of the control server in the present invention for reading, processing tracking summary information from the data storage and making a sampling collection decision. Detailed Embodiments

[0023] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the following will make a detailed description of the specific embodiments of the present invention in conjunction with the drawings. Many specific details are set forth in the following description in order to fully understand the present invention. However, the present invention can be implemented in many other ways different from those described herein. Those skilled in the art can make similar improvements without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0024] As Figures 1 - 3 shown, this embodiment provides a distributed tracking data downsampling processing method and a distributed tracking data downsampling processing system.

[0025] As Figure 1 shown, the distributed tracking data downsampling processing system mainly includes a data collection end, a central control end, and a tracking data back-end management platform.

[0026] The data collection terminal is deployed in each regional intranet and is used to collect trace data, its summary information, and indexes, and send the trace summary information to the central control terminal. It receives sampling collection decision data and extracts the corresponding trace data segments based on the sampling collection decision data and indexes and sends them to the trace data backend management platform. Specifically, the data collection terminal includes a file exporter and a trace agent collector. Among them, the file exporter is responsible for receiving the reported trace data and persisting it to the disk; the trace agent collector is responsible for extracting summary information from the trace data and building indexes, and at the same time persisting them to the disk, then sending the trace summary information to the central control terminal, and receiving the sampling collection decision of the central control terminal, indexing the corresponding trace data segments from the local and reporting them to the trace data backend management platform.

[0027] The central control terminal makes a sampling collection decision based on the trace summary information and sends the sampling collection decision to the data collection terminal. Specifically, the central control terminal includes a collection server, a data storage, and a control server. Among them, the collection server is responsible for establishing two-way communication with the trace agent collector, receiving the reported trace summary information and writing it into the data storage, and sending the sampling collection decision data to the trace agent collector; the control server is responsible for reading the trace summary information from the data storage, making a sampling collection decision, and then writing the sampling collection decision back into the data storage.

[0028] The trace data backend management platform is used for trace data storage and display.

[0029] The distributed trace data downsampling processing method, as Figure 1 shown, includes the following steps: Step S1: Servers in different regional intranets report trace data to the file exporter. Among them, each regional intranet contains multiple servers. It should be noted that the regional intranet is usually distinguished according to national regions, such as the South Korea region, the China region, the Thailand region, etc.; however, dividing the regions in China is also within the protection scope of the present invention, such as the Southeast region, the Southwest region, the Northwest region, the Northeast region, etc. According to the different traffic in each region, different numbers of servers are set.

[0030] Step S2: The file exporter receives the reported trace data and persists the trace data to the disk.

[0031] Step S3: The trace agent collector continuously reads the trace data from the disk and asynchronously extracts summary information from the trace data and builds indexes, and at the same time persists them to the disk.

[0032] The extraction of summary information from the trace data includes: Spanning the full text, standardizing the fields of the trace data; Determine error transactions based on HTTP status codes and custom error tags; Determine slow transactions based on a dynamic threshold; wherein, the dynamic threshold can be dynamically set according to the actual application; Perform hash sampling based on an adaptive ratio sampling algorithm; Generate trace summary information, which includes the trace data ID and its corresponding HTTP status code, error tag hit, transaction duration, or hash value.

[0033] Construct an index for the trace data, including: parsing the trace data ID from the trace summary information to construct a one-to-many mapping relationship from the trace data ID to the file and file line where the trace data is located; based on the index, after the trace proxy collector receives the sampled collection decision data, find the corresponding trace data segment according to the trace data ID.

[0034] The trace proxy collector writes the trace summary information to the local disk in the form of text lines, writes the index information to the local disk through the local KV storage engine, and isolates the I / O of the trace data, trace summary information, and index information.

[0035] The summary information with the same trace data ID is stored in a directory named after the value of the trace data ID (including the HTTP status code, error tag hit, transaction duration, hash value), and the control server reads all the corresponding summary information according to the trace data ID.

[0036] Step S4: The trace proxy collector reports the trace summary information to the collection server of the central control end, and the collection server receives the reported trace summary information and writes it into the data storage.

[0037] Step S5: The control server of the central control end reads and processes the trace summary information from the data storage, makes a sampled collection decision, and then writes it back into the data storage.

[0038] In the data storage, set the data survival time to 5 minutes, and the control server initiates the decision-making process by scanning every minute, as shown in the following table:

[0039] Among them, TTL is the survival time, which is used to limit the longest time for data packets to be transmitted in the network.

[0040] Specifically, the control server reads and processes the trace summary information from the data storage and makes a sampled collection decision (i.e., the decision-making process), as Figure 3 shown, the decision-making process includes: Step S51: Determine whether the data in the previous two minutes of the current time has been decided; If so, mark the data that has been decided and end. If not, read all the trace summary information corresponding to the time (the corresponding time refers to: two minutes before the current time) from the data storage; Step S52: Determine whether all the trace summary information corresponding to the time read from the data storage is empty; If so, mark the empty data and end. If not, query the summary information directory data in the data storage using the trace data ID; Step S53: Determine whether the summary information directory data can be queried in the data storage using the trace data ID; If it cannot be queried, mark the corresponding data and end. If it can be queried, query the HTTP status code and error flag hit in the summary information directory data; Step S54: Determine whether the HTTP status code is 0 and whether there is an error flag hit; If either the HTTP status code is 0 or there is an error flag hit, it indicates that there is an error transaction. Determine whether the corresponding trace data ID has already created a sampling task. If so, mark the corresponding data and end. If not, create a sampling task for the trace data ID in the data storage (i.e., sampling collection decision); If neither the HTTP status code is 0 nor there is an error flag hit, query the transaction duration in the summary information directory data; Step S55: Determine whether the transaction duration exceeds the dynamic threshold; If it exceeds, it indicates that there is a slow transaction. Determine whether the corresponding trace data ID has already created a sampling task. If so, mark the corresponding data and end. If not, create a sampling task for the trace data ID in the data storage (i.e., sampling collection decision); If it does not exceed, it indicates that the trace data status is normal. Query the hash value in the summary information directory data; Step S56: Determine whether the hash value is hit; If it is hit, determine whether the corresponding trace data ID has already created a sampling task. If so, mark the corresponding data and end. If not, create a sampling task for the trace data ID in the data storage (i.e., sampling collection decision); If it is not hit, mark the corresponding data and end.

[0041] Step S6: The collection server reads the sampling collection decision data from the data storage and sends the sampling collection decision data to the trace proxy collector.

[0042] Step S7: The tracing agent collector receives the sampling collection decision data from the central control terminal, indexes the corresponding tracing data segments from the local according to the sampling collection decision data, and reports the tracing data segments to the tracing data backend management platform.

[0043] The working process of this embodiment will be further described below: As Figure 2 shown, the intranet in Area A has multiple servers (Server A1, A2, A3...), File Exporter A, Disk A, and Tracing Agent Collector A. The intranet in Area A is communicatively connected to the central control terminal and the tracing data backend management platform. Among them, Server A1, A2, and A3 respectively report the tracing data to File Exporter A. File Exporter A receives the reported tracing data and persists the tracing data to Disk A. Tracing Agent Collector A reads the tracing data from Disk A, extracts the summary information of the tracing data and constructs an index, and at the same time persists the tracing summary information and the index to Disk A. Tracing Agent Collector A communicates bidirectionally with the central control terminal to transmit the tracing summary information and the sampling collection decision. Tracing Agent Collector A indexes the corresponding tracing data segments from the local and reports the tracing data segments to the tracing data backend management platform.

[0044] The intranet in Area B has multiple servers (Server B1, B2, B3...), File Exporter B, Disk B, and Tracing Agent Collector B. The intranet in Area B is communicatively connected to the central control terminal and the tracing data backend management platform. Among them, Server B1, B2, and B3 respectively report the tracing data to File Exporter B. File Exporter B receives the reported tracing data and persists the tracing data to Disk B. Tracing Agent Collector B reads the tracing data from Disk B, extracts the summary information of the tracing data and constructs an index, and at the same time persists the tracing summary information and the index to Disk B. Tracing Agent Collector B communicates bidirectionally with the central control terminal to transmit the tracing summary information and the sampling collection decision. Tracing Agent Collector B indexes the corresponding tracing data segments from the local and reports the tracing data segments to the tracing data backend management platform.

[0045] And so on, the intranets in different areas can all be communicatively connected to the central control terminal and the tracing data backend management platform.

[0046] Therefore, the central control terminal can collect error transactions of the tracing data, slow transactions whose collection time exceeds the threshold, and normal transactions collected at a certain ratio in the intranets of different areas, which can not only ensure the comprehensiveness and reliability of the downsampled collection of the tracing data, but also greatly reduce the data collection scale.

[0047] The above are only the preferred embodiments of the present invention and do not impose any formal restrictions on the present invention. Any person skilled in the art can, without departing from the scope of the technical solution of the present invention, make many possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above, or modify it into equivalent embodiments with equivalent changes. Therefore, all equivalent changes made in accordance with the shape, structure and principle of the present invention without departing from the content of the technical solution of the present invention shall be covered by the protection scope of the present invention.

Claims

1. A method for downsampling distributed tracing data, characterized in that It includes the following steps: Step S1: Servers in the intranet of different regions report tracking data to the file exporter; Step S2: The file exporter receives the reported tracking data and persists the tracking data to the disk; Step S3: The tracking agent collector continuously reads the tracking data from the disk, asynchronously extracts summary information and constructs an index for the tracking data, and persists it to the disk at the same time; Step S4: The tracking agent collector reports the tracking summary information to the collection server of the central control end, and the collection server receives the reported tracking summary information and writes it into the data storage; Step S5: The control server of the central control end reads and processes the tracking summary information from the data storage, makes a sampling collection decision, and writes it back into the data storage; Step S6: The collection server reads the sampling collection decision data from the data storage and distributes the sampling collection decision data to the tracking agent collector; Step S7: The tracking agent collector receives the sampling collection decision data of the central control end, finds the corresponding tracking data segment from the local index according to the sampling collection decision data, and reports it to the tracking data back-end management platform.

2. The distributed trace data downsampling processing method according to claim 1, wherein In step S3, when extracting summary information from the tracking data, it includes: Across the full text, perform field standardization on the tracking data; Based on the HTTP status code and custom error markers, determine error transactions; Based on dynamic thresholds, determine slow transactions; Based on the adaptive proportional sampling algorithm, perform hash sampling; Generate tracking summary information, which includes the tracking data ID and its corresponding HTTP status code, error marker hit, transaction duration, or hash value.

3. The distributed trace data downsampling processing method according to claim 2, wherein In step S3, when constructing an index for the tracking data, it includes: parsing the tracking data ID from the tracking summary information to construct a one-to-many mapping relationship between the tracking data ID and the file and file line where the tracking data is located; based on the index, after the tracking agent collector receives the sampling collection decision data, find the corresponding tracking data segment according to the tracking data ID.

4. The distributed trace data downsampling processing method according to claim 3, wherein, The tracking agent collector writes the tracking summary information to the local disk in the form of text lines, writes the index information to the local disk through the local KV storage engine, and performs IO exclusive isolation on the tracking data, tracking summary information, and index information.

5. The distributed trace data downsampling processing method according to claim 2, wherein, The summary information with the same tracking data ID is stored in a directory named after the value of the tracking data ID, and the control server reads all the corresponding summary information according to the tracking data ID.

6. The distributed trace data downsampling processing method according to claim 5, wherein, In step S5, the control server reads and processes the tracking summary information from the data storage and makes a sampling collection decision, including: Step S51: Determine whether the data in the previous two minutes of the current time has been decided; If so, mark the decided data and end. If not, read all the tracking summary information corresponding to the time from the data storage; Step S52: Determine whether all the tracking summary information read from the data storage corresponding to the time is empty; If so, mark the empty data and end. If not, query the summary information directory data in the data storage using the tracking data ID; Step S53: Determine whether the summary information directory data can be queried in the data storage using the tracking data ID; If query fails, mark the corresponding data and end. If query succeeds, query the HTTP status code and error flag hit in the summary information catalog data; Step S54: Determine whether the HTTP status code is 0 and whether there is an error flag hit; If either the HTTP status code is 0 or there is an error flag hit, it indicates an error transaction. Determine whether a sampling task has been created for the corresponding trace data ID. If yes, mark the corresponding data and end. If no, create a sampling task for the trace data ID in the data storage; If neither the HTTP status code is 0 nor there is an error flag hit, query the transaction duration in the summary information catalog data; Step S55: Determine whether the transaction duration exceeds the dynamic threshold; If it exceeds, it indicates a slow transaction. Determine whether a sampling task has been created for the corresponding trace data ID. If yes, mark the corresponding data and end. If no, create a sampling task for the trace data ID in the data storage; If it does not exceed, it indicates that the trace data status is normal. Query the hash value in the summary information catalog data; Step S56: Determine whether the hash value is hit; If it is hit, determine whether a sampling task has been created for the corresponding trace data ID. If yes, mark the corresponding data and end. If no, create a sampling task for the trace data ID in the data storage; If it is not hit, mark the corresponding data and end.

7. The distributed trace data downsampling processing method according to claim 6, wherein, In the data storage, set the data survival time to 5 minutes, and control the server to initiate the decision-making process by scanning every minute.

8. A distributed trace data downsampling processing system for implementing the distributed trace data downsampling processing method according to any one of claims 1-7, characterized in that, It includes a data collection end, a central control end, and a trace data back-end management platform; The data collection end is deployed in each regional intranet, used to collect trace data, its summary information, and indexes, send the trace summary information to the central control end, receive the sampling collection decision data, and extract the corresponding trace data segments based on the sampling collection decision data and indexes and send them to the trace data back-end management platform; The central control end makes a sampling collection decision based on the trace summary information and sends the sampling collection decision to the data collection end; The trace data back-end management platform is used for trace data storage and display.

9. The distributed trace data downsampling processing system according to claim 8, wherein The data collection end includes a file exporter and a trace proxy collector; The file exporter is responsible for receiving the reported trace data and persisting it to the disk; The trace proxy collector is responsible for extracting summary information and building indexes for the trace data, persisting them to the disk at the same time, then sending the trace summary information to the central control end, receiving the sampling collection decision from the central control end, indexing the corresponding trace data segments from the local and reporting them to the trace data back-end management platform.

10. The distributed trace data downsampling processing system according to claim 9, wherein The central control end includes a collection server, a data storage, and a control server; The collection server is responsible for establishing two-way communication with the trace proxy collector, receiving the reported trace summary information and writing it into the data storage, and sending the sampling collection decision data to the trace proxy collector; The control server is responsible for reading the trace summary information from the data storage, making a sampling collection decision, and then writing it back into the data storage.

Images