Distributed tracing data downsampling processing method and system

Abstract

The present invention belongs to the field of data processing technology, and particularly relates to a method and system for processing downsampling of distributed tracking data. The method comprises: servers in different regional networks report tracking data, extract summary information and construct indexes for the tracking data, a central control terminal reads and processes the tracking summary information, and makes sampling and collection decisions. Based on the sampling and collection decision data, the corresponding tracking data fragments are indexed locally and reported to a tracking data backend management platform. The system comprises a data collection terminal, a central control terminal, and a tracking data backend management platform. The central control terminal of the present invention can collect error transactions, slow transactions that take longer than a threshold, and normal transactions in a certain proportion for tracking data in different regional networks, thereby ensuring the comprehensiveness and reliability of downsampling collection of tracking data and greatly reducing the scale of data collection.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a distributed tracing data downsampling processing method and system. Background Art

[0002] Distributed tracing is a technology that helps monitor and quickly locate problems by capturing data about function calls or service calls within an application.

[0003] The quality of distributed tracing depends on the rich data supply. The amount of data is positively correlated with the following factors: the scale of the sampled objects, such as the number of microservice processes and servers; the sampling depth, such as the function tracking level; and the sampling frequency. For example, when tracing an API (application programming interface), the number of queries per second (QPS) of the API increases, and the number of traces also increases linearly.

[0004] What's more, the business is deployed globally and involves cross-region calls. To restore the true link status, it is necessary to collect tracking data generated in various regions around the world.

[0005] Of this massive amount of link data, normal link data can account for over 95%. However, the value of this normal tracking data is almost the same as the 1% of normal data, as it is only used to draw the service traffic topology. Therefore, downsampling is necessary to reduce the data size.

[0006] Currently, the following downsampling solutions are available: 1. Proportional sampling: A low sampling ratio will result in the loss of error transactions (i.e., erroneous transactions) and slow transactions, while a high sampling ratio cannot meet the requirement of reducing data size. 2. Attribute-based selective sampling: This cannot effectively handle the collection of error and slow transactions because these transactions cannot be predicted in advance, and there is no way to set attributes in advance to force their collection. 3. Tail sampling: This method requires pre-aggregating data in one place. After collecting the complete transaction link data, the decision to collect it is made based on the transaction status. This method can effectively handle error and slow transactions and can also customize the proportional sampling of normal transactions. However, it comes at the cost of large amounts of data transmission and storage, and does not solve the problem.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0008] The purpose of the present invention is to solve the technical problems existing in the background technology. To this end, a distributed tracking data downsampling processing method and system are provided.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0010] The distributed tracing data downsampling processing method includes the following steps:

[0011] Step S1: Servers in different regional networks report tracking data to the file exporter;

[0012] Step S2: The file exporter receives the reported tracking data and persists the tracking data to disk;

[0013] Step S3: The tracing agent collector continuously reads tracing data from the disk, asynchronously extracts summary information and builds indexes for the tracing data, and persists it to the disk.

[0014] Step S4: The tracking agent collector reports the tracking summary information to the collection server of the central control end, and the collection server receives the reported tracking summary information and writes it into the data storage;

[0015] Step S5: The control server at the central control end reads and processes the tracking summary information from the data storage, makes a sampling and collection decision, and then writes it back to the data storage;

[0016] Step S6: The collection server reads the sampling and collection decision data from the data storage and sends the sampling and collection decision data to the tracking agent collector;

[0017] Step S7: The tracking agent collector receives the sampling and collection decision data from the central control end, indexes the corresponding tracking data segment from the local according to the sampling and collection decision data, and reports it to the tracking data backend management platform.

[0018] The following is a technical solution further defined in the method of the present invention. In step S3, extracting summary information from the tracking data includes:

[0019] Standardize the fields of tracking data across the entire text;

[0020] Determine error transactions based on HTTP status codes and custom error markers;

[0021] Determine slow transactions based on dynamic thresholds;

[0022] Hash sampling is performed based on the adaptive proportional sampling algorithm;

[0023] Generates tracking summary information, which includes the tracking data ID and its corresponding HTTP status code, error marker hit, transaction duration or hash value.

[0024] The following is a technical solution further defined by the method of the present invention. In step S3, the tracing data is indexed, including: obtaining the tracing data ID from the tracing summary information, and constructing a one-to-many mapping relationship between the tracing data ID and the file and file line where the tracing data is located; based on the index, after the tracing agent collector receives the sampling and collection decision data, it finds the corresponding tracing data segment according to the tracing data ID.

[0025] The following is a technical solution further defined by the method of the present invention: the tracking agent collector writes the tracking summary information in the form of text lines sequentially to the local disk, and writes the index information to the local disk through the local KV storage engine, and performs IO exclusive isolation of the tracking data, tracking summary information and index information.

[0026] The following is a technical solution further limited by the method of the present invention: summary information of the same tracking data ID is stored in a directory named with the value of the tracking data ID, and the control server reads all corresponding summary information according to the tracking data ID.

[0027] The following is a technical solution further defined in the method of the present invention. In step S5, the control server reads and processes the tracking summary information from the data storage and makes a sampling and collection decision, including:

[0028] Step S51: Determine whether the data of the first two minutes of the current time has been decided;

[0029] If yes, mark the data that has been decided and end; if no, read all the tracking summary information of the corresponding time from the data storage;

[0030] Step S52: determining whether all the tracking summary information of the corresponding time read from the data storage is empty;

[0031] If yes, mark it as empty data and end; if no, use the tracking data ID to query the summary information directory data in the data storage;

[0032] Step S53: determining whether the summary information directory data can be found in the data storage using the tracking data ID;

[0033] If the query fails, mark the corresponding data and end; if the query succeeds, check the HTTP status code and error mark in the summary information directory data;

[0034] Step S54: Determine whether the HTTP status code is 0 and whether there is an error flag hit;

[0035] If the HTTP status code is 0 or the error mark is hit, it means that an error transaction is involved. Determine whether a sampling task has been created for the corresponding tracking data ID. If so, mark the corresponding data and end. If not, create a sampling task for the tracking data ID in the data storage;

[0036] If the HTTP status code is not 0 or the error flag is not hit, the transaction time in the summary information directory data is queried;

[0037] Step S55: Determine whether the transaction time exceeds the dynamic threshold;

[0038] If it exceeds, it means that a slow transaction is involved. Determine whether a sampling task has been created for the corresponding tracking data ID. If so, mark the corresponding data and end. If not, create a sampling task for the tracking data ID in the data memory;

[0039] If it does not exceed, it means that the tracking data status is normal, and query the hash value in the summary information directory data;

[0040] Step S56: Determine whether the hash value matches;

[0041] If it hits, determine whether the corresponding tracking data ID has created a sampling task. If so, mark the corresponding data and end. If not, create a sampling task for the tracking data ID in the data memory;

[0042] If there is no hit, mark the corresponding data and end.

[0043] The following is a technical solution further limited by the method of the present invention: in the data storage, the data survival time is set to 5 minutes, and the control server scans and initiates the decision process every minute.

[0044] A distributed tracing data downsampling processing system, used to implement the above-mentioned distributed tracing data downsampling processing method, includes a data collection terminal, a central control terminal, and a tracing data backend management platform;

[0045] The data collection end is deployed in each regional network and is used to collect tracking data and its summary information and index, and send the tracking summary information to the central control end. It receives the sampling and collection decision data and extracts the corresponding tracking data fragments based on the sampling and collection decision data and index and sends them to the tracking data backend management platform;

[0046] The central control end makes sampling and collection decisions based on the tracking summary information and sends the sampling and collection decisions to the data collection end;

[0047] Tracking data backend management platform, used for tracking data storage and display.

[0048] The following is a technical solution further defined by the system in the present invention, wherein the data collection end includes a file exporter and a tracking agent collector;

[0049] The file exporter is responsible for receiving the reported tracking data and persisting it to disk;

[0050] The tracking agent collector is responsible for extracting summary information and indexing the tracking data, persisting it to disk, sending the tracking summary information to the central control end, receiving the sampling and collection decisions from the central control end, and indexing the corresponding tracking data fragments locally and reporting them to the tracking data backend management platform.

[0051] The following is a technical solution further defined by the system in the present invention: the central control end includes a collection server, a data storage device and a control server;

[0052] The collection server is responsible for establishing two-way communication with the tracking agent collector, receiving the reported tracking summary information and writing it into the data storage, and sending the sampling and collection decision data to the tracking agent collector;

[0053] The control server is responsible for reading the tracking summary information from the data storage, making sampling and collection decisions, and then writing it back to the data storage.

[0054] Compared with the prior art, the present invention has the following technical effects:

[0055] The central control terminal of the present invention can collect error transactions, slow transactions with a time consumption higher than a threshold, and normal transactions in a certain proportion for tracking data of different regional networks, which can not only ensure the comprehensiveness and reliability of the downsampling collection of tracking data, but also greatly reduce the scale of data collection.

[0056] The present invention will be further described below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 It is a system connection block diagram of the present invention;

[0059] Figure 2 This is a system connection block diagram for different regional networks of the present invention;

[0060] Figure 3It is a flow chart of the control server reading and processing the tracking summary information from the data storage and making sampling and collection decisions in the present invention. DETAILED DESCRIPTION

[0061] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0062] like Figure 1-3 As shown, this embodiment provides a distributed tracing data downsampling processing method and a distributed tracing data downsampling processing system.

[0063] like Figure 1 As shown in the figure, the distributed tracking data downsampling processing system mainly includes a data collection end, a central control end, and a tracking data backend management platform.

[0064] The data collection end is deployed in each regional network and is used to collect tracking data and its summary information and index, and send the tracking summary information to the central control end, receive the sampling and collection decision data, and extract the corresponding tracking data fragment based on the sampling and collection decision data and index and send it to the tracking data backend management platform. Specifically, the data collection end includes a file exporter and a tracking agent collector. Among them, the file exporter is responsible for receiving the reported tracking data and persisting it to the disk; the tracking agent collector is responsible for extracting the summary information and indexing the tracking data, and persisting it to the disk at the same time, and then sending the tracking summary information to the central control end, and receiving the sampling and collection decision of the central control end, from the local index to the corresponding tracking data fragment and reporting it to the tracking data backend management platform.

[0065] The central control end makes sampling and collection decisions based on the tracking summary information and sends these sampling and collection decisions to the data collection end. Specifically, the central control end includes a collection server, a data storage, and a control server. The collection server is responsible for establishing two-way communication with the tracking agent collector, receiving reported tracking summary information and writing it to the data storage, and sending sampling and collection decision data to the tracking agent collector. The control server is responsible for reading the tracking summary information from the data storage, making sampling and collection decisions, and then writing the sampling and collection decisions back to the data storage.

[0066] Tracking data backend management platform, used for tracking data storage and display.

[0067] Distributed tracing data downsampling processing method, such as Figure 1As shown, the following steps are included:

[0068] Step S1: Servers in different regional intranets report tracking data to the file exporter. Each regional intranet contains multiple servers. It should be noted that regional intranets are typically divided by country, such as South Korea, China, Thailand, etc.; however, regional divisions within China, such as Southeast, Southwest, Northwest, and Northeast, are also within the scope of this invention. The number of servers is set based on the traffic volume of each region.

[0069] Step S2: The file exporter receives the reported tracking data and persists the tracking data to the disk.

[0070] Step S3: The tracing agent collector continuously reads tracing data from the disk, asynchronously extracts summary information and builds indexes for the tracing data, and persists it to the disk.

[0071] Extract summary information from tracking data, including:

[0072] Standardize the fields of tracking data across the entire text;

[0073] Determine error transactions based on HTTP status codes and custom error markers;

[0074] Determine slow transactions based on dynamic thresholds, where the dynamic thresholds can be set dynamically based on actual applications.

[0075] Hash sampling is performed based on the adaptive proportional sampling algorithm;

[0076] Generates tracking summary information, which includes the tracking data ID and its corresponding HTTP status code, error marker hit, transaction duration or hash value.

[0077] Indexing the tracing data includes: parsing the tracing summary information to obtain the tracing data ID, and building a one-to-many mapping relationship between the tracing data ID and the file and file line where the tracing data is located; based on the index, after the tracing agent collector receives the sampling and collection decision data, it finds the corresponding tracing data fragment according to the tracing data ID.

[0078] The tracing agent collector writes the tracing summary information sequentially to the local disk in the form of text lines, and writes the index information to the local disk through the local KV storage engine. The tracing data, tracing summary information, and index information are isolated from each other through IO exclusive use.

[0079] The summary information of the same tracking data ID is stored in a directory named after the tracking data ID value (including HTTP status code, error mark hit, transaction time, hash value). The control server reads all the corresponding summary information based on the tracking data ID.

[0080] Step S4: The tracking agent collector reports the tracking summary information to the collection server of the central control end. The collection server receives the reported tracking summary information and writes it into the data storage.

[0081] Step S5: The control server at the central control end reads and processes the tracking summary information from the data storage, makes a sampling and collection decision, and then writes it back to the data storage.

[0082] In the data storage, set the data lifetime to 5 minutes, and control the server to scan and initiate the decision-making process every minute, as shown in the following table:

[0083]

[0084] Among them, TTL is the survival time, which is used to limit the maximum time for data packets to be transmitted in the network.

[0085] Specifically, the control server reads and processes the trace summary information from the data storage and makes sampling and collection decisions (i.e., decision-making process), e.g. Figure 3 As shown, the decision-making process includes:

[0086] Step S51: Determine whether the data of the first two minutes of the current time has been decided;

[0087] If yes, mark the data that has been decided and end; if no, read all the tracking summary information of the corresponding time (the corresponding time is: two minutes before the current time) from the data storage;

[0088] Step S52: determining whether all the tracking summary information of the corresponding time read from the data storage is empty;

[0089] If yes, mark it as empty data and end; if no, use the tracking data ID to query the summary information directory data in the data storage;

[0090] Step S53: determining whether the summary information directory data can be found in the data storage using the tracking data ID;

[0091] If the query fails, mark the corresponding data and end; if the query succeeds, check the HTTP status code and error mark in the summary information directory data;

[0092] Step S54: Determine whether the HTTP status code is 0 and whether there is an error flag hit;

[0093] If the HTTP status code is 0 or the error mark is hit, it means that an error transaction is involved. Determine whether a sampling task has been created for the corresponding tracking data ID. If so, mark the corresponding data and end. If not, create a sampling task for the tracking data ID in the data storage (i.e., sampling collection decision);

[0094] If the HTTP status code is not 0 or the error flag is not hit, the transaction time in the summary information directory data is queried;

[0095] Step S55: Determine whether the transaction time exceeds the dynamic threshold;

[0096] If it exceeds, it means that a slow transaction is involved. Determine whether a sampling task has been created for the corresponding tracking data ID. If so, mark the corresponding data and end. If not, create a sampling task for the tracking data ID in the data storage (i.e., sampling collection decision);

[0097] If it does not exceed, it means that the tracking data status is normal, and query the hash value in the summary information directory data;

[0098] Step S56: Determine whether the hash value matches;

[0099] If a hit is found, it is determined whether a sampling task has been created for the corresponding tracking data ID. If so, the corresponding data is marked and the task ends. If not, a sampling task is created for the tracking data ID in the data memory (i.e., sampling collection decision).

[0100] If there is no hit, mark the corresponding data and end.

[0101] Step S6: The collection server reads the sampling and collection decision data from the data storage, and sends the sampling and collection decision data to the tracking agent collector.

[0102] Step S7: The tracking agent collector receives the sampling and collection decision data from the central control end, indexes the corresponding tracking data segment from the local according to the sampling and collection decision data, and reports the tracking data segment to the tracking data backend management platform.

[0103] The working process of this embodiment will be further described below:

[0104] like Figure 2As shown, the intranet in area A has multiple servers (servers A1, A2, A3...), file exporter A, disk A, and tracking agent collector A. The intranet in area A is connected to the central control end and the tracking data backend management platform. Servers A1, A2, and A3 report tracking data to file exporter A respectively. File exporter A receives the reported tracking data and persists the tracking data to disk A. Tracking agent collector A reads tracking data from disk A, extracts summary information and builds indexes for the tracking data, and persists the tracking summary information and indexes to disk A. Tracking agent collector A communicates bidirectionally with the central control end to transmit tracking summary information and sampling and collection decisions. Tracking agent collector A indexes the corresponding tracking data segments locally and reports the tracking data segments to the tracking data backend management platform.

[0105] The B-area network has multiple servers (servers B1, B2, B3, etc.), file exporter B, disk B, and tracking agent collector B. The B-area network communicates with the central control end and the tracking data backend management platform. Servers B1, B2, and B3 report tracking data to file exporter B, which receives the reported tracking data and persists it to disk B. Tracking agent collector B reads tracking data from disk B, extracts summary information and builds an index for the tracking data, and persists the tracking summary information and index to disk B. Tracking agent collector B communicates bidirectionally with the central control end to transmit tracking summary information and sampling and collection decisions. Tracking agent collector B indexes the corresponding tracking data segments locally and reports the tracking data segments to the tracking data backend management platform.

[0106] Similarly, different regional intranets can communicate with the central control terminal and the tracking data background management platform.

[0107] Therefore, the central control end can collect erroneous transactions, slow transactions with a time consumption higher than a threshold, and normal transactions in a certain proportion for tracking data collected in different regional networks. This can not only ensure the comprehensiveness and reliability of downsampling of tracking data, but also greatly reduce the scale of data collection.

[0108] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any person skilled in the art can utilize the methods and technical contents disclosed above to make many possible variations and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments with equivalent variations. Therefore, any equivalent variations made in accordance with the shape, structure, and principles of the present invention without departing from the content of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed tracing data downsampling processing method, characterized in that: The following steps are involved: Step S1: Servers in different regional networks report tracking data to the file exporter; Step S2: The file exporter receives the reported tracking data and persists the tracking data to disk; Step S3: The tracing agent collector continuously reads tracing data from the disk, asynchronously extracts summary information and builds indexes for the tracing data, and persists it to the disk. The tracing summary information includes the tracing data ID and its corresponding HTTP status code, error flag, transaction duration, or hash value. Step S4: The tracking agent collector reports the tracking summary information to the collection server of the central control end, and the collection server receives the reported tracking summary information and writes it into the data storage; Step S5: The control server at the central control end reads and processes the tracking summary information from the data storage, makes a sampling and collection decision, and then writes it back to the data storage; In step S5, the control server reads and processes the trace summary information from the data storage and makes sampling collection decisions, including: Step S51: Determine whether the data of the first two minutes of the current time has been decided; If yes, mark the data that has been decided and end; if no, read all the tracking summary information of the corresponding time from the data storage; Step S52: determining whether all the tracking summary information of the corresponding time read from the data storage is empty; If yes, mark it as empty data and end; if no, use the tracking data ID to query the summary information directory data in the data storage; Step S53: determining whether the summary information directory data can be found in the data storage using the tracking data ID; If the query fails, mark the corresponding data and end; if the query succeeds, check the HTTP status code and error mark in the summary information directory data; Step S54: Determine whether the HTTP status code is 0 and whether there is an error flag hit; If the HTTP status code is 0 or the error mark is hit, it means that an error transaction is involved. Determine whether a sampling task has been created for the corresponding tracking data ID. If so, mark the corresponding data and end. If not, create a sampling task for the tracking data ID in the data storage; If the HTTP status code is not 0 or the error flag is not hit, the transaction time in the summary information directory data is queried; Step S55: Determine whether the transaction time exceeds the dynamic threshold; If it exceeds, it means that a slow transaction is involved. Determine whether a sampling task has been created for the corresponding tracking data ID. If so, mark the corresponding data and end. If not, create a sampling task for the tracking data ID in the data memory; If it does not exceed, it means that the tracking data status is normal, and query the hash value in the summary information directory data; Step S56: Determine whether the hash value matches; If it hits, determine whether the corresponding tracking data ID has created a sampling task. If so, mark the corresponding data and end. If not, create a sampling task for the tracking data ID in the data memory; If there is no hit, mark the corresponding data and end; Step S6: The collection server reads the sampling and collection decision data from the data storage and sends the sampling and collection decision data to the tracking agent collector; Step S7: The tracking agent collector receives the sampling and collection decision data from the central control end, indexes the corresponding tracking data segment from the local according to the sampling and collection decision data, and reports it to the tracking data backend management platform.

2. The distributed tracing data downsampling processing method according to claim 1, wherein: In step S3, the tracing data is indexed, including: parsing the tracing summary information to obtain the tracing data ID, and constructing a one-to-many mapping relationship between the tracing data ID and the file and file line where the tracing data is located; based on the index, after the tracing agent collector receives the sampling and collection decision data, it finds the corresponding tracing data segment according to the tracing data ID.

3. The distributed tracing data downsampling processing method according to claim 2, wherein: The tracing agent collector writes the tracing summary information sequentially to the local disk in the form of text lines, and writes the index information to the local disk through the local KV storage engine. The tracing data, tracing summary information, and index information are isolated from each other through IO exclusive use.

4. The distributed tracing data downsampling processing method according to claim 1, wherein: The summary information of the same tracking data ID is stored in a directory named after the value of the tracking data ID. The control server reads all the corresponding summary information according to the tracking data ID.

5. The distributed tracing data downsampling processing method according to claim 1, wherein: In the data storage, the data lifetime is set to 5 minutes, and the control server scans and initiates the decision-making process every minute.

6. A distributed tracing data downsampling processing system, configured to implement the distributed tracing data downsampling processing method according to any one of claims 1 to 5, characterized in that: Including data collection terminal, central control terminal and tracking data back-end management platform; The data collection end is deployed in each regional network and is used to collect tracking data and its summary information and index, and send the tracking summary information to the central control end. It receives the sampling and collection decision data and extracts the corresponding tracking data fragments based on the sampling and collection decision data and index and sends them to the tracking data backend management platform; The central control end makes sampling and collection decisions based on the tracking summary information and sends the sampling and collection decisions to the data collection end; Tracking data backend management platform, used for tracking data storage and display.

7. The distributed tracing data downsampling processing system according to claim 6, wherein: The data collection end includes a file exporter and a tracking agent collector; The file exporter is responsible for receiving the reported tracking data and persisting it to disk; The tracking agent collector is responsible for extracting summary information and indexing the tracking data, persisting it to disk, sending the tracking summary information to the central control end, receiving the sampling and collection decisions from the central control end, and indexing the corresponding tracking data fragments locally and reporting them to the tracking data backend management platform.

8. The distributed tracing data downsampling processing system according to claim 7, wherein: The central control terminal includes a collection server, a data storage device and a control server; The collection server is responsible for establishing two-way communication with the tracking agent collector, receiving the reported tracking summary information and writing it into the data storage, and sending the sampling and collection decision data to the tracking agent collector; The control server is responsible for reading the tracking summary information from the data storage, making sampling and collection decisions, and then writing it back to the data storage.

Images