Tracking data compression method and system

Abstract

The invention discloses a tracking data compression method and system, which are used for solving the technical problem that the existing tracking data compression technology mostly depends on a universal lossless compression algorithm for data compression, so that the tracking data compression effect is limited. The method comprises the following steps: acquiring a plurality of historical tracking span data, and preprocessing the plurality of historical tracking span data to generate a plurality of historical universal fields and a plurality of historical local fields; constructing a dictionary and a span retrieval tree by adopting a Hash algorithm according to the plurality of historical universal fields and the plurality of historical local fields; and when multiple pieces of to-be-compressed tracking span data are received, compressing the multiple pieces of to-be-compressed tracking span data by adopting the dictionary and the preset separators based on the path node sequence of the span retrieval tree to generate target compressed data.

Description

Technical Field

[0001] The present invention relates to the technical field of data compression, and in particular to a tracking data compression method and system. Background Art

[0002] In modern cloud systems, distributed tracing provides a detailed end-to-end view of requests across multiple services, known as trace data. Its basic building block is the span. Each span records the execution details of a request within a single service, including attributes such as the operation name, timestamp, tags, and context. These attributes enable fine-grained correlation and analysis of the request lifecycle.

[0003] However, in cloud service systems, tracing all requests generates massive amounts of tracing span data. Since each request typically involves multiple service calls, a single request generates multiple interrelated spans, causing the amount of data to grow exponentially. For example, Google generates approximately 1,000 TB of raw tracing data per day, containing trillions of span records; Netflix processes over 2 billion requests per day, with each request generating an average of 5-10 spans, meaning tens of billions of tracing spans need to be stored and analyzed daily. This data explosion significantly increases the costs of network transmission, computational processing, and storage, posing a significant challenge to the observability architecture of distributed systems.

[0004] Existing trace data compression technologies mostly rely on general-purpose lossless compression algorithms, such as gzip (GNU Zip) and bzip2 (Burrows-Wheeler Transform Zip). While these general-purpose algorithms can efficiently process arbitrary binary sequences, they are not designed for the structured nature of trace span data and can only utilize redundant information within short sliding windows, resulting in limited compression effectiveness for trace data. Summary of the Invention

[0005] The present invention provides a tracking data compression method and system for solving the technical problem that most existing tracking data compression technologies rely on general lossless compression algorithms for data compression, resulting in limited tracking data compression effects.

[0006] A first aspect of the present invention provides a tracking data compression method, comprising:

[0007] Acquire a plurality of historical tracking span data, and pre-process the plurality of historical tracking span data to generate a plurality of historical general fields and a plurality of historical local fields;

[0008] Using a hash algorithm to construct a dictionary and a span search tree according to the plurality of historical general fields and the plurality of historical local fields;

[0009] When a plurality of tracing span data to be compressed are received, the plurality of tracing span data to be compressed are compressed using the dictionary and a preset separator based on the path node sequence of the span search tree to generate target compressed data.

[0010] Optionally, the preprocessing of the plurality of historical tracking span data to generate a plurality of historical general fields and a plurality of historical local fields includes:

[0011] Counting the values ​​corresponding to the multiple keys in each of the historical tracking span data, and outputting the total number of values ​​corresponding to each key in each of the historical tracking span data;

[0012] Comparing the total number of values ​​corresponding to each key in each of the historical tracking span data with a preset value number threshold;

[0013] The field associated with any key corresponding to the total number of values ​​less than or equal to the preset value number threshold is used as a historical general field;

[0014] Any field associated with a key corresponding to a total number of values ​​greater than the preset value quantity threshold is used as a historical local field.

[0015] Optionally, the adopting of a hash algorithm to construct a dictionary and a span search tree according to the plurality of the historical general fields and the plurality of the historical local fields includes:

[0016] Convert each of the historical general fields based on the hash algorithm, and output an identifier corresponding to each of the historical general fields;

[0017] Constructing a dictionary using each of the historical general fields, an identifier corresponding to each of the historical general fields, and a plurality of the historical local fields;

[0018] Sort the historical general fields in ascending order according to the total number of values ​​corresponding to the historical general fields to determine the insertion order;

[0019] Based on the insertion order, a span search tree is constructed using identifiers corresponding to each of the historical general fields and a plurality of the historical local fields.

[0020] Optionally, compressing the plurality of tracking span data to be compressed using the dictionary and a preset delimiter based on the path node order of the span search tree to generate target compressed data includes:

[0021] Mapping the plurality of to-be-compressed general fields in the to-be-compressed tracking span data using the dictionary to generate a plurality of identifiers corresponding to the to-be-compressed general fields;

[0022] Using the preset delimiter to connect the identifiers corresponding to the general fields to be compressed according to the order of the path nodes of the span search tree to generate a character string of path information to be compressed;

[0023] Performing hash processing on the path information character string to be compressed to generate a target path identifier;

[0024] The target path identifier and the local fields to be compressed in the plurality of tracking span data to be compressed are merged to generate target compressed data.

[0025] Optionally, the construction process of the path hash structure corresponding to the span search tree is specifically as follows:

[0026] Based on the path node sequence of the span search tree, the identifiers in the span search tree are connected using the preset separator to generate a path information character string corresponding to the span search tree;

[0027] Performing hash processing on the path information character string corresponding to the span search tree to generate a path identifier corresponding to the span search tree;

[0028] A path hash structure is constructed using the path information character string and the path identifier corresponding to the span retrieval tree.

[0029] Optionally, the decompression process of the plurality of tracking span data to be compressed is specifically as follows:

[0030] Matching a target path identifier in the target compressed data with a path identifier corresponding to the span search tree in the path hash structure;

[0031] If they are compatible, the path information character string corresponding to the path identifier compatible with the target path identifier in the path hash structure is used as the target path information character string;

[0032] Using the dictionary to restore the target path information character string to generate a plurality of general fields to be compressed;

[0033] The plurality of general fields to be compressed and the local fields to be compressed in the target compressed data are merged to generate a plurality of tracking span data to be compressed.

[0034] A second aspect of the present invention provides a tracking data compression system, comprising:

[0035] An acquisition module, configured to acquire a plurality of historical tracking span data, and pre-process the plurality of historical tracking span data to generate a plurality of historical general fields and a plurality of historical local fields;

[0036] A construction module, configured to construct a dictionary and a span search tree based on the plurality of historical general fields and the plurality of historical local fields using a hash algorithm;

[0037] The compression module is used to compress the plurality of tracing span data to be compressed using the dictionary and a preset separator based on the path node order of the span search tree when receiving the plurality of tracing span data to be compressed, so as to generate target compressed data.

[0038] A third aspect of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-described tracking data compression methods.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the steps of any one of the above-mentioned tracing data compression methods.

[0040] A fifth aspect of the present invention provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer is caused to perform the steps of the tracking data compression method as described in any one of the above items.

[0041] It can be seen from the above technical solutions that the present invention has the following advantages:

[0042] The above technical solution of the present invention provides a tracking data compression method. First, multiple historical tracking span data are obtained and preprocessed to generate multiple historical general fields and multiple historical local fields. Next, a hash algorithm is used to construct a dictionary and a span search tree based on the multiple historical general fields and multiple historical local fields. Finally, when multiple tracking span data to be compressed are received, the multiple tracking span data to be compressed are compressed using the dictionary and a preset delimiter based on the path node order of the span search tree to generate target compressed data. Based on the above method, by preprocessing the multiple historical tracking span data to distinguish between general fields and local fields, and using the hash algorithm to construct a dictionary and a span search tree based on the multiple historical general fields and multiple historical local fields, mapping relationships can be established for the general fields, effectively integrating a large number of recurring general fields, and thus achieving efficient compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A flowchart of the steps of a tracking data compression method provided in Example 1 of the present invention;

[0045] Figure 2 A schematic diagram of the workflow of the microservice link tracing system provided in Example 1 of the present invention;

[0046] Figure 3 A schematic diagram of the overall system architecture of Tracezip proposed by the present invention is provided in Example 1 of the present invention;

[0047] Figure 4 A schematic diagram of a dictionary construction process provided in the first embodiment of the present invention;

[0048] Figure 5 A schematic diagram of a span example and the SRT structure thereof provided in the first embodiment of the present invention;

[0049] Figure 6 A schematic diagram of the process of compressing tracking span data provided in the first embodiment of the present invention;

[0050] Figure 7 A schematic diagram of the SRT path hash construction process (path hash structure construction process) provided in the first embodiment of the present invention;

[0051] Figure 8 A schematic diagram of a receiving end decompression process provided in the first embodiment of the present invention;

[0052] Figure 9 Schematic diagram of the dynamic compression effect of Tracezip on different systems provided by Example 1 of the present invention;

[0053] Figure 10 This is a schematic diagram of the static compression effect of Tracezip on the Alibaba dataset provided in Example 1 of the present invention;

[0054] Figure 11 This is a schematic diagram of the compression rate (MB / s) of Tracezip on the Alibaba dataset provided in Example 1 of the present invention;

[0055] Figure 12This is a structural block diagram of a tracking data compression system provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0056] The embodiments of the present invention provide a tracking data compression method and system for solving the technical problem that most existing tracking data compression technologies rely on general lossless compression algorithms for data compression, resulting in limited tracking data compression effects.

[0057] Explanation of terms:

[0058] Cloud Service Systems: Systems built on cloud computing technology, where applications and services are loosely coupled, enabling scalability and operational flexibility. However, this also complicates software development and maintenance (such as testing, debugging, and diagnosis) due to the cascading effects of failures.

[0059] Distributed tracing: A technology used to monitor and test cloud service systems. It provides a detailed end-to-end view of requests across multiple services, known as trace data. Its basic building block is the span, which records the interactions and associated properties of requests across different machines and components, enabling correlation and analysis of the request lifecycle.

[0060] Trace Compression: By removing redundant data shared between multiple spans, spans are converted into a more concise representation to reduce the computation and storage overhead in distributed tracing, thereby lowering the transmission and storage costs of tracing data.

[0061] Trace sampling: A technique used in production environments to reduce tracing overhead. It can be categorized into head-based sampling and tail-based sampling. Head-based sampling randomly determines whether to trace a request at a low sampling rate as it enters the system. Tail-based sampling captures trace data for all requests and then decides whether to retain it based on execution details.

[0062] Span: The basic unit of distributed tracing, representing a single piece of work performed by a request on different machines and components. It contains attributes such as the span name, parent span ID, span ID, start / end timestamps, and events. This information helps to correlate and analyze the life cycle of a request.

[0063] Span Retrieval Tree (SRT): This invention uses a prefix tree-based data structure to continuously retrieve key-value pairs shared across multiple spans on the server side. Through a synchronization mechanism, the backend can use this structure to retrieve the same data transmitted by previous spans, reducing data redundancy and transmission volume.

[0064] Zero-code Instrumentation: OpenTelemetry provides a capability that uses libraries, plugins, or agents to detect the libraries used by an application, allowing the collection of observability data (including logs, metrics, and tracing) from the application without modifying the source code. It supports multiple programming languages ​​and popular libraries and frameworks.

[0065] Key-Value Pair (KV Pair): The basic unit of information representation in tracking data, consisting of a key (string) and a value (primitive type or structured type).

[0066] Universal Fields: In the Span Retrieval Tree (SRT), the key-value pairs associated with non-leaf nodes (except the root node). These fields are more repetitive in multiple spans and are part of the SRT that can be compressed.

[0067] Local Fields: Unique key-value pairs in a span that are not shared with other spans, such as fields related to ID and time information. They are usually parts that cannot be compressed by SRT. Their key values ​​are stored in leaf nodes and reconstructed based on the leaf node keys during back-end decompression.

[0068] Compression Ratio (CR): An indicator used to measure the effectiveness of a compression method. It is defined as the ratio of the original data size to the compressed data size. A higher ratio indicates better compression performance.

[0069] Hashing: This technology used in the present invention to accelerate path search converts each unique path in the SRT into a path identifier and quickly finds the path corresponding to the span by maintaining a mapping relationship between the path and the identifier.

[0070] Differential Update Mechanism: This invention is used to efficiently synchronize SRT and dictionary structures between the server and the backend, only propagating incremental changes in the data structure, ensuring data consistency while minimizing synchronization overhead.

[0071] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0072] See also Figure 1 , Figure 1 This is a flowchart of the steps of a tracking data compression method provided in Example 1 of the present invention.

[0073] The present invention provides a tracking data compression method, comprising:

[0074] Step 101: Acquire multiple historical tracking span data, and pre-process the multiple historical tracking span data to generate multiple historical general fields and multiple historical local fields.

[0075] Please note that Figure 2 , Figure 2 The following figure illustrates a typical distributed system tracing process. When a new request arrives at an application, the tracing framework assigns it a unique identifier, trace_id (①). However, not all requests are traced; the system determines whether to trace a request by setting a sampled flag. The trace_id and sampled flag are passed between applications along with the request, which is crucial for the integrity and consistency of the tracing. If sampled is set, each component that handles the request (e.g., a microservice instance) generates tracing data (②), or a span, using the tracing framework's client library (e.g., OpenTelemetry). The location where a span is generated is called a tracepoint. A request or different operations within a single request may have multiple tracepoints serving different requests. The framework's client library then queues, serializes, and transmits (③) the tracing spans to its centralized backend server for collection. The backend is responsible for continuously receiving (④), processing (⑤), and storing (⑥) the tracing data. Based on trace_id, parent_span_id, and span_id, the backend can assemble spans scattered across different components into a unified coherent trace.

[0076] Further, see Figure 3This paper proposes a tracing data compression method, and its corresponding system architecture is called Tracezip. This paper adds an online compression module to the traditional distributed system tracing framework to improve the efficiency of tracing by reducing the overhead of span transmission. The system architecture is as follows Figure 3 As shown, the present invention adds a compression module (⑦) on the server side and adds a corresponding decompression module (⑧) at the entry of the backend trace data collector. In the compression module, the present invention maintains two data structures, namely the span retrieval tree (SRT) and a dictionary, to dynamically capture the redundancy between spans. When a tracepoint generates a span, it uses the above structure for compression. In the decompression module, Tracezip restores the spans to their original form based on the SRT and the dictionary. In order to ensure consistent and reliable data transmission, it is crucial to accurately synchronize these data structures between the server and the backend. To achieve this, the present invention also develops a differential update mechanism (⑨). This mechanism is designed to accurately locate and propagate incremental changes in data structures, ensuring an efficient synchronization process while minimizing overhead and maximizing data consistency.

[0077] Specifically, step 101 may include the following sub-steps S11-S14:

[0078] Step S11: Count the values ​​corresponding to multiple keys in each historical tracking span data, and output the total number of values ​​corresponding to each key in each historical tracking span data;

[0079] Step S12: comparing the total number of values ​​corresponding to each key in each historical tracking span data with a preset value number threshold;

[0080] Step S13: any key-associated field corresponding to the total number of values ​​that is less than or equal to a preset value threshold is used as a historical general field;

[0081] Step S14: any key-associated field corresponding to the total number of values ​​greater than a preset value threshold is used as a historical local field.

[0082] It should be noted that in order to distinguish keys of different natures, a threshold is set in advance, namely the preset value quantity threshold ( ). This threshold plays a key role in division, and the keys are divided into two categories based on the number of values ​​each key has (the total number of values ​​corresponding to each key) compared with the threshold. Among them, Universal Fields: If the number of values ​​a key has is less than or equal to the threshold, then the field corresponding to this key is identified as a universal field. Universal fields mean that they are relatively stable and recurring in multiple spans, that is, the changes in their values ​​are relatively limited, and they have a high commonality. Therefore, universal fields are the focus of compression. Local Fields: On the contrary, when the number of values ​​a key has exceeds the set threshold, the field corresponding to the key will be regarded as a local field. The characteristic of local fields is that their values ​​vary greatly in different spans, the diversity is more obvious, and they are relatively unique parts of each span. The present invention does not compress local fields.

[0083] Step 102: Use a hash algorithm to construct a dictionary and a span search tree based on multiple historical general fields and multiple historical local fields.

[0084] Specifically, step 102 may include the following sub-steps S21-S24:

[0085] Step S21: Convert each historical general field based on a hash algorithm and output an identifier corresponding to each historical general field;

[0086] Step S22: construct a dictionary using each historical general field, an identifier corresponding to each historical general field, and multiple historical local fields;

[0087] Step S23: sort each historical general field in ascending order according to the total number of values ​​corresponding to each historical general field to determine the insertion order;

[0088] Step S24: Based on the insertion order, a span search tree is constructed using the identifiers corresponding to each historical general field and multiple historical local fields.

[0089] The insertion order is determined by sorting the historical common fields in ascending order when constructing the Span Retrieval Tree (SRT). This order directly affects the tree's hierarchical structure and compression efficiency.

[0090] Please note that Figure 4When a batch of spans is generated, the first step is to scan all key-value pairs contained within each span and calculate the frequency of each key's corresponding value and the number of values ​​for each key. After completing these steps, the dictionary is constructed. The core function of the dictionary is to create a mapping mechanism for common fields, mapping common fields to shorter identifiers. This involves converting the originally long common field names or their corresponding contents into shorter identifiers using a hashing algorithm.

[0091] Furthermore, after completing the distinction between general fields and local fields, the general fields need to be sorted. The specific sorting rule is to arrange the general fields in ascending order according to the number of values ​​they have. This is done to determine a reasonable insertion order when constructing the span search tree (SRT) later, thereby improving the compression rate. At the same time, the general fields (the number of values ​​is less than or equal to the threshold) have been distinguished during the dictionary construction process. fields) and local fields (the number of values ​​exceeds the threshold The insertion order of the span search tree is created by sorting the common fields in ascending order according to their values, and a mapping relationship to a shorter identifier is created for the common fields.

[0092] Furthermore, if Figure 5 As shown, Figure 5 A set of span samples and their corresponding SRT structures are shown. If a span is subsequently produced with a new redundancy pattern, it will be seamlessly integrated into the SRT and dictionary. This integration is crucial because it adjusts the above two data structures to support the compression of subsequent spans. The present invention accelerates the above process by using mapping and hashing techniques. At the same time, it also shows how a batch of spans are organized into an SRT structure. The part of a batch of spans that is identified as a common field is moved to the non-leaf node part of the SRT, and a span can be restored by traversing from top to bottom along a path of the SRT. Figure 5 Taking (a) as an example, the key operation has two different values ​​WRITE and READ, and the key span_id has a different value for each row.

[0093] It's worth noting that the hash functions and dictionary construction algorithms in Tracezip are interchangeable. In Tracezip, the primary function of the hash function is to convert the paths in the Span Retrieval Tree (SRT) into uniquely identifying hash values, enabling rapid location and retrieval of path information at the output. However, there's no single hash function that implements this function; a variety of alternatives exist. For example, in addition to the common SHA-256 algorithm, MD5, SHA-1, and CRC32 can also be used. These algorithms all convert input data (such as path information in the SRT) into fixed-length hash values. While they differ in hash value length, computational complexity, and security, they all share some relevance in fulfilling the basic functions of path identification and retrieval.

[0094] Furthermore, the dictionary construction algorithm in Tracezip is primarily used to distinguish between universal and local fields and establish mappings for universal fields to achieve data compression and efficient processing. However, this process is not unique; there are multiple feasible approaches to constructing this dictionary construction algorithm. For example, a dictionary construction algorithm based on frequency statistics can be used. This algorithm counts the frequency of each key-value pair across all spans, treating key-value pairs with higher frequency as universal fields and those with lower frequency as local fields. This method classifies field types based on the actual frequency of key-value pairs in the data, more directly reflecting the distribution characteristics of the data.

[0095] Step 103 : When a plurality of tracking span data to be compressed are received, the plurality of tracking span data to be compressed are compressed based on the path node sequence of the span search tree using a dictionary and a preset separator to generate target compressed data.

[0096] The tracking span data to be compressed is tracking span data (tracking data) that needs to be compressed.

[0097] The target compression data includes a local field (local field value) and a target path identifier.

[0098] Please note that Figure 6, based on the dictionary that has been constructed, for all common fields in the tracking span data (i.e., span data), they are converted one by one into predetermined short identifiers. For example, if a common field was originally a more complex name or a statement of a certain length, it is transformed into a concise identifier through the mapping rules of the dictionary. Specifically, using the previously determined common field insertion order (i.e., the order in which the optional values ​​of the common fields are sorted from small to large), the common fields converted into identifiers are inserted into the SRT in sequence. Starting from the root node of the SRT, the identifier nodes corresponding to these common fields are gradually inserted downward according to a certain hierarchical structure to build the main framework structure of the SRT. These non-leaf nodes carry the relevant information of the common fields.

[0099] For those parts that are determined to be local fields, since their values ​​are highly unique, they are not suitable to be integrated into the main structure of the tree for unified processing like general fields. Therefore, local fields are directly stored in the leaf nodes of SRT, such as Figure 5 When you need to obtain the complete information of a span or perform decompression and restoration operations, you can obtain the specific content corresponding to these local fields by traversing to the leaf node, and then restore the complete span original data in conjunction with the general fields.

[0100] Specifically, step 103 may include the following sub-steps S31-S34:

[0101] Step S31: Use a dictionary to map the common fields to be compressed in the plurality of tracking span data to be compressed, and generate identifiers corresponding to the common fields to be compressed;

[0102] Step S32: using a preset delimiter to connect the identifiers corresponding to the general fields to be compressed in the order of the path nodes of the span search tree to generate a character string of path information to be compressed;

[0103] Step S33: Hash the path information character string to be compressed to generate a target path identifier;

[0104] Step S34: Merge the target path identifier and the local fields to be compressed in the plurality of tracking span data to be compressed to generate target compressed data.

[0105] The path node order of a span search tree is the order in which the common field nodes are arranged along the path from the root node to the leaf nodes in the span search tree (SRT). This order is used to locate the hierarchical relationship of fields during compression and decompression. The path node order is determined by the insertion order: fields are inserted into the tree in ascending order of the number of values, and the path node order reflects this insertion order.

[0106] It should be noted that when a new span is generated at the information output end, the value of its common information is extracted according to the order in the span retrieval tree (the order of the path nodes in the span retrieval tree), and then the path search for this span is quickly completed by checking whether the path information of the span exists in the corresponding relationship structure.

[0107] Assume there is a simple SRT structure used to store span information related to web page requests. The tree contains some common information as non-leaf nodes. For example, assume there is a span type WebRequest, as shown in Table 1, which contains the following information:

[0108] Table 1 Span WebRequest

[0109]

[0110] The key-value pairs of the above assumed spans are shown in Table 2:

[0111] Table 2 Key-value pairs of span WebRequest

[0112]

[0113] Furthermore, it is identified on the SRT link as (token_1 => token_2 => token_3 => token_4). Using the hash algorithm to convert paths into identifiers, starting from the starting node (i.e., the root node "WebRequest"), a comma, a pre-set delimiter (similar to the delimiter in the CSV standard format), is used to concatenate the hash values ​​of the contents of the non-terminal nodes (i.e., the aforementioned "RequestType," "Protocol," and "Server" general information nodes) in the insertion order of the span retrieval tree. The resulting path information string is: "token_1, token_2, token_3, token_4." A hash algorithm is then used to generate an identifier with path characteristics based on this path information string. Assume that the identifier obtained by the hash algorithm is "Hash_001" (in practice, it will be a specific string or encoding generated based on the specific hash algorithm):

[0114] ;

[0115] in, , , , Represents the original values ​​of the 1st, 2nd, ... to the nth general fields arranged in the order of path insertion (the order of path nodes in the span search tree); the present invention maps the original value to a shorter identifier through the dictionary constructed before the algorithm, map() is a dictionary mapping operation for field values, concat() is a string concatenation operation, concatenates the mapped identifiers, and maps the concatenated path to a path hash value through the hash() operation, and finally obtains is the compressed path identifier, that is, the target path identifier.

[0116] Next, the information exporter maintains a structure containing the correspondence between paths and identifiers. The information exporter converts the SRT structure into a path hash structure in real time and streams the compressed tracking span data (target compressed data) to the receiver.

[0117] Optionally, the construction process of the path hash structure corresponding to the span retrieval tree is as follows:

[0118] Based on the path node sequence of the span search tree, the identifiers in the span search tree are connected using a preset separator to generate a path information string corresponding to the span search tree;

[0119] Performing hash processing on the path information character string corresponding to the span search tree to generate a path identifier corresponding to the span search tree;

[0120] The path information string and path identifier corresponding to the span retrieval tree are used to construct a path hash structure.

[0121] Please note that Figure 7 Before sending SRT, a hash algorithm is used to convert each unique path in the span search tree into a unique path identifier. The specific operation process is as follows Figure 7 As shown: for each path of the span retrieval tree, starting from the starting node, based on a specific separator symbol (preset separator), the key values ​​of the non-terminal nodes are connected in order (the order of the path nodes of the span retrieval tree) through the dictionary mapping identifier to form path information (path information string), and the path information string is hashed using a hash algorithm to generate a path identifier. The path information string and the path identifier are used to construct and maintain a structure containing the correspondence between the path and the identifier (i.e., the path hash structure) at the information output end.

[0122] Optionally, the decompression process of multiple tracking span data to be compressed is as follows:

[0123] Matching a target path identifier in the target compressed data with a path identifier corresponding to a span search tree in the path hash structure;

[0124] If they are compatible, the path information string corresponding to the path identifier compatible with the target path identifier in the path hash structure is used as the target path information string;

[0125] Use a dictionary to restore the target path information string to generate multiple common fields to be compressed;

[0126] The plurality of general fields to be compressed and the local fields to be compressed in the target compressed data are merged to generate a plurality of tracking span data to be compressed.

[0127] Please note that Figure 8 During decompression, the backend decompression module first receives the target compressed data sent by the server, which includes the target path identifier (such as "Hash_001") and the compressed data of possible local field values. Then, in the path and identifier correspondence structure (path hash structure) maintained by the information output end, it quickly searches and locates the corresponding path string (such as "token_1, token_2, token_3, token_4") based on the received path identifier. It then parses the content of the common information node and determines the common field values ​​according to the path order (such as "token_2" for "RequestType", "token_3" for "Protocol", and "token_4" for "Server"). It then uses the dictionary to restore the actual values ​​of the common fields. If there are local field values, they are processed according to pre-defined rules (such as time offset and time base calculation). Finally, the common fields and the processed local fields (if any) are merged to obtain the tracking span data to be compressed before compression, thereby completing the reconstruction of the original span and restoring it to the state before compression that contains all the original information.

[0128] It's worth noting that to maximize Tracezip's compression ratio, both the output and receiver must maintain consistent copies of the span search tree and dictionary structure. Since simply sending the latest version of the entire structure with each update would incur additional transmission costs and delay decompression, Tracezip implements an incremental update mechanism. By default, the sender no longer performs a full dictionary build for each compression; instead, it performs incremental updates based on the previous dictionary. If the sender has previously sent a span and the current transmission does not trigger an SRT reconstruction, it only sends the newly added dictionary data. On the receiver, a structure containing the mappings between identifiers and paths is maintained in the reverse direction. For any update to a non-terminal node, the affected paths can be easily identified and updated. For example, when a new path is generated, a new entry is added to the output's mapping structure and synchronized with the receiver. For path deletions, the output sends the corresponding identifier to the receiver to record the deletion. Other updates are treated as a combination of path addition and path deletion operations, resulting in more resource-efficient synchronization. Furthermore, when the dictionary itself becomes too large, or when the selectable values ​​of some fields that were originally considered general fields exceed a set threshold, this mechanism is implemented. , it will trigger the SRT reconstruction operation. At this time, the system will abolish the original SRT dictionary and re-execute the algorithm's dictionary construction process.

[0129] For comparison purposes, existing technologies can be used as a reference. Distributed tracing in cloud service systems faces the following challenges: High tracing overhead in cloud service systems: Applications and services in cloud service systems are loosely coupled, and the cascading effects of failures complicate testing, debugging, and diagnosis. While important, distributed tracing generates large amounts of tracing data, resulting in high network, compute, and storage costs. To reduce this overhead, existing tracing frameworks often employ header-based sampling, which only traces a small number of requests and can easily miss edge-case system behavior. Tail-based sampling, while tracking all requests, requires filtering based on execution details, resulting in significant cost. Compression technology has limitations: Existing log compression is performed on offline data on the server backend, primarily to save storage space. The algorithms are complex and cannot meet the requirements of real-time tracing collection. Furthermore, they often require processing the entire dataset to achieve optimal performance, making them unsuitable for online tracing scenarios. While general-purpose compression algorithms (such as gzip and bzip) can be used for compression, they are designed to encode arbitrary binary sequences and can only exploit redundant information within a short sliding window, resulting in limited compression effectiveness for tracing data.

[0130] To reduce these overheads, existing tracing frameworks typically use tracing sampling techniques, which are mainly divided into head-based sampling and tail-based sampling. Head-based sampling randomly selects requests for tracing at a low sampling rate (such as 1%) when requests enter the system. The sampling decision is made before the request is executed, and only the sampled requests are traced. This method may miss the edge case behavior of the system. Tail-based sampling captures the tracing data of all requests, and then decides whether to retain the trace based on the execution details (such as latency, HTTP status code). Many machine learning techniques are applied to this, but because the true value of the tracing data is inherently unpredictable, this sampling method is difficult to capture all the key information required for effective software testing and fault diagnosis, and it incurs huge costs because all requests must be traced and the tracing data must be ingested to make sampling decisions.

[0131] Furthermore, when applying a parser-based compression method to production logs, poor performance was observed. Analysis of the reasons included suboptimal implementation and assumptions that were unsuitable for large-scale logs. To address the large proportion of numerical values ​​in space that is poorly handled by classic compression algorithms, techniques such as delta timestamps, correlation, identification, and elastic encoding were proposed. Furthermore, through empirical research, observations of log data characteristics were obtained, and the LogShrink compression method was proposed, which exploits the commonalities and variability of log data. An analyzer based on the longest common subsequence and entropy techniques was used to identify potential commonalities and variability, and a clustering-based sequence sampler was introduced to accelerate analysis. In addition, a divide-and-conquer preprocessing strategy was adopted to improve compatibility with subsequent general dictionary-based compressors.

[0132] Furthermore, general-purpose compression algorithms, such as gzip, are designed to encode arbitrary binary sequences. For example, the core Deflate algorithm in gzip has certain characteristics and limitations when processing text data. This algorithm primarily relies on a relatively short sliding window to mine redundant information in the data.

[0133] However, general-purpose compression algorithms such as gzip are designed for encoding arbitrary binary sequences. While they can reduce data size in some scenarios, their compression effectiveness is limited for distributed tracing data, which has specific structures and redundancy patterns. They cannot fully utilize the global redundancy in the data, and even struggle to meet the real-time requirements of distributed tracing scenarios. Offline log compression methods often involve complex algorithms and cannot meet the real-time requirements of tracing collection. Furthermore, they typically require processing the entire log dataset to achieve optimal performance. Tracing data collection services for distributed systems require a lightweight compression algorithm that can be generated in a streaming manner. Tracezip leverages the streaming nature and semantic redundancy of tracing data collection services to implement a lightweight online compression algorithm.

[0134] In response to the above issues, this paper proposes a tracing data compression method. By designing a new data structure (Span Retrieval Tree (SRT)) and a series of optimization strategies (including SRT construction, optimization, and synchronization mechanisms), this method leverages redundant information in tracing data for online and efficient tracing compression. This method enhances the ability to monitor edge case behavior without increasing transmission overhead, improves the efficiency of distributed tracing, and enables efficient system monitoring. Specifically, this paper designs a new data structure, the Span Retrieval Tree (SRT), based on the prefix tree principle. It effectively extracts sets of key-value pairs shared across multiple spans, represents common fields in the form of paths, and accurately captures redundancy in distributed tracing data structures, thus laying the foundation for efficient compression. Furthermore, a hashing algorithm is used to convert each unique path in the SRT into an identifier with path characteristics. By maintaining a path-identifier correspondence structure at the information output, path searches can be quickly completed when generating new spans, effectively solving the path search speed issues associated with chain storage structures and improving overall system performance.

[0135] Furthermore, the proposed Tracezip solution leverages global redundancy in distributed tracing data. Through its designed SRT data structure, it continuously extracts sets of key-value pairs shared across multiple spans. This approach effectively consolidates numerous recurring common fields, significantly reducing data size and achieving higher compression ratios. For example, when processing multiple spans containing similar operations (such as database accesses or service calls of the same type), SRT can represent common components of these operations (such as common fields like database connection parameters and service names) as concise paths, significantly reducing data redundancy. Experiments show that, as a standalone solution, Tracezip achieves compression ratios of 3.94 to 6.46 when processing trace data from open source cloud systems. Furthermore, Tracezip utilizes a series of optimization strategies, such as SRT reconstruction and map-based tree compression, to effectively control data structure size and avoid excessive memory usage. On the server side, it maintains only necessary data structures (such as SRT, hash paths, and dictionary mappings), and can adjust thresholds based on actual conditions to balance compression performance and memory usage. Experiments show that even when processing Alibaba trace data with diverse spans, by properly setting thresholds, Tracezip can achieve significant compression improvements (such as a 33.8% performance boost) while keeping memory usage at a low level (e.g., 2.56MB). This ensures that the system runs efficiently without affecting normal service execution and avoids system performance degradation or resource exhaustion caused by the data compression process.

[0136] Further, Figure 9-10This article demonstrates the compression performance of Tracezip on various systems. It demonstrates that Tracezip achieves high compression ratios when processing trace data from various systems. For example, when processing trace data from open source cloud systems (such as the Train Ticket microservices benchmark, gRPC, and Servlet), compression ratios range from 3.94 to 6.46. When combined with traditional general-purpose compression algorithms (gzip, bzip2, and lzma), Tracezip further improves compression performance. In tests across multiple open source cloud systems, combining Tracezip with gzip achieved performance improvements of 10% to 45%, with the specific improvement varying depending on the system. For example, in the Train Ticket benchmark, the compression ratio of Tracezip combined with gzip increased from 10.90 (using gzip alone) to 16.26, an improvement of approximately 33.0%. When compressing the 26.15GB Alibaba dataset, combining Tracezip with lzma increased the compression ratio from 7.3 to 10.48, a 43.6% improvement. This shows that Tracezip can take advantage of additional compression when used with existing compression tools to better handle different types of distributed tracing data.

[0137] Furthermore, Tracezip's core operations have a low time complexity. The time complexity of constructing and reconstructing the span search tree (SRT) is O(m), where m is the number of span attributes. In practical scenarios, m is usually small (e.g., less than 20). Other key operations, such as hashing, dictionary mapping, and path searching, all have a time complexity of O(1). This means that the execution time of these operations is not significantly affected by the size of the data and can be completed in constant time. Overall, Tracezip's time complexity is linear. This efficient time complexity feature enables the system to maintain high performance when processing large amounts of trace data, without causing exponential processing time increases due to the increase in data volume, ensuring the stability and responsiveness of the system under high load conditions.

[0138] Further, if Figure 11 As shown in the figure, Tracezip performs well in terms of processing speed and can meet the real-time requirements of distributed tracing systems. =1000, its processing speed can reach 78.92MB / s. In comparison, the fastest traditional algorithm, gzip, processes at only approximately 12.5MB / s. This is primarily due to Tracezip's ability to identify and bypass incompressible attributes (such as local fields like trace IDs, span IDs, and authentication tokens), focusing on compressing redundant common fields. This reduces unnecessary computational overhead and improves data processing efficiency. At the same time, through reasonable data structures (such as the use of efficient hashing algorithms to accelerate path searches and optimized span retrieval tree structures) and algorithm design, the compression and decompression processes can be carried out quickly, ensuring that the system can promptly process continuously generated tracing data and achieve efficient distributed tracing.

[0139] In an embodiment of the present invention, a tracking data compression method is provided. First, multiple historical tracking span data are obtained and preprocessed to generate multiple historical general fields and multiple historical local fields. Next, a hash algorithm is used to construct a dictionary and a span search tree based on the multiple historical general fields and multiple historical local fields. Finally, upon receiving multiple tracking span data to be compressed, the multiple tracking span data to be compressed are compressed using the dictionary and a preset delimiter based on the order of the path nodes in the span search tree to generate target compressed data. Based on the above scheme, by preprocessing the obtained multiple historical tracking span data to distinguish between general fields and local fields, and then using the hash algorithm to construct a dictionary and a span search tree based on the multiple historical general fields and multiple historical local fields, mapping relationships can be established for the general fields, effectively integrating a large number of recurring general fields, and thus achieving efficient compression.

[0140] See also Figure 12 , Figure 12 This is a structural block diagram of a tracking data compression system provided in Example 2 of the present invention.

[0141] The present invention provides a tracking data compression system, comprising:

[0142] The acquisition module 1201 is used to acquire a plurality of historical tracking span data, and pre-process the plurality of historical tracking span data to generate a plurality of historical general fields and a plurality of historical local fields;

[0143] A construction module 1202 is configured to construct a dictionary and a span search tree based on multiple historical general fields and multiple historical local fields using a hash algorithm;

[0144] The compression module 1203 is configured to compress the plurality of tracing span data to be compressed using a dictionary and a preset separator based on the path node order of the span search tree when receiving the plurality of tracing span data to be compressed, and generate target compressed data.

[0145] Furthermore, the acquisition module 1201 is specifically configured to:

[0146] Count the values ​​corresponding to multiple keys in each historical tracking span data, and output the total number of values ​​corresponding to each key in each historical tracking span data;

[0147] Compare the total number of values ​​corresponding to each key in each historical tracking span data with the preset value number threshold;

[0148] The field associated with any key whose total number of values ​​is less than or equal to the preset value threshold is used as a historical general field;

[0149] The field associated with any key corresponding to the total number of values ​​greater than a preset value threshold is used as a historical local field.

[0150] Furthermore, the construction module 1202 is specifically configured to:

[0151] Based on the hash algorithm, each historical general field is converted and the identifier corresponding to each historical general field is output;

[0152] A dictionary is constructed using each historical general field, an identifier corresponding to each historical general field, and multiple historical local fields;

[0153] Sort each historical general field in ascending order according to the total number of values ​​corresponding to each historical general field to determine the insertion order;

[0154] Based on the insertion order, the span retrieval tree is constructed using the identifiers corresponding to each historical general field and multiple historical local fields.

[0155] Furthermore, the compression module 1203 is specifically configured to:

[0156] A dictionary is used to map the common fields to be compressed in the plurality of tracking span data to be compressed, and identifiers corresponding to the plurality of common fields to be compressed are generated;

[0157] Connecting the identifiers corresponding to the general fields to be compressed according to the order of the path nodes of the span search tree using a preset delimiter to generate a character string of path information to be compressed;

[0158] Hash the path information string to be compressed to generate a target path identifier;

[0159] The target path identifier and the local fields to be compressed in the plurality of tracking span data to be compressed are merged to generate target compressed data.

[0160] In an optional system embodiment, the system further includes:

[0161] The first module is used to connect the identifiers in the span search tree using a preset separator based on the path node sequence of the span search tree to generate a path information character string corresponding to the span search tree;

[0162] The second module is used to perform hash processing on the path information character string corresponding to the span search tree to generate a path identifier corresponding to the span search tree;

[0163] The third module is used to construct a path hash structure by using the path information character string and the path identifier corresponding to the span retrieval tree.

[0164] In an optional system embodiment, the system further includes:

[0165] a fourth module, configured to match a target path identifier in the target compressed data with a path identifier corresponding to a span search tree in the path hash structure;

[0166] A fifth module is configured to, if the path information string is matched, use the path information string corresponding to the path identifier matched with the target path identifier in the path hash structure as the target path information string;

[0167] The sixth module is used to restore the target path information string using a dictionary to generate multiple common fields to be compressed;

[0168] The seventh module is used to merge multiple common fields to be compressed and local fields to be compressed in the target compressed data to generate multiple tracking span data to be compressed.

[0169] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0170] An embodiment of the present invention further provides a computer device including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the tracking data compression method of any of the above embodiments.

[0171] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the tracking data compression method according to any of the above embodiments are implemented.

[0172] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the trace data compression method of any of the above embodiments when the computer program / instruction is executed by a processor.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0175] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A tracking data compression method, characterized in that: include: Acquire a plurality of historical tracking span data, and pre-process the plurality of historical tracking span data to generate a plurality of historical general fields and a plurality of historical local fields; Using a hash algorithm to construct a dictionary and a span search tree according to the plurality of historical general fields and the plurality of historical local fields; When a plurality of tracing span data to be compressed are received, the plurality of tracing span data to be compressed are compressed using the dictionary and a preset separator based on the path node sequence of the span search tree to generate target compressed data.

2. The tracking data compression method according to claim 1, wherein: The preprocessing of the plurality of historical tracking span data to generate a plurality of historical general fields and a plurality of historical local fields includes: Counting the values ​​corresponding to the multiple keys in each of the historical tracking span data, and outputting the total number of values ​​corresponding to each key in each of the historical tracking span data; Comparing the total number of values ​​corresponding to each key in each of the historical tracking span data with a preset value number threshold; The field associated with any key corresponding to the total number of values ​​less than or equal to the preset value number threshold is used as a historical general field; Any field associated with a key corresponding to a total number of values ​​greater than the preset value quantity threshold is used as a historical local field.

3. The tracking data compression method according to claim 1, wherein: The method of using a hash algorithm to construct a dictionary and a span search tree based on the plurality of historical general fields and the plurality of historical local fields includes: Convert each of the historical general fields based on the hash algorithm, and output an identifier corresponding to each of the historical general fields; Constructing a dictionary using each of the historical general fields, an identifier corresponding to each of the historical general fields, and a plurality of the historical local fields; Sort each of the historical general fields in ascending order according to the total number of values ​​corresponding to each of the historical general fields to determine an insertion order; Based on the insertion order, a span search tree is constructed using identifiers corresponding to each of the historical general fields and a plurality of the historical local fields.

4. The tracking data compression method according to claim 1, wherein: The method of compressing the plurality of tracking span data to be compressed using the dictionary and a preset delimiter based on the path node sequence of the span search tree to generate target compressed data includes: Mapping the plurality of to-be-compressed general fields in the to-be-compressed tracking span data using the dictionary to generate a plurality of identifiers corresponding to the to-be-compressed general fields; Using the preset delimiter to connect the identifiers corresponding to the general fields to be compressed according to the order of the path nodes of the span search tree to generate a character string of path information to be compressed; Performing hash processing on the path information character string to be compressed to generate a target path identifier; The target path identifier and the local fields to be compressed in the plurality of tracking span data to be compressed are merged to generate target compressed data.

5. The tracking data compression method according to claim 4, characterized in that: The construction process of the path hash structure corresponding to the span search tree is specifically as follows: Based on the path node sequence of the span search tree, the identifiers in the span search tree are connected using the preset separator to generate a path information character string corresponding to the span search tree; Performing hash processing on the path information character string corresponding to the span search tree to generate a path identifier corresponding to the span search tree; A path hash structure is constructed using the path information character string and the path identifier corresponding to the span retrieval tree.

6. The tracking data compression method according to claim 5, characterized in that: The decompression process of the plurality of tracking span data to be compressed is specifically as follows: Matching a target path identifier in the target compressed data with a path identifier corresponding to the span search tree in the path hash structure; If they are compatible, the path information character string corresponding to the path identifier compatible with the target path identifier in the path hash structure is used as the target path information character string; Using the dictionary to restore the target path information character string to generate a plurality of general fields to be compressed; The plurality of general fields to be compressed and the local fields to be compressed in the target compressed data are merged to generate a plurality of tracking span data to be compressed.

7. A tracking data compression system, characterized in that: include: An acquisition module, configured to acquire a plurality of historical tracking span data, and pre-process the plurality of historical tracking span data to generate a plurality of historical general fields and a plurality of historical local fields; A construction module, configured to construct a dictionary and a span search tree based on the plurality of historical general fields and the plurality of historical local fields using a hash algorithm; The compression module is used to compress the plurality of tracing span data to be compressed using the dictionary and a preset separator based on the path node order of the span search tree when receiving the plurality of tracing span data to be compressed, so as to generate target compressed data.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the tracking data compression method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the tracking data compression method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer is caused to perform the trace data compression method according to any one of claims 1 to 6.

Images