A method and system for call chain data compression based on information redundancy

Abstract

The application discloses a kind of based on information redundancy's call chain data compression method and system, method includes: for the call chain data collected, similar data is extracted using KMeans algorithm, the data point in the call chain data is grouped, and each cluster group is obtained;Wherein, the data similarity in the same cluster group is high, and the data similarity between different cluster groups is low;Using the offline tail sampling strategy based on hierarchical clustering, the cluster group is sampled and handled, and a structured file is output;The structured file is losslessly compressed using a lossless compression method, and the compression of the call chain data is completed.The application is based on the characteristics of call chain data, combined with the sampling idea of lossy compression, greatly reduces the size and storage burden of the compressed file on the basis of maintaining data analysis value, improves efficiency and real-time performance, can guarantee data integrity and strengthen generalization ability, and can be widely applied in computer technology field.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a call chain data compression method and system based on information redundancy. Background Art

[0002] In today's distributed systems, a single user request may trigger a sequence of operations across multiple services, often distributed across servers or compute nodes. This complex service interaction is meticulously recorded in call chain data, or trace data, which details each service call's timestamp, execution path, response time, and any errors encountered. Collectively, this data forms a comprehensive "trace" of the request processing, providing a transparent view of the entire process from request initiation to completion.

[0003] As correctness and performance issues in distributed systems have become increasingly prominent over the past decade, end-to-end tracing has become a critical tool for addressing these challenges. This technology not only accurately models system load, resource usage, and time series, but also detects abnormal requests in real time, providing a solid foundation for system stability monitoring and fault diagnosis. Its importance has been recognized across the industry, with standardization work underway and adoption across multiple enterprises and development frameworks. In today's evolving technology ecosystem, the consistent view provided by end-to-end tracing is key to gaining insight into the complex interactions of systems across various components, layers, machines, and management domains.

[0004] However, with the expansion of system scale and the prevalence of microservice architectures, recording call chain data has become both more critical and more challenging. This data is not only essential for continuous system operation, rapid fault location, and performance monitoring, but also, as system complexity increases, the volume of data is growing exponentially. This growth poses unprecedented challenges to data storage, processing, and analysis, urgently requiring innovative technical solutions. Therefore, developing effective call chain data compression methods to reduce storage costs and improve processing efficiency has become urgent.

[0005] With the development of information technology, distributed systems, especially microservice architectures, have become increasingly common. In these systems, the management of call chain data, or trace data, is crucial for system performance monitoring and troubleshooting. Call chain data records the calls and responses between services in detail, providing essential information for optimizing system performance. However, as the number of services increases, the amount of call chain data has rapidly expanded, and traditional data storage and processing methods are no longer able to meet the needs of modern applications.

[0006] Existing text compression technologies are mainly divided into two categories:

[0007] (1) Lossless compression technology: Lossless compression technology ensures that data can be completely restored to its original state after compression. Common algorithms include:

[0008] Entropy coding methods: such as Huffman coding and arithmetic coding, which optimize the code length based on the probability model of the data and are applicable to various data types, including text and images.

[0009] Dictionary encoding method: such as gz ip and l zma, which identifies and replaces repeated data sequences by establishing a dictionary and is widely used in file compression.

[0010] Predictive coding methods, such as PPMd and DeepZip, rely on historical data to predict future data and then encode the prediction results.

[0011] (2) Lossy compression technology: Lossy compression technology provides a higher compression ratio while allowing for the loss of some data. Common methods include:

[0012] Uniform random sampling: simplifies the data set, but may ignore the potential value of the data and cause key information to be lost.

[0013] Feature-based biased sampling: Selectively sampling based on the “interestingness” or information richness of the data, which is more suitable for dynamic and changing call chain data types.

[0014] The above-mentioned related technologies have the following disadvantages:

[0015] (1) Limitations of lossless compression technology: Although lossless compression technologies can maintain data integrity, they usually face the following challenges:

[0016] Inefficiency: When processing large amounts of structured data, such as call chain data, existing lossless compression algorithms often cannot effectively handle redundant information in the data, resulting in limited compression ratios and processing speeds. High complexity: Efficient lossless compression algorithms such as arithmetic coding require high computing resources, which is impractical in resource-constrained environments or environments that require fast responses. Poor generalization ability: Predictive coding methods rely on the analysis of historical data, and the models trained by these methods have limited generalization capabilities and may not be effective when faced with variable data distributions.

[0017] (2) Disadvantages of lossy compression technology: Although lossy compression technology can improve data processing efficiency, it has the following problems: Data loss: Lossy compression methods such as sampling reduce storage requirements by discarding some data, but this method often fails to ensure the integrity of key data and may lead to the loss of important information. Low coverage: Uniform random sampling technology cannot effectively capture abnormal or rare execution paths, which is particularly important for fault diagnosis and system performance optimization. Complex configuration: Advanced lossy compression technology requires complex configuration and tuning. For dynamically changing system environments, this method lacks adaptability and flexibility. Summary of the Invention

[0018] In view of this, an embodiment of the present invention provides an efficient and real-time call chain data compression method and system based on information redundancy to ensure data integrity and enhance generalization capabilities.

[0019] An aspect of an embodiment of the present invention provides a call chain data compression method based on information redundancy, comprising the following steps:

[0020] For the collected call chain data, the KMeans algorithm is used to extract similar data, and the data points in the call chain data are grouped to obtain various cluster groups; wherein, the data similarity within the same cluster group is high, and the data similarity between different cluster groups is low;

[0021] The cluster group is sampled using an offline tail sampling strategy based on hierarchical clustering, and a structured file is output;

[0022] The structured file is subjected to lossless compression in a lossless compression manner to complete the compression of the call chain data.

[0023] Optionally, the collected call chain data is subjected to a KMeans algorithm to extract similar data, and data points in the call chain data are grouped to obtain cluster groups, including the following steps:

[0024] The call chain data of microservices is collected through the OpenTe lemetry framework and processed by a graph generation unit built with a NetworkX library function to obtain a corresponding directed graph. The directed graph is used to represent the call relationship between each service component in the call chain data.

[0025] Processing the nodes in the directed graph by applying the Node2Vec algorithm through a node vectorization unit to convert the node data in the graph structure into a vector form;

[0026] The vectorized node data is normalized by using a standardization tool through a standardization processing unit to obtain a standardized data stream;

[0027] Performing feature selection on the standardized data stream by a feature selection unit to obtain a target feature set;

[0028] The Kmeans algorithm is implemented by the clustering processing unit to cluster the data points, and the data is divided into a plurality of different clusters, each of which has a corresponding cluster label.

[0029] Optionally, the adopting of an offline tail sampling strategy based on hierarchical clustering to perform sampling processing on the cluster group and outputting a structured file comprises the following steps:

[0030] Calculate the relative proportion of each KMeans cluster in the overall data set and determine the number of samples required to be extracted from each cluster;

[0031] For the data within each KMeans cluster, the linkage function in the scipy library is used to calculate the linkage matrix of the hierarchical clustering, and the AgglomerativeClustering function in the sk-learn library is used to execute the hierarchical clustering algorithm to build a tree structure. The purity principle of the tree structure is used to place data points with high similarity in the same branch to obtain the hierarchical clustering result;

[0032] According to the hierarchical clustering results, samples are extracted from each sub-cluster to complete the sampling;

[0033] After sampling, the silhouette_score function of the sklearn library is used to calculate the silhouette coefficient of each cluster to measure the clustering quality.

[0034] All extracted samples are deduplicated, integrated into a new data structure, and output to obtain the structured file.

[0035] Optionally, extracting samples from each sub-cluster according to the hierarchical clustering result is specifically as follows:

[0036] The established tree structure is randomly traversed from the root node to each child node until the leaf node and sampling is performed.

[0037] Optionally, the calculation formula for calculating the silhouette coefficient is:

[0038]

[0039] Among them, S(i) represents the silhouette coefficient of data point i; b(i) represents the average distance from data point i to the nearest cluster, and is called inter-cluster dissimilarity; a(i) represents the average distance from data point i to other data points in its cluster, and is called intra-cluster dissimilarity;

[0040] The calculation formula of the intra-cluster dissimilarity is:

[0041]

[0042] The calculation formula of the inter-cluster dissimilarity is:

[0043]

[0044] Among them, C i represents the cluster where data point i is located; d(i,j) represents the distance between data points i and j; C k Different from C i Other clusters of .

[0045] Optionally, the lossless compression processing of the structured file in a lossless compression manner to complete the compression of the call chain data includes the following steps:

[0046] Constructing a data mapping construction unit to record the mapping of each unique value in the data to a new numeric identifier; when the data preprocessing unit receives the structured file, each value is checked and encoded by the mapping construction unit, mapping English characters to integers, retaining a reversible compressed mapping relationship, and using the pickle library for dictionary serialization and storage;

[0047] Calculating the relative center value of each predefined data cluster using the data processing unit, performing median calculation on the data points of each cluster by reading the cluster index, thereby obtaining the center value associated with the cluster;

[0048] receiving the processed data through a data compression unit and performing compression conversion to convert the obtained compressed data into a specific data type;

[0049] The compressed data is compressed twice by using the bz2 general compression algorithm through the data storage unit to complete the compression of the call chain data.

[0050] Optionally, the method further comprises the following steps:

[0051] Decompress using the bz2 algorithm, match the compressed data with the corresponding center value and add them together;

[0052] Use the mapping relationship to restore data item by item to obtain the intermediate data set;

[0053] By deserializing and querying the compression dictionary, the original value corresponding to each compressed value is found to complete the decompression of the data.

[0054] Another aspect of an embodiment of the present invention further provides a call chain data compression system based on information redundancy, including:

[0055] The first module is used to extract similar data from the collected call chain data using the KMeans algorithm, and group the data points in the call chain data to obtain cluster groups. The data similarity within the same cluster group is high, and the data similarity between different cluster groups is low.

[0056] The second module is used to perform sampling processing on the cluster group using an offline tail sampling strategy based on hierarchical clustering and output a structured file;

[0057] The third module is used to perform lossless compression on the structured file in a lossless compression manner to complete the compression of the call chain data.

[0058] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0059] The memory is used to store programs;

[0060] The processor executes the program to implement the method described above.

[0061] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0062] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0063] The embodiment of the present invention uses the KMeans algorithm to extract similar data from the collected call chain data, groups the data points in the call chain data, and obtains various cluster groups; wherein, the data similarity within the same cluster group is high, and the data similarity between different cluster groups is low; the cluster groups are sampled using an offline tail sampling strategy based on hierarchical clustering to output a structured file; and the structured file is losslessly compressed using a lossless compression method to complete the compression of the call chain data. The present invention combines the sampling idea of ​​lossy compression with the characteristics of call chain data, greatly reducing the size and storage burden of the compressed file while maintaining the value of data analysis, improving efficiency and real-time performance, ensuring data integrity, and enhancing generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 An overall step flow chart provided for an embodiment of the present invention;

[0066] Figure 2 A flow chart of the clustering process provided by an embodiment of the present invention;

[0067] Figure 3 A flowchart of the sampling process provided by an embodiment of the present invention;

[0068] Figure 4 A flowchart of a lossless compression process provided by an embodiment of the present invention;

[0069] Figure 5 A schematic diagram of hierarchical clustering tree construction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the objects, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0071] It will be understood that the terms "first," "second," and the like used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are merely used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of," "when," or "in response to a determination."

[0072] The terms "at least one", "plurality", "each", "any", etc. used in the present invention include at least one, two or more, multiple, two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0074] Before describing the embodiments of the present invention in detail, some related technologies involved in the embodiments of the present invention are first described as follows:

[0075] Call chain: A term used to describe the sequence of method or function calls from one execution point to another. This concept is primarily used in software development, particularly in performance analysis, debugging, and system design. The call chain can help developers understand the program's execution flow and the dependencies between its parts. Specifically, when a program is executed, it may call one function from another. These functions may further call other functions, forming a sequence or chain of calls. Each function call is a link in the chain, and this chain can help trace the execution path of the program. For example, in a complex application, understanding the call chain can help determine which parts of the code may cause performance bottlenecks or errors.

[0076] Clustering: Clustering is a technique for dividing objects in a dataset into groups or clusters, where objects within each group share similar attributes or characteristics, while objects across different groups exhibit significant differences. The core purpose of clustering algorithms is to automatically classify data based on its inherent structure, without pre-assigned class labels. It is a form of unsupervised learning, commonly used for data preprocessing, data compression, and as an aid to other complex analytical tasks.

[0077] OpenTe lemetry: OpenTe lemetry is an open source project for capturing, generating, managing, and exporting telemetry data (such as metrics, logs, and traces). It aims to provide a unified and standardized telemetry solution for cloud-native software and services. OpenTe lemetry provides a comprehensive toolset and API that enables developers to collect detailed data about software performance and behavior in a variety of programming languages ​​and environments. It aims to provide a seamless integration method to help developers automatically collect system data without rewriting or redeploying applications.

[0078] The present invention provides a method for compressing call chain data based on information redundancy, which relates to the field of computer technology. The method for compressing call chain data based on information redundancy provided in the present invention can be applied to a terminal or a server, or can be software running on a terminal or a server. In some embodiments, the terminal can be a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and an in-vehicle terminal, etc., but is not limited to these. The server side can be configured as an independent physical server, or as a server cluster or distributed system consisting of multiple physical servers. It can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application that implements the method for compressing call chain data based on information redundancy, etc., but is not limited to the above forms.

[0079] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0080] Specifically, in response to the problems existing in the prior art, the embodiments of the present invention analyze and explore the characteristics and structural features of the call chain data to discover and eliminate information redundancy. The clustering part of the embodiment of the present invention uses the KMeans algorithm based on the call graph to extract similar data and group the data points in the data set so that the similarity of data points in the same group is high and the similarity of data points between different groups is low. The sampling part greatly reduces the amount of data through hierarchical clustering and intelligent sampling strategies, while retaining the representativeness of the data and improving the efficiency and effectiveness of data analysis. The theory is that similar execution paths in the tree construction process are likely to be placed in the same branch, so these branches are relatively deeper. Uncommon tracking paths are often located in independent branches with fewer nodes, so the hierarchy is shallower and easier to be sampled. The lossless compression part ensures that no information is lost while compressing the data volume, and maximizes the compression of call chain data and the elimination of redundancy.

[0081] The input for this embodiment of the present invention is the standard format of microservice call chain data collected by the OpenTe lemetry framework, in a structured table format (CSV file format). This data contains information such as the call type, call start timestamp, call duration, call success flag, IDs for the complete call chain and individual calls, the ID of the parent call, the ID in the configuration management database, and the name of the called service. The output for this embodiment of the present invention is a compressed source data package and two important intermediate products for data recovery: a bz2-compressed dictionary file and a central value file.

[0082] Specifically, if Figure 1 As shown, a call chain data compression method based on information redundancy according to an embodiment of the present invention includes the following steps:

[0083] For the collected call chain data, the KMeans algorithm is used to extract similar data, and the data points in the call chain data are grouped to obtain various cluster groups; wherein, the data similarity within the same cluster group is high, and the data similarity between different cluster groups is low;

[0084] The cluster group is sampled using an offline tail sampling strategy based on hierarchical clustering, and a structured file is output;

[0085] The structured file is subjected to lossless compression in a lossless compression manner to complete the compression of the call chain data.

[0086] Optionally, the collected call chain data is subjected to a KMeans algorithm to extract similar data, and data points in the call chain data are grouped to obtain cluster groups, including the following steps:

[0087] The call chain data of microservices is collected through the OpenTe lemetry framework and processed by a graph generation unit built with a NetworkX library function to obtain a corresponding directed graph. The directed graph is used to represent the call relationship between each service component in the call chain data.

[0088] Processing the nodes in the directed graph by applying the Node2Vec algorithm through a node vectorization unit to convert the node data in the graph structure into a vector form;

[0089] The vectorized node data is normalized by using a standardization tool through a standardization processing unit to obtain a standardized data stream;

[0090] Performing feature selection on the standardized data stream by a feature selection unit to obtain a target feature set;

[0091] The Kmeans algorithm is implemented by the clustering processing unit to cluster the data points, and the data is divided into a plurality of different clusters, each of which has a corresponding cluster label.

[0092] Optionally, the adopting of an offline tail sampling strategy based on hierarchical clustering to perform sampling processing on the cluster group and outputting a structured file comprises the following steps:

[0093] Calculate the relative proportion of each KMeans cluster in the overall data set and determine the number of samples required to be extracted from each cluster;

[0094] For the data within each KMeans cluster, the linkage function in the scipy library is used to calculate the linkage matrix of the hierarchical clustering, and the AgglomerativeClustering function in the sk-learn library is used to execute the hierarchical clustering algorithm to build a tree structure. The purity principle of the tree structure is used to place data points with high similarity in the same branch to obtain the hierarchical clustering result;

[0095] According to the hierarchical clustering results, samples are extracted from each sub-cluster to complete the sampling;

[0096] After sampling, the silhouette_score function of the sklearn library is used to calculate the silhouette coefficient of each cluster to measure the clustering quality.

[0097] All extracted samples are deduplicated, integrated into a new data structure, and output to obtain the structured file.

[0098] Optionally, extracting samples from each sub-cluster according to the hierarchical clustering result is specifically as follows:

[0099] The established tree structure is randomly traversed from the root node to each child node until the leaf node and sampling is performed.

[0100] Optionally, the calculation formula for calculating the silhouette coefficient is:

[0101]

[0102] Among them, S(i) represents the silhouette coefficient of data point i; b(i) represents the average distance from data point i to the nearest cluster, and is called inter-cluster dissimilarity; a(i) represents the average distance from data point i to other data points in its cluster, and is called intra-cluster dissimilarity;

[0103] The calculation formula of the intra-cluster dissimilarity is:

[0104]

[0105] The calculation formula of the inter-cluster dissimilarity is:

[0106]

[0107] Among them, C i represents the cluster where data point i is located; d(i,j) represents the distance between data points i and j; C k Different from C i Other clusters of .

[0108] Optionally, the lossless compression processing of the structured file in a lossless compression manner to complete the compression of the call chain data includes the following steps:

[0109] Constructing a data mapping construction unit to record the mapping of each unique value in the data to a new numeric identifier; when the data preprocessing unit receives the structured file, each value is checked and encoded by the mapping construction unit, mapping English characters to integers, retaining a reversible compressed mapping relationship, and using the pickle library for dictionary serialization and storage;

[0110] Calculating the relative center value of each predefined data cluster using the data processing unit, performing median calculation on the data points of each cluster by reading the cluster index, thereby obtaining the center value associated with the cluster;

[0111] receiving the processed data through a data compression unit and performing compression conversion to convert the obtained compressed data into a specific data type;

[0112] The compressed data is compressed twice by using the bz2 general compression algorithm through the data storage unit to complete the compression of the call chain data.

[0113] Optionally, the method further comprises the following steps:

[0114] Decompress using the bz2 algorithm, match the compressed data with the corresponding center value and add them together;

[0115] Use the mapping relationship to restore data item by item to obtain the intermediate data set;

[0116] By deserializing and querying the compression dictionary, the original value corresponding to each compressed value is found to complete the decompression of the data.

[0117] The following describes the specific implementation process of the present invention in detail using a specific application scenario as an example:

[0118] The technical solution of the present invention consists of three parts: a clustering part, a sampling part, and a lossless compression part.

[0119] The main process of the present invention is:

[0120] 1.Reference Figure 2 The clustering part mainly designs the KMeans clustering method based on the call graph, which can extract similar information to a large extent. The process is:

[0121] (1) The microservice call data collected by the OpenTe lemetry framework is processed by a graph generation unit built by a networkx library function. This unit is responsible for converting the input data into a directed graph to more accurately represent the call relationship between each service component in the call chain data.

[0122] (2) After the graph data is formed, the node vectorization unit applies the Node2Vec algorithm to process the nodes in the directed graph, converting the nodes in the graph structure into vector form for mathematical processing and analysis in subsequent steps.

[0123] (3) Subsequently, the normalization processing unit uses a variety of normalization methods to process the vectorized node data to ensure that the comparison of data of different dimensions has a unified scale. The normalization processing methods available in this solution include but are not limited to:

[0124] StandardSca ler: Transforms the data into a standard normal distribution with mean 0 and standard deviation 1 by subtracting the mean and dividing by the standard deviation.

[0125] MinMaxScaler: Scales the data to between the specified minimum and maximum values ​​(usually 0 to 1), which is suitable for cases where the feature is within the known minimum and maximum range.

[0126] Normalizer: Scales each sample to the unit norm, usually used for normalization of sparse matrices or eigenvectors.

[0127] (4) The standardized data flows to the feature selection unit, which selects the feature set that has the greatest impact on the clustering results based on advanced algorithms. This step is key to ensuring the efficiency and effectiveness of the clustering algorithm. Feature selection algorithms available for this solution include but are not limited to:

[0128] Variance Thresholding: Select the most effective features by removing features whose variance is lower than a certain threshold.

[0129] Mutual Information Method: Calculate the mutual information between each feature and the target variable, and select features with high mutual information.

[0130] Recursive Feature Elimination (RFE): Recursively train the model to gradually remove the least important features.

[0131] (5) After feature selection, the data enters the clustering processing unit, which implements the KMeans algorithm to cluster the data points. Through this algorithm, the system can divide the data into multiple different clusters. The data points within each cluster are close to each other in the feature space, while the data points between different clusters are far apart. In this way, the call chain data can be divided into different clusters. The information represented by the data in each cluster is similar and has a certain similarity. Based on this, redundancy within the cluster can be eliminated, which is what is done in the subsequent part. The data obtained in this part will have cluster labels.

[0132] 2. If Figure 3 As shown, the sampling part of the embodiment of the present invention designs an offline tail sampling strategy based on hierarchical clustering, which is specifically designed for the data compression problem in call chain data analysis. The tail sampling strategy performs better than the head abstraction strategy in eliminating a large amount of redundant data and retaining edge cases such as anomalies. This technical solution takes the following steps to intelligently sample the pre-clustered data to ensure that the data volume is effectively reduced while maintaining the value of data analysis. The process is as follows:

[0133] (1) This part calculates the relative proportion of each KMeans cluster in the overall data set, and intelligently determines the number of samples required to be extracted from each cluster while ensuring the minimum number of samples for each cluster, ensuring that the number of samples extracted can reflect the distribution of the original data set.

[0134] (2) For the data within each KMeans cluster, this section uses the linkage function in the scipy library to calculate the linkage matrix for hierarchical clustering, and uses the AgglomerativeClustering function in the sklearn library to execute the hierarchical clustering algorithm, constructing a tree structure. Using the purity principle of the tree structure, data points with high similarity are placed on the same branch, providing a logical basis for subsequent sampling. This section is based on the theory that similar and common execution paths are likely to be placed on the same branch, so these branches are relatively deep. In contrast, uncommon tracking paths and unique data are often located in independent branches with fewer nodes, resulting in a shallower hierarchy and easier sampling.

[0135] (3) Based on the hierarchical clustering results, this part extracts samples from each sub-cluster. Specifically, this part randomly traverses each sub-node from the root node to the leaf node of the established tree structure and performs sampling. At the same time, it ensures that even when the sub-cluster size is small, at least one sample can be extracted from each sub-cluster to maintain the diversity of samples and the representativeness of the dataset.

[0136] (4) After sampling, this part calculates the silhouette coefficient for each cluster through the silhouette_score function of the sklearn library. The silhouette coefficient is a measure used to evaluate the quality of clustering. Its calculation formula is as follows:

[0137]

[0138] Among them, S(i) represents the silhouette coefficient of data point i, and its value range is [-1,1]. The specific definition is as follows:

[0139] a(i) is the average distance from data point i to other data points in its cluster, which is called intra-cluster dissimilarity. The calculation formula is:

[0140]

[0141] Among them, C i is the cluster where data point i is located, and d(i,j) is the distance between data points i and j.

[0142] b(i) is the average distance from data point i to the nearest cluster, called inter-cluster dissimilarity, and is calculated as:

[0143]

[0144] Among them, C k Different from C i Other clusters of .

[0145] The closer the silhouette coefficient S(i) is to 1, the better the clustering effect of data point i is; the closer S(i) is to -1, the more suitable the data point i is to be assigned to other clusters; S(i) is close to 0, which means that the data point i is located on the boundary of two clusters.

[0146] Through the above steps, this step evaluates the aggregation effect of clusters at each level of clustering, identifies the best and worst performing clusters, and pays extra attention to these clusters in the sampling, and samples them again to ensure that the key edge cases in the data set can be captured.

[0147] (5) Finally, all extracted samples are integrated into a new data structure after deduplication and output as a structured file for further analysis.

[0148] 3. Such as Figure 4 As shown, the lossless compression part of the embodiment of the present invention further reduces the data volume on the basis of lossy compression and ensures that the compressed data can be completely restored to its original state. The process is as follows:

[0149] (1) Dictionary Compression: During the dictionary compression phase of the present invention, a data mapping unit is established to record the mapping of each unique value in the data to a new numeric identifier. After the data preprocessing unit receives the data, the mapping unit checks and encodes each value, primarily mapping English characters to integers. This maintains a reversible compression mapping relationship and uses a pickle library for dictionary serialization and storage.

[0150] (2) Storing Center Values ​​and Offsets: The present invention utilizes a data processing unit to calculate the relative center value of each predefined data cluster. By reading the cluster index and performing a median calculation on each cluster's data point, the center value associated with the cluster is obtained, avoiding the need to introduce floating-point numbers to expand storage space. The median calculation process is as follows:

[0151] 1) Sorting: Arrange all data points belonging to the same cluster in ascending order.

[0152] 2) Median: If the number of data points is odd, the median is the value of the data point in the middle of the sorted data. If the number of data points is even, the median is the average of the two data points in the middle of the sorted data. To avoid introducing floating-point numbers here, the final result will be retained as an integer.

[0153] 3) Afterwards, this part subtracts each data point from the center value of its cluster and stores the resulting offset.

[0154] (3) Data Compression and Storage: The data compression unit receives the processed data and performs compression conversion. The resulting compressed data is further converted into a specific data type to reduce memory usage. The compression technology of the present invention also includes a data storage unit that uses the bz2 general compression algorithm to perform secondary compression on the compressed data to maximize storage efficiency.

[0155] (4) Restoration process: First, decompress the data using the bz2 algorithm. The compressed data is matched with its corresponding central value and added. The data is restored item by item using the mapping relationship to obtain the intermediate data set. Then, by deserializing and querying the compression dictionary, the original value corresponding to each compressed value is found to complete the data decompression.

[0156] The data set used in the experiment of the technical solution of the present invention is the real production data of the enterprise. Specifically, the data format is the call chain data collected by the OpenTe lemetry framework.

[0157] In general, the embodiments of the present invention achieve the compression ratio results shown in Table 1 through clustering, sampling, and lossless compression in actual experiments on call chain data of different sizes:

[0158] Table 1 Overall compression ratio of data of different sizes

[0159]

[0160] As shown in Table 1, on real call chain datasets from enterprises of different sizes, the scheme of the present invention achieved an overall average compression ratio of 3.2%, which is significantly higher than the currently common compression schemes. This shows that the scheme has a strong compression effect on call chain data and effectively reduces data redundancy.

[0161] The intermediate products of this solution also have practical application value. The sampling process extracts critical call chain data, allowing operations personnel to use a smaller dataset in most scenarios. Furthermore, the data after the lossless compression process of dictionary-based centralization still reflects the inter-call relationship, which is quite practical.

[0162] Specifically, the key experimental results of each part are as follows:

[0163] 1. Clustering: The clustering part tests the call graph-based KMeans clustering algorithm under different cluster settings, and obtains the experimental results shown in Table 2 below:

[0164] Table 2 Clustering experimental results

[0165] Cluster settings Silhouette Coefficient Calinski-Harabasz Index 3 0.6335 43282.25 4 0.6357 42124.13 5 0.5982 48201.18 6 0.6709 54885.71

[0166] As shown in Table 2, under different cluster number settings, the KMeans clustering algorithm based on the call graph maintains a high silhouette coefficient and variance ratio criterion, indicating that the clustering effect is good and the similarity extraction is relatively successful.

[0167] 2. Sampling part: The main experimental result in the sampling part is the sampling ratio, which has been specifically described in Table 1 of the overall results. The experiment is based on the dendrogram function of the pyp lot library to visualize the hierarchical clustering tree, as shown in Figure 5 As shown, the sampling part successfully built a tree, and thus tree sampling was performed based on it.

[0168] 3. Lossless Compression: The experimental results of the lossless compression section mainly focus on the compression ratio. In addition to the compression ratio test of the complete compression process in Table 1, the additional single-step experimental results of this section are shown in Table 3 below. The experiments were conducted using the original call chain dataset.

[0169] Table 3 Lossless compression experimental results

[0170] Original data size Data size after lossless compression Compression ratio 3,902,636 bytes 581,092 bytes 14.89％ 6,454,247 bytes 965,404 bytes 14.96％ 9,106,016 bytes 1,315,819 bytes 14.45％ 13,008,565 bytes 1,938,235 bytes 14.90％

[0171] As shown in Table 3, the lossless compression part can achieve a compression ratio of 14.9% under the existing call chain dataset, indicating that the lossless compression part also has a strong compression effect on the call chain data.

[0172] In summary, compared with the prior art, the present invention has the following features:

[0173] 1. The KMeans algorithm based on the directed call graph can capture the key information of mutual calls in call chain data to the greatest extent, extract the commonalities of related structures, and assign appropriate cluster labels.

[0174] 2. The offline tail sampling strategy based on hierarchical clustering makes full use of the cluster labels generated by the previous process to perform hierarchical clustering tree sampling. The cluster labels of the previous clustering provide convenience for the best and worst clustering judgment of the sampling part, and can further focus on marginal data.

[0175] 3. The combination of lossless compression and lossy compression such as sampling minimizes the storage space required for call chain data while maintaining the value of data analysis.

[0176] 4. The reuse of clusters reduces the cost and running time required for each process.

[0177] 5. For different forms of call chain data, the procedures adopted are basically the same and have high generalization.

[0178] Compared with the prior art, the present invention has the following advantages:

[0179] 1. Based on the characteristics of call chain data, the sampling concept of lossy compression is combined to significantly reduce the size and storage burden of compressed files while maintaining the value of data analysis.

[0180] 2. After obtaining the similarity of call chain data and performing cluster analysis and sampling, it is convenient for operation and maintenance personnel to observe and process the call chain data, without having to analyze it from massive amounts of raw data.

[0181] 3. Similar to the structured analysis method logshr ink for log data, the present invention provides a structured analysis and compression method for call chain data, which supports single-step and single-column operation compression and has greater flexibility.

[0182] An embodiment of the present invention further provides a call chain data compression system based on information redundancy, comprising:

[0183] The first module is used to extract similar data from the collected call chain data using the KMeans algorithm, and group the data points in the call chain data to obtain cluster groups. The data similarity within the same cluster group is high, and the data similarity between different cluster groups is low.

[0184] The second module is used to perform sampling processing on the cluster group using an offline tail sampling strategy based on hierarchical clustering and output a structured file;

[0185] The third module is used to perform lossless compression on the structured file in a lossless compression manner to complete the compression of the call chain data.

[0186] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0187] An embodiment of the present invention further provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor implements the aforementioned method for compressing call chain data based on information redundancy when executing the computer program. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0188] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0189] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned call chain data compression method based on information redundancy.

[0190] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0191] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0192] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0193] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0194] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0195] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0196] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0197] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0198] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0199] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0200] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A call chain data compression method based on information redundancy, characterized in that: The following steps are involved: For the collected call chain data, the KMeans algorithm is used to extract similar data, and the data points in the call chain data are grouped to obtain various cluster groups; wherein, the data similarity within the same cluster group is high, and the data similarity between different cluster groups is low; The cluster groups are sampled using an offline tail sampling strategy based on hierarchical clustering, and a structured file is output; Performing lossless compression on the structured file to complete the compression of the call chain data; The collected call chain data is subjected to a KMeans algorithm to extract similar data, and the data points in the call chain data are grouped to obtain cluster groups, including the following steps: The call chain data of microservices is collected through the OpenTelemetry framework and processed by a graph generation unit built with a NetworkX library function to obtain a corresponding directed graph. The directed graph is used to represent the call relationship between each service component in the call chain data. Processing the nodes in the directed graph by applying the Node2Vec algorithm through a node vectorization unit to convert the node data in the graph structure into a vector form; The vectorized node data is normalized by using a standardization tool through a standardization processing unit to obtain a standardized data stream; Performing feature selection on the standardized data stream by a feature selection unit to obtain a target feature set; The Kmeans algorithm is implemented by the clustering processing unit to cluster the data points, and the data is divided into a plurality of different clusters, each of which has a corresponding cluster label; The method of sampling the cluster groups using an offline tail sampling strategy based on hierarchical clustering and outputting a structured file comprises the following steps: Calculate the relative proportion of each KMeans cluster in the overall data set and determine the number of samples required to be extracted from each cluster; For the data within each KMeans cluster, the linkage function in the scipy library is used to calculate the linkage matrix of the hierarchical clustering, and the AgglomerativeClustering function in the sklearn library is used to execute the hierarchical clustering algorithm to build a tree structure. The purity principle of the tree structure is used to place data points with high similarity in the same branch to obtain the hierarchical clustering result; According to the hierarchical clustering results, samples are extracted from each sub-cluster to complete the sampling; After sampling is completed, the silhouette coefficient of each cluster is calculated through the silhouette_score function of the sklearn library to measure the clustering quality; All extracted samples are deduplicated, integrated into a new data structure, and output to obtain the structured file.

2. The method for compressing call chain data based on information redundancy according to claim 1, characterized in that: The step of extracting samples from each sub-cluster according to the hierarchical clustering results is as follows: The established tree structure is randomly traversed from the root node to each child node until the leaf node and sampling is performed.

3. The call chain data compression method based on information redundancy according to claim 1 is characterized in that: The calculation formula for calculating the silhouette coefficient is: in, Representative data points Silhouette coefficient; Representative data points The average distance to the nearest cluster is called inter-cluster dissimilarity; Representative data points The average distance to other data points in the cluster where it belongs is called intra-cluster dissimilarity; The calculation formula of the intra-cluster dissimilarity is: The calculation formula of the inter-cluster dissimilarity is: in, Representative data points The cluster in which it is located; Representative data points and the distance between them; Different from Other clusters of .

4. The method for compressing call chain data based on information redundancy according to claim 1, characterized in that: The method of performing lossless compression on the structured file to complete the compression of the call chain data includes the following steps: Constructing a data mapping construction unit to record the mapping of each unique value in the data to a new numeric identifier; when the data preprocessing unit receives the structured file, each value is checked and encoded by the mapping construction unit, mapping English characters to integers, retaining a reversible compressed mapping relationship, and using the pickle library for dictionary serialization and storage; Calculating the relative center value of each predefined data cluster using the data processing unit, performing median calculation on the data points of each cluster by reading the cluster index, thereby obtaining the center value associated with the cluster; The data compression unit receives the processed data and performs compression conversion, converting the obtained compressed data into a data type that can reduce memory usage; The compressed data is compressed twice by using the bz2 general compression algorithm through the data storage unit to complete the compression of the call chain data.

5. A call chain data compression method based on information redundancy according to any one of claims 1 to 4, characterized in that: The method further comprises the following steps: Decompress using the bz2 algorithm, match the compressed data with the corresponding center value and add them together; Use the mapping relationship to restore data item by item to obtain the intermediate data set; By deserializing and querying the compression dictionary, the original value corresponding to each compressed value is found to complete the decompression of the data.

6. A system for implementing the call chain data compression method based on information redundancy according to any one of claims 1 to 5, characterized in that: include: The first module is used to extract similar data from the collected call chain data using the KMeans algorithm, and group the data points in the call chain data to obtain cluster groups. The data similarity within the same cluster group is high, and the data similarity between different cluster groups is low. The second module is used to sample the cluster group using an offline tail sampling strategy based on hierarchical clustering and output a structured file; The third module is used to perform lossless compression on the structured file in a lossless compression manner to complete the compression of the call chain data.

7. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images