Data compression method and device, data decompression method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a data compression method and device, a data decompression method and device, electronic equipment, a storage medium and a computer program product. The data compression method comprises the steps that for each piece of data in log data to be compressed, the following processing is executed: the current data is analyzed to obtain variable groups of at least one key, each variable group comprises a plurality of variables, and one variable is a value of the key corresponding to the corresponding variable group; the incidence relation between the variable groups of any two keys in the at least one key is determined, and the incidence relation indicates redundant content between the variable groups of the two keys; performing redundancy elimination processing on the variable group of the at least one key based on the incidence relation to obtain the processed variable group of the at least one key; and compressing the incidence relation and the processed variable group of the at least one key to obtain data after current data compression.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and in particular to a data compression method and apparatus, a data decompression method and apparatus, electronic equipment, storage media, and computer program products. Background Technology

[0002] Currently, existing compression schemes for semi-structured log data are mainly based on the idea of ​​restructuring the log data. Although structuring can bring many benefits, it can also bring side effects. For example, once a relationship is found, such as two variables having the same value, without structuring, these two values ​​would be close to each other and could be easily detected by the compressor. However, after structuring, the two variables are divided into two different variable groups, which makes the originally close values ​​even more distant. At this time, it is more difficult for the compressor to detect this redundancy, resulting in low log data compression rate and affecting log data storage. Summary of the Invention

[0003] This disclosure provides a data compression method and apparatus, a data decompression method and apparatus, an electronic device, a storage medium, and a computer program product to at least solve the problem of low log data compression rate in related technologies.

[0004] According to a first aspect of the present disclosure, a data compression method is provided, comprising: for each piece of data in log data to be compressed, performing the following processing: parsing the current data to obtain at least one set of variables with a key, wherein each set of variables contains multiple variables, and one variable is a value of a key corresponding to the set of variables; determining an association relationship between any two sets of variables with at least one key, wherein the association relationship indicates redundant content between the two sets of variables with different keys; performing redundancy removal processing on the set of variables with at least one key based on the association relationship to obtain a processed set of variables with at least one key; and compressing the association relationship and the processed set of variables with at least one key to obtain compressed data of the current data.

[0005] According to a second aspect of the present disclosure, a data decompression method is provided, comprising: for each piece of data in log data to be decompressed, performing the following processing: decompressing the compressed data of the current data to obtain the association relationship of the current data and a variable group of at least one key after processing; based on the association relationship, restoring the redundant part removed from the variable group of at least one key after processing to obtain a variable group of at least one key after restoration; and obtaining the current data based on the variable group of at least one key after restoration.

[0006] According to a third aspect of the present disclosure, a data compression apparatus is provided, comprising: a first processing unit configured to perform compression processing on each piece of data in log data to be compressed through the following subunits: a parsing subunit configured to parse the current data to obtain at least one set of variables for a key, wherein each set of variables contains multiple variables, and one variable is a value of a key corresponding to the set of variables; a determining subunit configured to determine an association relationship between any two sets of variables for at least one key, wherein the association relationship indicates redundant content between the two sets of variables for the key; a redundancy removal subunit configured to perform redundancy removal processing on the set of variables for at least one key based on the association relationship to obtain a processed set of variables for at least one key; and a compression subunit configured to compress the association relationship and the processed set of variables for at least one key to obtain compressed data of the current data.

[0007] According to a fourth aspect of the present disclosure, a data decompression apparatus is provided, comprising: a second processing unit configured to perform decompression processing on each piece of data in log data to be decompressed through the following subunits: a decompression subunit configured to decompress the compressed data of the current data to obtain the association relationship of the current data and a variable group of at least one key after processing; a recovery subunit configured to recover the redundant parts removed from the variable group of at least one key after processing based on the association relationship to obtain a variable group of at least one key after recovery; and an acquisition subunit configured to obtain the current data based on the variable group of at least one key after recovery.

[0008] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the method as described above.

[0009] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by at least one processor, causes at least one processor to perform the method described above.

[0010] According to a seventh aspect of the present disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, implement the method described above.

[0011] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: According to the data compression method and apparatus, data decompression method and apparatus, electronic equipment, storage medium, and computer program products disclosed herein, it is possible to extract some redundant content between different variable groups in each piece of log data, referred to as the correlation relationship. Then, using this correlation relationship, redundancy removal processing is performed on the variable groups, and subsequently, the redundancy-removed variable groups and correlation relationships are compressed. This transforms multiple identical data sets into a single data set, greatly mitigating the potential side effects of directly performing structured data processing, thereby improving the log data compression rate. Therefore, this disclosure solves the problem of low log data compression rate in related technologies.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0014] Figure 1 This is a schematic diagram illustrating the storage format of Parquet according to exemplary embodiments of the present disclosure; Figure 2 This is a flowchart illustrating a data compression method according to exemplary embodiments of the present disclosure; Figure 3 This is a schematic diagram illustrating a JSON data according to an exemplary embodiment of the present disclosure; Figure 4 This is a schematic diagram illustrating an association identification process according to exemplary embodiments of the present disclosure; Figure 5 This is a schematic diagram illustrating a parse tree construction process according to exemplary embodiments of the present disclosure; Figure 6 This is a schematic diagram illustrating the result of template replacement for each node in the parse tree according to an exemplary embodiment of the present disclosure; Figure 7 This is a schematic diagram illustrating a dictionary encoding process according to exemplary embodiments of the present disclosure; Figure 8 This is a schematic diagram illustrating a byte-aware bit packing encoding method according to exemplary embodiments of the present disclosure; Figure 9 This is a flowchart illustrating a data decompression method according to exemplary embodiments of the present disclosure; Figure 10 This is a schematic diagram illustrating a compressor for semi-structured data according to exemplary embodiments of the present disclosure; Figure 11 This is a block diagram illustrating a data compression apparatus according to exemplary embodiments of the present disclosure; Figure 12 This is a block diagram illustrating a data decompression apparatus according to exemplary embodiments of the present disclosure; Figure 13 This is a diagram illustrating a computing environment coupled to a user interface according to exemplary embodiments of the present disclosure. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0016] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0017] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0018] Modern internet companies record data at the event level: system operation events, scheduling decision events, etc. These data records are stored in a semi-structured format, such as semi-structured log data. Common semi-structured data formats include JSON (JavaScript Object Notation), eXtensible Markup Language (XML), and parameter lists. Compared to traditional row-column structures, semi-structured data offers greater flexibility and scalability, adapting to dynamic changes in data distribution and providing a solid data foundation for modern applications. The collected semi-structured data can help internet companies train recommendation systems and user profiling systems, and also provides reliable data for operations and maintenance systems, offering strong support for company operations.

[0019] However, for data holders, storing this semi-structured data incurs enormous costs. On the one hand, the scale of this data is massive; for example, some studies claim that large companies generate over 10PB of semi-structured log data daily, with a total weekly generation of approximately 60PB. On the other hand, this data requires long-term storage for root cause analysis when system errors occur.

[0020] Given the massive data volume and long storage duration, semi-structured data needs to be compressed during storage to reduce storage costs. Currently, this data is typically stored in columnar files (such as Parquet and ORC) in data warehouses in string format. These file formats already support general lossless compression algorithms for string data types, including lightweight encoding methods (dictionary encoding, etc.) and heavyweight compression (byte-granular compression tools such as gzip and Zstandard). Using these compression methods can reduce storage space usage without causing any data loss. However, semi-structured data has unique characteristics; compared to general data, it contains a large amount of redundant structural information. Therefore, using these general methods to process raw semi-structured data usually does not achieve good compression results.

[0021] Currently, there are several compression tools specifically designed to compress semi-structured data, offering better compression ratios than general-purpose methods. These tools are primarily based on the idea of ​​restructuring the semi-structured data. Examples include the μSlope and Parquet systems. A brief description of these two systems follows: The μSlope system can losslessly compress semi-structured log data and support fast searching without requiring complete decompression. μSlope stores this representation only once, rather than with each record, a concise representation of the log data's schema structure. This system further "structures" the semi-structured data by grouping records with the same schema structure into the same tables, thus making each table structured. Experimental results based on real production data show that μSlope's compression ratio ranges from 21.9:1 to 186.8:1, significantly outperforming existing semi-structured data management systems (SSDMS), and its compression rate is 2.34 times higher than Zstandard, while its search speed is 5.77 times faster than other SSDMS systems.

[0022] This technical solution features three key innovations: it proposes a merged parse tree to represent a common log pattern and uses pattern mapping to concisely represent each pattern, thereby significantly reducing storage redundancy; it divides records into different tables according to patterns, ensuring that each table is structured, thus allowing the application of relational table optimization strategies to further improve compression efficiency; and it designs a query processing algorithm based on pattern information and an encoding table, improving query efficiency through query optimization and partial decompression, representing a cutting-edge technology in academia.

[0023] Parquet has become the standard for data storage in recent years due to several advantages: 1. Data compression: By applying various encoding and compression algorithms, Parquet files reduce memory consumption and data size. 2. Columnar storage: Fast data retrieval is crucial in data analysis workloads, and columnar storage is a key requirement for rapid access. 3. Language independence: Developers can use different programming languages ​​to manipulate data in Parquet files, offering excellent scalability. 4. Open-source format: Not vendor-locked. 5. Support for complex data types. In traditional row-based storage, data is stored as a sequence of rows. When accessing data, the engine typically scans every row from beginning to end, resulting in significant overhead. In reality, accessing data in most cases only requires data from a few specific columns; the rest is redundant for a single operation. Therefore, column-based storage emerged to address this need. Each column is a separate entity, physically independent of the others. Parquet is a column-based format that stores data in row groups, with a structure like... Figure 1As shown, in Parquet, Column represents a column, Row Group represents a row group, product represents a product, customer represents a consumer, country represents the city where the purchase was made, Data represents the date of purchase, and sales Amount represents the amount spent. In Parquet's design, columns are still stored as independent units, but an additional structure called RowGroup is introduced. Parquet files contain metadata, meaning each Parquet file contains "data about the data," such as the maximum and minimum values ​​in a specific column within a specific row group. Additionally, each Parquet file includes a footer that stores information such as format version, schema information, and column metadata. To optimize performance and eliminate unnecessary data structures, the engine reads the metadata first when reading Parquet files.

[0024] However, the μSlope system is primarily designed for semi-structured text logs with highly repetitive schemas, making it unsuitable for data with entirely different schemas. Furthermore, the μSlope system employs an indexless design, which, while offering advantages in storage efficiency, results in slower search speeds compared to cutting-edge index-based search tools. The Parquet system incurs additional overhead when reading metadata, therefore a single Parquet file should be at least several hundred MB in size. While Parquet is relatively easy to read, it consumes excessive storage space for infrequently accessed, cold data, requiring significant compression.

[0025] To address the aforementioned issues, this publication delves into the factors influencing the returns of structured methods, attempting to answer why structured methods underperform on specific data. Based on experimental results obtained from test data, the analysis yields the following insights: 1. Cardinality refers to the proportion of unique elements in a set. Data with a significantly different cardinality distribution benefits more from structuring. Structuring can group elements with the same semantic information together, making it more likely that the compressor, within its sliding window, will discover more redundancy.

[0026] 2. The correlation between values ​​tends to lead to a decrease in compression ratio after structuring, but aggregating values ​​of the same key can separate variable groups with different cardinality, creating a more uniform data distribution for the compressor. Therefore, before aggregating values, it is necessary to identify and eliminate such partial redundancy to avoid its impact on the compression ratio.

[0027] 3. The length of the key has a very limited impact on the yield of structured data. Although the key part is only retained once during the structuring process, the decision on whether to structure a certain piece of data should not be affected by the length of the key.

[0028] 4. The structuring process can significantly improve the compression rate of a type of data characterized by low frequency and high interquartile range, where the keys are often associated with rare or anomalous events in the system. Redundancy in this type of data can be detected and eliminated by deploying structuring.

[0029] In summary, existing general compression schemes generally perform poorly on semi-structured data, and existing semi-structured data compression tools also perform unsatisfactorily on the publicly available test dataset. While structuring data offers many advantages over existing compression tools, it can also introduce side effects in certain situations. These side effects include two aspects: 1. The problem of redundant data being stored multiple times due to the relationships between values; 2. The problem of misalignment between elements and compression granularity due to low cardinality of values ​​during encoding.

[0030] Therefore, this disclosure proposes a compression method for semi-structured log data based on analysis and observation. The focus is on fully utilizing the benefits of structuring while avoiding its side effects, thereby improving the compression performance of semi-structured data. Specifically, it proposes a relationship-aware structuring method. This method extracts redundant content between different variable groups, termed "relationships," and then uses these relationships to remove redundancy from the variable groups. The resulting combination of the removed variable groups and relationships is then compressed, converting multiple identical data sets into a single, unified dataset. This significantly mitigates the potential side effects of direct structuring, thus improving the compression ratio. Furthermore, this disclosure proposes a byte-aware bit-width adjustment method to package elements within a byte and uses padding mechanisms to avoid cross-byte storage of elements. This allows for byte-level alignment during encoding, synchronizing the granularity of the encoding and compression processes, maintaining byte-level semantics, and achieving a superior compression ratio compared to existing methods.

[0031] Hereinafter, data compression methods and apparatus, data decompression methods and apparatus, electronic devices, storage media, and computer program products according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0032] Figure 2 This is a flowchart illustrating a data compression method according to exemplary embodiments of the present disclosure, such as... Figure 2 As shown, for each piece of data in the log data to be compressed, the following processing is performed: In step S201, the current data is parsed to obtain at least one set of variables with a key, wherein each set of variables contains multiple variables, and one variable is a value of the key corresponding to the set of variables.

[0033] As an example, the data to be compressed can be semi-structured log data, which is text data and may include, but is not limited to, JSON objects, JSON arrays, key-value pairs, etc.

[0034] In step S202, the association relationship between the variable groups of any two keys in at least one key is determined, wherein the association relationship indicates the redundant content between the variable groups of the two keys.

[0035] Specifically, this disclosure, through in-depth investigation, found that the correlation between variable groups is a significant reason for the decrease in compression rate during the structuring process, such as... Figure 3 As shown, taking a single JSON data entry from log data as an example, there are three instances of inline redundancy in the values ​​corresponding to the keys "st_reco_id" and "st_ad_id" in the original JSON data. For ease of discussion, this disclosure uses "200647" and "A-wQLG" as examples. After structuring, the two "200647" and two "A-wQLG" are assigned to different variable groups. During encoding, "200647" and "A-wQLG" are encoded twice in the two variable groups. Therefore, compared to the unstructured original representation, this redundancy distance is extended further. As the number of elements in the variable group increases, the data encoded in variable group 0 becomes increasingly longer. General compressors typically use a sliding window approach to identify redundancy, and the window size is limited. Therefore, the sliding window cannot simultaneously include variable groups 0 and 1, and thus cannot capture this long-distance redundancy, resulting in a decrease in the final compression ratio.

[0036] To mitigate this side effect, this disclosure proposes a relationship-aware structured approach. This approach extracts partially redundant information between different variable groups, termed "relationships." These relationships are then used during the actual data parsing process to deduplicate the data, avoiding the storage of unnecessary parts. For example, when partial redundancy is detected, only one part is retained, while the redundant part is deleted. The deleted part can be replaced with a pointer, which is not limited in this disclosure. This relationship-aware structured approach greatly alleviates the potential side effects of direct structuring. Multiple sets of redundant data are transformed into a single data source, which can be replaced by pointers pointing to this data source in other locations. For example, Figure 3 In the "st_ad_id" variable group, "200647" and "A-wQLG" will have their pointers replaced to indicate that they are duplicates of previously encountered data. Thus, even with long redundant distances, since this disclosure has pre-processed and identified these two parts, there will be no further reduction in compression ratio.

[0037] The following describes one process for obtaining association relationships, but this disclosure is not limited to this method: According to an exemplary embodiment of this disclosure, before processing each piece of data in the log data to be compressed, a predetermined number of data entries can be obtained from the log data to be compressed as training samples; based on the common variable groups of the training samples, the association relationships of the training samples are determined, wherein the common variable groups are variable groups of keys whose frequency of occurrence in the training samples reaches a first predetermined threshold; and the association relationship between variable groups of at least two keys is selected from the association relationships. Through this embodiment, the association relationships of uncommon variable groups can be ignored, thus eliminating metadata bloat and avoiding recording excessive metadata to process uncommon variable groups; moreover, this embodiment can extract the association relationships between different variable groups of common variable groups offline, facilitating the determination of the corresponding association relationships when parsing the data to be stored subsequently.

[0038] As an example, a certain amount of data can be sampled from the log data to be compressed as training samples. For example, 0.1% of the data in the log data to be compressed can be selected, and this disclosure does not limit this. Then, common variable groups in the training samples are identified. Specifically, each data in the training samples can be parsed to obtain variable groups with at least one key. Then, variable groups with keys whose frequency reaches a first predetermined threshold are identified as common variable groups. Subsequent processing focuses only on these common variable groups, that is, the association relationship of the training samples is determined based on the common variable groups. For uncommon variable groups, they can be classified into an anomaly set for unified processing. For example, subsequent redundancy removal can be skipped and compression can be performed directly, and this disclosure does not limit this. Such processing can eliminate metadata bloat and avoid recording too much metadata to process uncommon variable groups. For example, if the association relationship is only valid in one variable group, and relevant metadata information is recorded for each variable group to record the validity of this association relationship, it will cause metadata bloat.

[0039] As an example, to obtain common variable groups, it may be necessary to statistically analyze the frequency of occurrence of the corresponding keys in the training samples. Specifically, a parser can be used to structure the training samples, obtain the corresponding key-value pairs, and record the frequency of each key. After scanning the training samples, the frequency of occurrence of each key is obtained. Then, the variable groups of keys whose frequency reaches a first predetermined threshold are identified as common variable groups. Keys whose frequency is lower than the first predetermined threshold will be considered as abnormal keys, and the variable groups corresponding to these keys can be retained in the abnormal set. It should be noted that the aforementioned first predetermined threshold can be 0.5%, and this disclosure does not limit it.

[0040] However, relationships derived from common variable groups can be categorized into common and uncommon relationships. For example, if a relationship is only valid for one variable group, recording metadata for each variable group to document its validity would bloat the metadata. Therefore, for relationships involving common variable groups, only common relationships need to be obtained. The following describes one method for obtaining common relationships, but this disclosure is not limited to this approach: According to an exemplary embodiment of this disclosure, the above-described determination of the association relationship of training samples based on common variable groups in training samples may include: selecting variable groups from common variable groups whose number of JSON array type variables does not exceed a preset value, and designating them as first variable groups; splitting the variables of each first variable group with a template according to the corresponding template to obtain predetermined sub-variables of each first variable group, wherein the predetermined sub-variables of the first variable group correspond to wildcards in the corresponding templates, and the corresponding template of the first variable group includes the non-alphanumeric symbol of the first variable in the first variable group, one of the multiple identical sub-variables after all variables in the first variable group are divided according to the symbol, and multiple different sub-variables represented by the same wildcard; and splitting all... The first occurrence of a subvariable with the same subvariable among the predefined subvariables is identified as the source subvariable of the same subvariable. In response to the source subvariable and any predefined subvariable having different keys in their respective variable groups and meeting the order requirements, the predefined association relationship is determined to have occurred once. Here, meeting the order requirements means that the current order of the two keys is consistent with their order of appearance in the current data. In response to the source subvariable and any predefined subvariable having variables that both conform to the template of their respective variable groups and have the same content as any predefined subvariable, the predefined association relationship is recorded as valid, and its valid value is incremented by 1. In response to the ratio of the valid value of the predefined association relationship to the total number of occurrences of the predefined association relationship reaching a first predefined threshold, the predefined association relationship is determined to be an association relationship of the training samples.

[0041] In this embodiment, only the common associations among common variable groups are considered, that is, the associations with an efficiency reaching a certain threshold, while uncommon associations can be discarded. This processing can further eliminate the expansion of metadata and avoid recording too much metadata to handle uncommon associations. Moreover, it can save computing resources, such as reducing the computational overhead of selecting the association of the current data and the computational overhead of performing template matching on variables.

[0042] Specifically, variable groups containing a large number of JSON array-type variables are quite special and will not be considered during the training phase. Similarly, the variable groups corresponding to the aforementioned exception keys are also quite special and will not be considered during training. Since the common variable groups are already those after removing the variable groups corresponding to exception keys, this embodiment only needs to remove variable groups containing a large number of JSON array-type variables.

[0043] As an example, we can filter out variable groups from common variable groups where the number of JSON array-type variables does not exceed a preset value; this is the first variable group. Then, for each first variable group with a template, each variable can be matched against the corresponding template. Successfully matched variables are split into multiple predefined sub-variables according to the template. Each predefined sub-variable corresponds exactly to a wildcard in the template, and the number of predefined sub-variables parsed for each variable is equal to the number of wildcards in the template. For example, each predefined sub-variable can be recorded as a tuple (...). , ),in, Indicates the first sub-variable in the predetermined sub-variable i The keys corresponding to each variable group Indicates the first i In the group of variables, the first j The predetermined sub-variable, i.e., the tuple represents the first... i The variable in the variable group is the first j One predefined sub-variable.

[0044] For the multiple predefined sub-variables obtained from the splitting, they can be processed sequentially and their fingerprints recorded in a hash table. Once a matching fingerprint is found in the hash table, a candidate association relationship (i.e., the predefined association relationship in the above embodiment) is discovered, which can be... , ) >( , This is the form in which () is recorded, in which ( , ) is the source child variable of the association relationship, ( , The first variable is the target sub-variable of the association relationship. It should be noted that in practical applications, when two sub-variables have different keys and their order matches the order in which they appear in the current data, a candidate association relationship is considered to exist. Then, it is determined whether the variables containing the two predefined sub-variables conform to the corresponding template. If either of the variables does not conform to the template, the candidate association relationship is considered invalid in this data. If both variables conform to the template and their corresponding content is the same, the candidate association relationship is considered valid. However, if the corresponding content of the two predefined sub-variables is different, the candidate association relationship is also considered invalid in this data.

[0045] Through the above processing, the total number of occurrences of each type of association candidate can be determined, as well as the number of times each type of association candidate is effective. For each association candidate, the effectiveness rate of the association candidate is calculated by the ratio of the effective number to the total number of occurrences. If the effectiveness rate exceeds a second predetermined threshold, the association candidate is used as an association in the training sample, thereby obtaining the set of associations for the training sample, i.e., the set of common associations. It should be noted that the above-mentioned second predetermined threshold can be set to 10%, and this disclosure does not limit it; it can be set as needed.

[0046] As an example, Figure 4 This demonstrates a process for identifying relationships, such as... Figure 4 The diagram only shows two predefined sub-variables, "200647" and "A-wQLG", omitting other predefined sub-variables. Two candidate association relationships can be obtained from the current data. , ) >( , )and( , ) >( , Subsequently, if the efficiency of these two candidate associations is higher than a second predetermined threshold, then these two associations are valid and can be used as two associations in the training samples in subsequent stages.

[0047] The following describes one method for obtaining the templates mentioned in the above embodiments, but this disclosure is not limited to this method: According to an exemplary embodiment of this disclosure, before splitting the variables of each first variable group with a template according to the corresponding template to obtain the predetermined sub-variables of each first variable group, for each first variable group, the non-alphanumeric symbols contained in the first variable of the current variable group can be determined as delimiter candidates; according to the order of occurrence of each delimiter candidate from high to low, each delimiter candidate is processed as follows: the predetermined node is split according to the current delimiter candidate to obtain multiple sub-nodes, wherein each sub-node contains the first sub-variable of all variables in the predetermined node after being split in the same splitting order, and in the first processing, the predetermined node is the current variable group; in response to the predetermined variable in the predetermined node If the number of variables exceeds a preset number, the current delimiter candidate is determined as a qualified delimiter, wherein all predetermined variables contain the same number of child nodes. If the number of predetermined variables in a predetermined node does not exceed the preset number, the current delimiter candidate is discarded. Child nodes that do not contain any delimiters are determined as leaf nodes. The next delimiter is determined as the current delimiter in the next processing step, and each child node (excluding leaf nodes) is determined as a predetermined node in the next processing step, until all nodes are leaf nodes. Based on all qualified delimiters and all leaf nodes, a template for the current variable group is obtained. If all delimiter candidates are discarded, it is determined that the current variable group does not have a template. This embodiment allows for convenient and rapid determination of whether a template exists for each variable group.

[0048] Specifically, based on experimental observations, redundancy within a row only partially exists in the values, that is, in the variables. For two variables with redundancy, they are not completely identical, but rather some of their elements are the same. For example, the above... Figure 3 The values ​​corresponding to `st_reco_id` and `st_ad_id` are not completely identical; they only share a portion of the data. If matching is performed at the variable granularity level, this partial redundancy cannot be detected. Therefore, to eliminate redundancy, the variables need to be decomposed into finer granularities. This disclosure notes that variables in the same variable group usually have the same pattern. This phenomenon is related to the initial design conventions; a consistent format facilitates understanding by upstream and downstream programmers. For example, "1|200647|liveStream:A wQLG|{pg:nflec}" clearly uses "|" as a separator, and the resulting substrings also have a certain pattern, such as being fixed as numbers or key-value pairs represented by colons. Based on this, this disclosure can extract the template for each variable group, that is, the common part of the variables in the variable group. The remaining part is then used as predefined sub-variables for redundancy lookup.

[0049] To obtain the template for each variable group, this disclosure presents a heuristic algorithm. The core of the algorithm is to determine the template using non-alphanumeric characters. Because developers, when defining formats, only choose characters with clear separating meanings to construct the format, making it easier for other developers to understand and use, such as common characters including but not limited to '|', ':', and '"', this disclosure uses non-alphanumeric characters as delimiters and tests their separating effect. If the effect is good, it is retained as part of the template.

[0050] Specifically, this embodiment constructs a parse tree offline on the variable groups. Initially, the root node contains all variables and is marked as separable. Then, iterates through each separable node, attempting to select a delimiter to separate the variables under that node. For example, for each variable group, all non-alphanumeric symbols in the first variable of the group are selected as candidate delimiters. Then, the delimiter candidates are traversed sequentially according to their frequency of occurrence in the first variable, starting with the most frequent candidate. The variables under a node of this candidate delimiter are then divided, and the number of sub-variables after each division is counted. If 90% of the variables under a node, after being divided by this candidate delimiter, result in the same number of sub-variables, then this candidate delimiter is considered a suitable delimiter. If the variables under a node do not meet the above condition after being divided by each candidate delimiter, then the variables under that node are considered indivisible, and the node is marked as "indivisible," thus stopping the next step. This embodiment continuously executes the splitting algorithm on the tree nodes until all nodes are marked as "indivisible," at which point the algorithm stops. It should be noted that variables whose root nodes no longer meet the above conditions can be discarded and not participate in subsequent operations; moreover, the above 90% can also be set to 80%, and this disclosure does not limit this.

[0051] As an example, Figure 5 An example of a parse tree construction process is given. Figure 5As shown, we take all variables of a variable group as variables under the root node. First, we look at the root node, which is initially separable. Therefore, we can try to choose a delimiter. The first variable includes two delimiter candidates: '|' and '_', both appearing three times. We can choose to try using '|' to split the root node first. We can see that each variable under the root node is split into four sub-variables. Since the proportion of variables with the most sub-variables (4) after splitting with the '|' delimiter reaches 100%, '|' is a suitable delimiter. Next, we traverse the nodes of the first level. Looking at the first node, we choose '_' as the delimiter, which can 100% divide all variables of this node into two sub-variables. For the second and third nodes, since these two nodes are purely numeric and do not contain any non-alphanumeric characters, they are considered indivisible and marked as leaf nodes. For the fourth node, we also choose '_' as the delimiter, which can 100% divide all variables of this node into three sub-variables. Next, we traverse the nodes of the second level. As we can see, none of the nodes in level 2 contain any non-alphanumeric characters; therefore, all nodes in this level are marked as leaf nodes. At this point, all nodes have become indivisible, and the parse tree is complete.

[0052] After the parse tree is constructed, a template extraction algorithm can be executed on the parse tree to generate a corresponding sub-template for each node. Then, an inorder traversal is performed to concatenate all the sub-templates in order according to the child node-node-child node order to obtain the complete template.

[0053] According to an exemplary embodiment of this disclosure, obtaining the template of the current variable group based on all qualified delimiters and all leaf nodes may include: for each leaf node, in response to the fact that the strings corresponding to each variable in the leaf node are the same, determining the string as a sub-template of the leaf node; in response to the fact that the strings corresponding to each variable in the leaf node are different and at least one string has a length not zero, determining a preset wildcard as a sub-template of the leaf node; in response to the fact that the strings corresponding to each variable in the leaf node are different and all strings have a length of zero, determining that the sub-template of the leaf node is empty; concatenating all sub-templates and all qualified delimiters to obtain the template of the current variable group. Through this embodiment, the template of each variable group can be obtained conveniently and quickly based on all qualified delimiters and all leaf nodes.

[0054] Specifically, Figure 6 Showing Figure 5Each node in the parse tree has undergone template replacement, preserving the structural relationship of the parse tree in the above embodiment, and replacing the content of each node. For example, for a separable node, the sub-template is the delimiter selected when constructing the parse tree; for an indivisible node, a sub-template can be generated, with three possibilities: 1. If all the strings contained in the node are the same, then the sub-template will be a constant string; 2. If at least one of the contained strings has a non-zero length, then the sub-template will be a preset wildcard (e.g., ...). Figure 6 The asterisk (*) can match any string; 3. If the length of all included strings is zero, then the sub-template is empty. After this replacement, each node receives its corresponding sub-template. Then, the traversal algorithm is executed to obtain the final template. For example... Figure 6 As shown, for separable nodes, the child template is the corresponding separator, such as '|' for the root node and '_' for the first-level node. For indivisible nodes, cases 1 and 2 arise. For example, a node containing "5206", "5304", and "2312" uses the wildcard "*" as its child template because these numbers are not identical and are not empty. Similarly, the first node in the last level contains the string "feed", so "feed" is its child template. Finally, the traversal algorithm yields the template for this variable group as "feed_photo|*|*|1_*_d57".

[0055] According to an exemplary embodiment of this disclosure, after determining the association relationships of training samples based on common variable groups in the training samples, for a first variable group with a template, in response to the fact that all association relationships between the first variable group and the training samples are irrelevant, the template of the first variable group is discarded. Through this embodiment, if a variable group without an association relationship has a template, the template can be discarded because there is no association relationship, and it will not participate in subsequent redundancy removal processing. Therefore, the template is useless, and deletion can avoid irrelevant calculations (because template matching will be performed subsequently, and redundancy removal will be performed based on the matching results), and can also reduce space usage.

[0056] As an example, after obtaining the common associations of the training samples, for some variable groups with templates, if the variable group does not involve any common associations, then the template of the variable group can be discarded to avoid irrelevant calculations and reduce space usage.

[0057] According to an exemplary embodiment of this disclosure, in an association relationship that includes the keys of the variable groups containing the source sub-variable and the target sub-variable in the same data and their respective orders in the variable groups, the target sub-variable is a predetermined sub-variable corresponding to a valid predetermined association relationship. Selecting an association relationship between variable groups containing any two keys from at least one set of keys from the association relationship may include: for each association relationship in the training samples, determining the current association relationship as a relevant association relationship related to the current data in response to the fact that both keys in the current association relationship appear in the current data and the order of the two keys in the current data is consistent with the order in the current association relationship; for each association relationship in all relevant association relationships, recording the relevant association relationship as valid in response to the fact that the variables containing the source sub-variable and the target sub-variable in the relevant association relationship both conform to the template of the corresponding variable group and the order of the source sub-variable and the target sub-variable in their respective variable groups corresponds to the same string; and determining the valid relevant association relationship as an association relationship between variable groups containing any two keys from at least one set of keys.

[0058] In this embodiment, for each piece of data, this disclosure will determine all relevant relationships related to the current data and only record whether the relevant relationships are valid. Relationships that are not related to the current data will be ignored and not recorded. This flexible design avoids the expansion of metadata.

[0059] Specifically, for each data point, the process first retrieves all relevant relationships associated with the current data from the common relationship set obtained during the training phase. For example, for each common relationship, if both keys in the current relationship appear in the current data and their order in the current data matches the order in the current relationship, then this common relationship is relevant to the current data. Next, for each relevant relationship, the process first checks whether the variables containing the source and target sub-variables conform to the corresponding template. If either variable does not conform to the template, the relevant relationship is invalid. If both variables conform to the template, string matching is performed, i.e., checking whether the content of the source and target sub-variables is the same. If they are not completely identical, the relevant relationship is invalid; if they are completely identical, the relevant relationship is valid.

[0060] return Figure 2 In step S203, the variable group of at least one key is deredundant based on the association relationship to obtain the processed variable group of at least one key.

[0061] Specifically, before redundancy removal, variable groups can be categorized into four types: PlainVar, ListVar, SketchVar, and RareVar. First, if the keys of a variable group are not common keys encountered during training, its type is determined to be RareVar. For the remaining variable groups, if more than 50% of the content in a variable group is a JSON array, then that variable group is of type ListVar. For variable groups that are not of type ListVar, template matching can be performed. If a template is matched, the type of this variable group is SketchVar; if no template is matched, then the type of this variable group is PlainVar.

[0062] It should be noted that the type of the variable groups in the training samples can be determined during the training phase. Specifically, after obtaining the common variable groups during the training phase, the type of the variable group corresponding to the abnormal key is first determined to be RareVar. For the remaining variable groups, if more than 50% of the content in a variable group is of JSON array type, then the variable group is of type ListVar. For variable groups that are not of type ListVar, template recognition can be performed on the variable group. If a template is recognized, then the type of this variable group is SketchVar; if no template is recognized, then the type of this variable group is PlainVar.

[0063] The compression method differs for each type of variable group in the subsequent compression stage. Specifically, PlainVar type variable groups are ordinary types, and all variables of this type are compressed in their original form; ListVar type variable groups store JSON arrays, and the array structure is further parsed, with each piece of data after parsing being compressed according to the method disclosed herein; SketchVar type variable groups match each variable against the corresponding template, and for variables that match successfully, only the sub-variable part is retained and redundancy is removed, while variables that do not match successfully are compressed in their original form, and metadata on whether a match was successful is recorded; RareVar type variable groups, in addition to retaining the value, also record the key corresponding to each value, i.e., they are compressed in their original form. This approach meets the different needs when storing different variables. ListVar can further structure nested arrays in the data, which can significantly improve the compression rate for data with many nested arrays. SketchVar can perform fine-grained parsing of variables with patterns, which can further improve the compression rate and facilitate the extraction of relationships. RareVar can handle outliers with low frequency.

[0064] The redundancy removal process is explained in detail below: According to an exemplary embodiment of this disclosure, redundancy removal processing is performed on a variable group with at least one key based on the association relationship to obtain a processed variable group with at least one key. This may include: for a second variable group with an association relationship, in response to the second variable group being the variable group containing the target sub-variable in the association relationship, deleting the string corresponding to the order of the target sub-variable in the second variable group; wherein the number of variables of type JSON array in the second variable group does not exceed a preset value and a template exists. Through this embodiment, for the second variable group, when partial redundancy is found in the corresponding association relationship, only one string is retained, reducing the compression amount.

[0065] As an example, for the second variable group with a valid correlation, i.e. the variable group of SketchVar mentioned above, if the second variable group is the variable group where the target sub-variable in the correlation is located, delete the string corresponding to the order of the target sub-variable in the second variable group, and only keep the string corresponding to the source sub-variable in the correlation.

[0066] It should be noted that, in addition to operating on the variable group, additional metadata information needs to be recorded to ensure the reversibility of the recovery process. Specifically, for each data entry, this disclosure ignores irrelevant relationships and records relevant relationships. For each relevant relationship, its validity is also recorded. During decompression, the data can be retrieved using the metadata. Specifically, first, all relevant relationships are identified based on the variables contained in the data entry; then, the validity of these relevant relationships is checked; finally, if the record is valid, the data at the source sub-variable location is copied to the target sub-variable location, completing the data recovery.

[0067] For variable groups that do not have a correlation, additional processing is required, specifically: According to an exemplary embodiment of this disclosure, for a third variable group without any association, the third variable group is split into multiple JSON object data. Each JSON object data is then independently deredundant using the complete data compression method of this disclosure. The number of JSON array type variables in the third variable group exceeds a preset value. For a fourth variable group without any association, the original form of the fourth variable group is used as the processed fourth variable group. The number of JSON array type variables in the fourth variable group does not exceed the preset value and there is no template. Through this embodiment, this disclosure also provides a corresponding processing method for variable groups without any association, thereby achieving complete data compression.

[0068] It should be noted that the third variable group mentioned above is the same as the variable group of type ListVar mentioned above; the fourth variable group mentioned above is the same as the variable group of type RareVar mentioned above and the variable group of type PlainVar mentioned above.

[0069] Since the parsing process described above only involves reading and deleting the last element of each variable group, a regular array container can perform these operations very efficiently. When recording this data, no copying is required; instead, the container records the start and end offsets of each element in memory. During final encoding, this data is copied all at once. This design avoids both the overhead of frequently allocating small amounts of memory and many ineffective memory copies.

[0070] In step S204, the variable group of the association relationship and at least one key after processing is compressed to obtain the compressed data of the current data.

[0071] Specifically, during compression, the variable data portion occupies the majority of the entire encoding space and is the most important part. However, for the subsequent decompression process to be reversible, various metadata also needs to be stored. In addition to relationships, the metadata portion can also include, but is not limited to, the following: object type, object template, array length, key-value pair separator, relationship validity, variable group template, etc. It should be noted that the above metadata can be recorded at the object level. For example, JSON objects and ordinary key-value pairs, one data record corresponds to one object; for JSON arrays, one data record may correspond to multiple objects.

[0072] The aforementioned object types are used to record the semi-structured form corresponding to each piece of data. This disclosure can handle multiple data types, including JSON objects, JSON arrays, and ordinary key-value pairs, while also being compatible with ordinary strings, i.e., categorized into variables of the PlainVar type. The object template is used to record which variables each object contains and the order of all variable names within that object. The array length is specifically for JSON array types; in this disclosure, the array length needs to record the total number of JSON objects contained in the array. The key-value pair separator is for ordinary key-value pair types; this disclosure needs to record the separator used when parsing each ordinary key-value pair data. The validity of association relationships serves the inline deduplication function. In each piece of data, this disclosure records all relevant association relationships and their validity. If an association relationship is irrelevant to this data, it will be ignored and not recorded. This flexible design avoids metadata bloat.

[0073] It should be noted that, considering that object templates are relatively sparse in semi-structured data, using a dictionary can reduce the overall space. Therefore, this disclosure uses a dictionary to record all possible object templates and the index of each object template in the dictionary. If two data entries contain completely identical variable names, then the recorded object template numbers will also be identical. Other metadata can also be encoded using dictionary encoding, and this disclosure does not impose any restrictions on this.

[0074] According to an exemplary embodiment of this disclosure, the above-described compression of the variable groups of the association relationship and the processed at least one key to obtain the compressed data of the current data may include: for each variable group in the processed at least one key variable group, performing the following processing: obtaining a predetermined number of variables in the current variable group as variable samples; compressing the variable samples using the original encoding method and the dictionary encoding method respectively, and determining the encoding method with the smaller compressed data volume as the encoding method of the current variable group; encoding the current variable group using the determined encoding method to obtain the encoded current variable group; encoding the association relationship using the dictionary encoding method to obtain the encoded association relationship; and bit-packing all the encoded variable groups and the encoded association relationship to obtain the compressed data of the current data. This design in this embodiment allows encoding selection to be completed with low overhead.

[0075] Specifically, this disclosure implements a dynamic selection method when determining the encoding type for each variable group. This involves choosing between the original encoding and dictionary encoding methods. Original encoding records all variables sequentially, separated by a fixed delimiter (such as a newline character). Dictionary encoding records data in two parts: a dictionary directory entry and a dictionary index entry. The dictionary directory entry records unique elements, and the dictionary index entry records the corresponding number. It's important to note that for the same variable group, different encoding methods result in different compressed data lengths. For example, dictionary encoding yields shorter data for some data, while original encoding yields shorter data for others. Generally, dictionary encoding performs better for sparse data; however, for data without duplicates, dictionary encoding offers no benefit and, compared to original encoding, the additional data recorded for index entries actually decreases the compression ratio. Therefore, this disclosure samples a portion of the data from the variable group (e.g., 1% of the data), encodes and compresses the data sample using two different encoding methods, compares the size of the compressed data, and selects the encoding method that performs better on the data sample as the final encoding method. Through this design, the encoding selection can be completed with low overhead.

[0076] According to an exemplary embodiment of this disclosure, when the determined encoding method is dictionary encoding, the above-mentioned bit-packing of the encoded variable group and the encoded association to obtain the compressed data of the current data may include: for each encoded data, performing the following processing, wherein an encoded data is an encoded variable group or an encoded association: obtaining the byte width of the element in the dictionary of the current encoded data; when the byte width is greater than 8, determining the minimum value among multiples of 8 that is greater than the byte width as the number of bits occupied when packing the element; when the byte width is less than or equal to 8, determining the minimum value among integer powers of 2 that is greater than the byte width as the number of bits occupied when packing the element; bit-packing the current encoded data based on the number of bits, wherein the empty part of the number of bits of each element during bit packing is padded with 0. Through this embodiment, the packing of elements within a byte is realized, and the padding mechanism avoids the cross-byte storage of elements, enabling byte-level alignment during the encoding process, thereby synchronizing the operation granularity of the encoding and compression processes, maintaining the semantics of byte granularity, and obtaining a better compression ratio compared with existing methods.

[0077] The following is a brief explanation of bit-packing methods in related technologies to introduce the byte-aware bit-width adjustment algorithm of this disclosure: In current dictionary encoding processes, the width of bit packing can lead to byte granularity misalignment, further resulting in a decrease in subsequent compression ratio. Specifically, if the cardinality of values ​​in the data differs significantly, structuring will bring greater improvement to the compression ratio of such data. To ensure the compression ratio of variable groups with smaller cardinality, previous works such as Parquet and ORC have used dictionary encoding to process this type of data, that is, using the same number to represent consistent content. The data is encoded into two parts: 1. Dictionary entries, which contain all unique elements after data deduplication; 2. Dictionary indexes, mapping each element to the corresponding element number in the dictionary entry, i.e., the index in the dictionary entry part. Through index mapping, the actual element corresponding to the data can be found. Typically, since the dictionary index part is an integer, these numbers are stored using bit-packing techniques to minimize space usage. If there are a total of N dictionary entries, then each number after bit packing will occupy... log2 Each bit. For example, assuming there are only 5 unique elements in the data, if bit packing is used, then each index occupies only 5 bits. log25 = 3 bits; without bit packing, using only regular integer encoding, each index would occupy 4 bytes. Therefore, after using bit packing, the space usage is only 3 bits ÷ (4 bytes × 8 bits / byte) = 9.375% of the previous amount. After this encoding, the dictionary entries and dictionary indexes will be further compressed by a byte-level compressor. Common byte-level compressors include Zstandard, LZMA, and gzip.

[0078] However, currently, dictionary encoding and byte-level compression are relatively independent, resulting in inconsistent operational granularity: dictionary encoding operates on data at the bit level, while compression operates at the byte level. Therefore, even if dictionary encoding achieves optimal compression ratio, the final compression result is suboptimal. Figure 7 Let's take an example to illustrate this problem. Figure 7 The image shows three elements, each represented by a different color, and each element occupies 7 bits. The three elements are arranged closely together from the beginning, occupying every available bit. The biggest problem with this design is that the content represented by each byte, from the element level, is unstable. For example, Figure 7 The first byte contains all 7 bits of the first element and the lower 1 bit of the second element. The second byte contains the higher 6 bits of the second element and the lower 2 bits of the third element. The third byte contains the higher 5 bits of the third element and the lower 3 bits of the fourth element, and so on. It can be seen that the semantic information between adjacent bytes is extremely unstable; only one byte containing the same semantic information appears every seven bytes. For the compressor, compression is based on continuous redundancy at the byte level. If there are significant differences in the distribution between consecutive bytes, the probability of continuous redundancy will decrease, resulting in a decrease in compression efficiency.

[0079] Therefore, this disclosure proposes a byte-aware bit-width scheduling method to adjust the bit width during bit-packing encoding, thereby improving the compression ratio. The basic principle of its design is to ensure the semantic information of each byte is as stable as possible while minimizing the space occupied. The algorithm logic may include, but is not limited to, the following:

[0080] First, calculate the original byte width of the element. =log2 Next, we will discuss two cases: when When the value is greater than 8, use the smallest multiple of 8 that is greater than the byte width for bit packing; when... At 8, the smallest power of 2 greater than the byte width is used for bit packing.

[0081] As an example, Figure 8 This demonstrates a byte-aware bit-packing encoding method, such as Figure 8 As shown, when applying byte-aware bit-packing encoding, two possible scenarios arise: padding and packing. Specifically, assuming a bit width of 5, the bit width is adjusted to the nearest power of 2, meaning each element occupies 8 bits. Any unfilled portions are padded with 0s. In this case, each element occupies a complete byte, avoiding the situation where a single element preceding it (as shown in the upper left part of the figure) spans bytes. Conversely, assuming a bit width of 3, the bit width is adjusted to the nearest power of 2, meaning each element occupies 4 bits. In this case, each byte contains two complete elements. Therefore, this embodiment both packs elements within a byte, saving space, and uses padding to avoid storing elements across bytes.

[0082] It should be noted that the byte width in this embodiment is adjusted upwards to a suitable size, rather than simply selecting the minimum encoding width. Although this encoding method causes some additional padding, resulting in the encoded data occupying more space, this padding is immediately absorbed by the compressor. Final experimental results show that this embodiment achieves good results when the data has a significantly different cardinality distribution.

[0083] According to exemplary embodiments of this disclosure, when the log data to be compressed is ordinary string data or key-value pair data, this disclosure also provides corresponding processing methods: when the log data to be compressed is ordinary string data, the log data to be compressed is parsed into a variable group without a template, and is compressed in the manner of a fourth variable group; when the log data to be compressed is key-value pair data, the log data to be compressed is parsed into a variable group of at least one key, and is compressed in the manner of a second variable group or a fourth variable group, wherein, along with the association relationship and the processed variable group of at least one key, the group separator between the two key-value pairs is also compressed.

[0084] Figure 9 This is a flowchart illustrating a data decompression method according to exemplary embodiments of the present disclosure, such as... Figure 9 As shown, for each piece of data in the log data to be decompressed, the following processing is performed: In step S901, the compressed data of the current data is decompressed to obtain the association relationship of the current data and a variable group of at least one key after processing; In step S902, based on the association relationship, the redundant parts removed from the variable group of at least one key after the recovery process are obtained to obtain the variable group of at least one key after the recovery process. In step S903, the current data is obtained based on the variable group of at least one key after recovery.

[0085] It should be noted that the recovery process also relies on the aforementioned metadata, but this disclosure does not impose any limitations on it.

[0086] To better understand this disclosure, the following is in conjunction with... Figure 10 Provide a systematic explanation. For example... Figure 10 As shown, this disclosure provides a compressor for semi-structured data. The compressor's workflow can be divided into three stages: a training stage, a compression stage, and a decompression stage. The figure only shows the training and compression stages. In the training stage, training samples are obtained by sampling the data to be compressed, and training is performed on these samples, i.e., template recognition and relation extraction, resulting in commonly used variable names (i.e., high-frequency variable names in the figure) and commonly used relations. Next, in the compression stage, all the data to be compressed is parsed, the corresponding structured results are saved, relevant metadata is recorded, and redundancy is removed based on common variable names and common relations. The deredundant variable groups are then packaged and compressed. Finally, in the decompression stage, the data is restored based on the data and metadata recorded in the compression stage. The following describes each of the three stages in detail: Training Phase: First, training samples are obtained by upsampling the data to be compressed. A scan of the training samples is performed to obtain all variable group names (i.e., keys), and the frequency of each variable name in the training samples is calculated. Frequently occurring variable groups are identified as common variable groups. Next, the types of these variable groups are determined (PlainVar, ListVar, SketchVar, RareVar). Finally, another scan of the training samples is performed to extract associations. Here, only SketchVar type variable groups are considered; other types can be ignored. After obtaining the common variable groups and common associations, the variable group types can be simplified: for SketchVar type variable groups, if the variable group is unrelated to all common associations, its type is converted to PlainVar, and the template is discarded. This avoids useless calculations.

[0087] Compression stage: Each piece of data in the data to be compressed is parsed sequentially, the parsing results are stored in memory, and the parsing results are encoded. Finally, a byte compression tool is used to compress the encoded results. The parsing results that need to be stored mainly include two aspects: variable data and metadata. Variable data refers to the actual content of the data, that is, all key-value pairs; metadata refers to structure-related information, recording the order and form in which these variables should be organized.

[0088] It should be noted that this disclosure uses different methods for different types of data to be stored, as detailed below: JSON objects: Each piece of data is parsed, storing all variables in their corresponding variable groups. Then, all relationships related to the current data are identified, and redundancy is removed based on valid relationships. Regarding metadata, the object type and object template are updated by adding the JSON object type and its corresponding template index, and all relevant relationships are recorded, along with their validity.

[0089] Plain string: Variables of this type can be saved in PlainVar. In the metadata, the object type will be updated to plain string, and the object template will be updated to its index in the entire variable group.

[0090] JSON Arrays: Upon recognizing the data as a JSON array, the system first adds the number of objects contained in the array (obtained during parsing) to the array length metadata, and simultaneously records the JSON array type in the object type. Next, the invention parses each JSON object sequentially and updates the metadata accordingly.

[0091] Key-value pairs: A dynamic delimiter recognition algorithm is used to identify delimiters between and within groups, and delimiting is performed based on these delimiters. All variables are stored in their corresponding variable groups. Then, all relationships related to the current data are identified, and redundancy is removed based on the valid relationships. Next, the object type and object template are updated by adding the corresponding key-value pair type and template information. Finally, the key-value pair delimiters are updated by adding the identified inter-group delimiters.

[0092] After parsing the data, the parsing results in memory also need to be encoded. For metadata, dictionary encoding can be used, and bit-packing technology can be used for packaging to make the expression as concise as possible. For variable data, the encoding method for each variable group is dynamically determined, and a fixed encoding method is used for encoding, followed by bit-packing technology. It should be noted that Zstandard can be used for compression of encoded data, and this disclosure does not limit it. Zstandard is currently the most common byte packing tool, which can provide a relatively good compression ratio-compression speed performance curve; considering that this disclosure has already performed sufficient deduplication in the structuring stage, Zstandard level 1 is selected here to avoid the compression tool performing excessive redundant calculations.

[0093] Decompression stage: Data recovery is required based on the compressed binary file. To achieve efficient decompression, no memory comparisons are needed (compression requires multiple comparisons to determine the final mapping location), and only the necessary data is copied to the corresponding location once. This design makes decompression several times faster than compression. The decompression workflow is detailed below: First, memory state initialization is completed by reading the compressed binary file into memory and executing the decompression algorithm. Next, the starting position of each part, including variable and metadata parts, is found according to the rules agreed upon during encoding. During decoding, the next element is read sequentially from the pointer of each part as needed. Specifically, the data type can be read first, and different recovery strategies can be adopted according to different types. JSON Objects: After accessing this type of data, for each data entry, the template index of the current object can be obtained, and the object template of the current object can be retrieved based on the template index. Then, based on the variable names (i.e., keys) in the object template, the variable content is read from the corresponding positions and restored. After obtaining all the restored key-value pairs, the necessary structural symbols are added between the key-value pairs according to the organization rules of JSON data to complete the data reconstruction.

[0094] For ordinary strings: simply access the next record from the special variable group and copy it directly to the output position; no other operations are required.

[0095] For JSON arrays: First, access the metadata that records the array length and find the number of objects corresponding to that record; then, restore each object in turn and concatenate them according to the rules of JSON arrays.

[0096] Key-value pairs: Similar to JSON objects, it also requires obtaining the template index of the current object, retrieving the object template based on the template index, and then reading the variable content from the corresponding location based on the variable name (i.e., the key) in the object template and restoring it. It also needs to access the group separator metadata to obtain the group separator determined during parsing. Finally, it is restored in the form of "key-intra-group separator-value-inter-group separator".

[0097] According to embodiments of this disclosure, the method of this disclosure is integrated into the Parquet framework to obtain ColZip encoding. ColZip encoding is used to implement the data compression method described in the above embodiments. ColZip encoding is conceptually similar to an encoding method that can re-encode a block of memory containing a large number of data records, rearranging the data according to the ColZip algorithm. Therefore, a ColZipEncoder is added to the Parquet framework, and the following three parts are implemented: `void write(DataRecord record)`: ColZipEncoder needs to implement the function of writing data records, that is, placing the data in a buffer and using special markers (such as newline characters) as delimiters. ColZip also needs to aggregate the data and perform operations in batch processing. Therefore, the data is not written directly to the output, but to the buffer, and processed together later.

[0098] `int getBufferedSize()`: The Parquet framework queries each Encoder for the current page size to determine if paging should occur. `ColZipEncoder` reports the buffer size as the return value. Furthermore, to prevent buffer overflows, `ColZipEncoder` sets a memory threshold. If the data in the current buffer exceeds the threshold, it reports a value significantly larger than the default `PageSize` to trigger paging. This is because the Parquet framework does not call this function to check the buffer size every time data is written; therefore, `ColZipEncoder` needs to make this judgment in advance.

[0099] Binary flush(): After a paging operation is triggered in the Parquet framework, ColZipEncoder needs to encode the data in the buffer using ColZip and return the binary encoded data. In a C++ framework, the library functions implemented in the prototype system of this disclosure can be called directly. In a Java framework, JNI calls are required to complete the C++ function calls. This disclosure wraps the functionality of ColZip and implements four library functions for Java to call.

[0100] In summary, this disclosure provides a relation-aware structured design and a byte-aware encoding design. The relation-aware structured design addresses the issue of reduced compression ratio after structuring due to the presence of relations. Relationship-aware structuring generally involves three processes: template identification, relation extraction, and relation utilization. First, the template of the variable group is identified, and the variables within it are broken down into finer parts according to the corresponding template. Then, common relations between the variable groups are extracted. The first two processes are performed offline; this disclosure uses a subset of the data to be stored for training. Finally, when parsing each piece of data, the obtained relations are used to remove redundancy, avoiding the storage of unnecessary parts. The byte-aware encoding design re-examines the width issue of bit packing during dictionary encoding, solving the problem of byte granularity misalignment caused by existing methods, which further leads to a decrease in subsequent compression ratio. The byte-aware encoding design, by performing byte alignment during the encoding process, synchronizes the operational granularity of the encoding and compression processes, achieving byte granularity alignment during encoding, maintaining the semantics of byte granularity, and obtaining a better compression ratio compared to existing methods. This disclosure also describes the work done in integrating the ColZip algorithm into the Parquet framework. The ColZip system supports end-to-end high-density compression of various semi-structured data types, including JSON objects, JSON arrays, and key-value pairs. The system's workflow for compressing semi-structured data can be divided into two phases: training and compression. During the training phase, ColZip identifies common keys in the data and uncovers potential relationships. During the compression phase, ColZip parses each data record sequentially, stores the data in the corresponding template, and employs techniques such as relationship-based redundancy removal and byte-aware encoding to achieve the final compression.

[0101] To demonstrate the feasibility of the method disclosed herein, ColZip underwent comprehensive performance testing through extensive experiments on both production and public datasets. Test results show that, compared to the general-purpose compression tool Zstandard, ColZip achieves a compression ratio of up to 2.98 times that of zstd-low and up to 2.26 times that of zstd-high, at similar bandwidths. This is comparable to the best current semi-structured data compression systems. Compared to Slope, ColZip's compression speed is basically the same. The slope is the same, but ColZip can handle more types of data, and the compression ratio can reach [missing information] on production datasets. ColZip achieves 1.21 to 2.50 times the compression ratio of Slope, and 1.11 to 2.84 times the compression ratio on public datasets. Furthermore, ColZip's decompression speed is... 6.09-9.95 times that of the Slope.

[0102] Figure 11 This is a block diagram illustrating a data compression apparatus according to exemplary embodiments of the present disclosure. (Refer to...) Figure 11 The device includes: a first processing unit 110, configured to compress each piece of data in the log data to be compressed through the following subunits: a parsing subunit 1102, configured to parse the current data to obtain variable groups of at least one key, wherein each variable group contains multiple variables, one of which is a value of the key corresponding to the variable group; a determining subunit 1104, configured to determine the association relationship between any two variable groups of at least one key, wherein the association relationship indicates redundant content between the variable groups of the two keys; a redundancy removal subunit 1106, configured to perform redundancy removal processing on the variable groups of at least one key based on the association relationship to obtain a processed variable group of at least one key; and a compression subunit 1108, configured to compress the association relationship and the processed variable group of at least one key to obtain compressed data of the current data.

[0103] Figure 12 This is a block diagram illustrating a data decompression apparatus according to exemplary embodiments of the present disclosure. (Refer to...) Figure 12 The device includes: a second processing unit 120, configured to decompress each piece of data in the log data to be decompressed through the following sub-units: a decompression sub-unit 1202, configured to decompress the compressed data of the current data to obtain the association relationship of the current data and a variable group of at least one key after processing; a recovery sub-unit 1204, configured to recover the redundant parts removed from the variable group of at least one key after processing based on the association relationship to obtain a variable group of at least one key after recovery; and an acquisition sub-unit 1206, configured to obtain the current data based on the variable group of at least one key after recovery.

[0104] Figure 13 A computing environment 1310 coupled to a user interface 1350 is shown. The computing environment 1310 may be part of a data processing server. The computing environment 1310 includes a processor 1320, memory 1330, and input / output (I / O) interface 1340.

[0105] Processor 1320 typically controls the overall operation of computing environment 1310, such as operations associated with display, data acquisition, data communication, and image processing. Processor 1320 may include one or more processors for executing instructions to perform all or some of the steps in the methods described above. Furthermore, processor 1320 may include one or more modules that facilitate interaction between processor 1320 and other components. The processor may be a central processing unit (CPU), microprocessor, microcontroller, graphics processing unit (GPU), etc.

[0106] Memory 1330 is configured to store various types of data to support the operation of computing environment 1310. Memory 1330 may include predefined software 1332. Examples of such data include instructions for any application or method operating on computing environment 1310, video datasets, image data, etc. Memory 1330 can be implemented using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0107] I / O interface 1340 provides an interface between processor 1320 and peripheral interface modules (such as keyboard, click wheel, buttons, etc.). Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I / O interface 1340 can be coupled to encoders and decoders.

[0108] In an embodiment, the computing environment 1310 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the methods described above.

[0109] According to embodiments of the present disclosure, an electronic device may be provided, the electronic device including at least one memory and at least one processor, the at least one memory storing a set of computer-executable instructions, which, when executed by the at least one processor, perform a method according to embodiments of the present disclosure.

[0110] As an example, the electronic device can be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, the electronic device 1000 is not necessarily a single electronic device, but can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The electronic device can also be part of an integrated control system or system manager, or can be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0111] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device may be interconnected via buses and / or networks.

[0112] According to embodiments of this disclosure, a computer-readable storage medium may also be provided, wherein when instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes to perform the method of the embodiments of this disclosure. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0113] According to embodiments of this disclosure, a computer program product may also be provided, including computer instructions that, when executed by a processor, implement the methods of embodiments of this disclosure.

[0114] Unless otherwise specifically stated, the order of steps in the method according to this disclosure is intended to be illustrative only, and the steps of the method according to this disclosure are not limited to the specific order described above, but may be changed according to actual circumstances. Furthermore, at least one step in the method according to this disclosure may be adjusted, combined, or omitted as needed.

[0115] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0116] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data compression method, characterized in that, include: For each piece of data in the log data to be compressed, perform the following processing: The current data is parsed to obtain at least one set of variables with a key. Each set of variables contains multiple variables, and each variable is a value of the key corresponding to the set of variables. Determine the association relationship between variable groups of any two of the at least one keys, wherein the association relationship indicates redundant content between the variable groups of the two keys; Based on the association relationship, the variable group of the at least one key is deredundant to obtain the processed variable group of the at least one key. The variable group containing the association relationship and at least one key after processing is compressed to obtain the compressed data of the current data.

2. The data compression method as described in claim 1, characterized in that, Before processing each piece of data in the log data to be compressed, the process also includes: A predetermined number of data entries are obtained from the log data to be compressed and used as training samples; Based on the common variable groups of the training samples, the association relationship of the training samples is determined, wherein the common variable groups are the variable groups of keys that appear in the training samples with a frequency of up to a first predetermined threshold. Wherein, determining the association relationship between variable groups of any two keys among the at least one key includes: Select the association relationship between any two of the at least one keys from the association relationship.

3. The data compression method as described in claim 2, characterized in that, Determining the association relationships of the training samples based on common variable groups in the training samples includes: Select variable groups from the common variable groups whose number of JSON array type variables does not exceed a preset value, and use them as the first variable group; Each variable in the first variable group with a template is split according to the corresponding template to obtain the predetermined sub-variables of each first variable group. The predetermined sub-variables of the first variable group correspond to wildcards in the corresponding templates. The corresponding template of the first variable group includes a non-alphanumeric symbol of the first variable in the first variable group, one of the multiple identical sub-variables after all variables in the first variable group are divided according to the symbol, and multiple different sub-variables represented by the same wildcard. The first occurrence of any identical sub-variable among all predefined sub-variables is identified as the source sub-variable of that identical sub-variable; In response to the fact that the keys corresponding to the variable groups to which the source sub-variable and any predetermined sub-variable belong are different and the order meets the requirements, it is determined that the predetermined association relationship occurs once, wherein the order meets the requirements means that the current order of the two keys is consistent with the order in which they appear in the current data; In response to the fact that the variables in which the source sub-variable and any predetermined sub-variable each reside conform to the template of the corresponding variable group and the content of the source sub-variable and any predetermined sub-variable are the same, the predetermined association relationship is recorded as valid and the valid value of the predetermined association relationship is incremented by 1; When the ratio of the valid value of the predetermined association to the total number of occurrences of the predetermined association reaches a second predetermined threshold, the predetermined association is determined to be the association of the training sample.

4. The data compression method as described in claim 3, characterized in that, Before splitting the variables of each first variable group with a template according to the corresponding template to obtain the predetermined sub-variables of each first variable group, the process also includes: For each first variable group, the non-alphanumeric symbols contained in the first variable in the current variable group are determined as delimiter candidates; Based on the order of frequency of each delimiter candidate from highest to lowest, the following processing is performed on each delimiter candidate in turn: The predetermined node is divided according to the current delimiter candidate to obtain multiple child nodes. Each child node contains the first child variable in the same division order after all variables in the predetermined node are divided. In the first processing, the predetermined node is the current variable group. In response to the number of predetermined variables in the predetermined node exceeding a preset number, the current delimiter candidate is determined as a qualified delimiter, wherein all predetermined variables contain the same number of child nodes; If the number of predetermined variables in the predetermined node does not exceed the preset number, the current delimiter candidate is discarded. The child nodes that do not contain any separators among the multiple child nodes are determined as leaf nodes; The next delimiter is determined as the current delimiter in the next process, and each child node other than the leaf node is determined as a predetermined node in the next process, until all nodes are leaf nodes; Based on all qualified delimiters and all leaf nodes, the template for the current variable group is obtained; In response to all delimiter candidates being discarded, it is determined that the current variable group does not have a template.

5. The data compression method as described in claim 4, characterized in that, The template for the current variable group, obtained based on all qualified delimiters and all leaf nodes, includes: For each leaf node, in response to the fact that the string corresponding to each variable in the leaf node is the same, the string is determined as the sub-template of the leaf node; In response to the fact that the strings corresponding to each variable in the leaf node are different and at least one string has a length that is not zero, the preset wildcard is determined as the sub-template of the leaf node; In response to the fact that the strings corresponding to each variable in the leaf node are different and the length of all strings is zero, it is determined that the sub-template of the leaf node is empty; Concatenate all sub-templates and all qualified separators to obtain the template for the current variable group.

6. The data compression method as described in claim 3, characterized in that, The association relationship includes the keys of the variable groups containing the source and target sub-variables within the same data entry, and their respective orders within those variable groups. The target sub-variable is a predefined sub-variable corresponding to a valid predefined association relationship. The step of selecting the association relationship between variable groups of any two keys from the at least one key from the association relationship includes: For each association in the association relationships of the training samples, in response to the fact that both keys in the current association appear in the current data and the order of the two keys in the current data is consistent with the order in the current association relationship, the current association relationship is determined to be a relevant association relationship related to the current data; For each of all relevant relationships, in response to the fact that the variables containing the source sub-variable and the target sub-variable in the relevant relationship all conform to the template of the corresponding variable group and the order of the source sub-variable and the target sub-variable are the same in the corresponding strings of their respective variable groups, the relevant relationship is recorded as valid; A valid correlation is defined as the correlation between variable groups of any two of the at least one key.

7. The data compression method as described in claim 6, characterized in that, The step of performing redundancy removal processing on the variable group of at least one key based on the association relationship to obtain the processed variable group of at least one key includes: For a second variable group with the aforementioned association, in response to the second variable group being the variable group containing the target sub-variable in the association, the string corresponding to the order of the target sub-variable in the second variable group is deleted; wherein, the number of JSON array type variables in the second variable group does not exceed the preset value and a template exists.

8. The data compression method as described in claim 1, characterized in that, The step of compressing the variable group of the association relationship and at least one key after processing to obtain the compressed data of the current data includes: For each variable group in the processed variable group with at least one key, perform the following processing: Obtain a predetermined number of variables from the current variable group as a variable sample; The variable samples are compressed using both the original encoding method and the dictionary encoding method, and the encoding method with the smaller compressed data volume is determined as the encoding method for the current variable group; The current variable group is encoded using a defined encoding method to obtain the encoded current variable group; The association relationship is encoded using dictionary encoding to obtain the encoded association relationship; Bit packing is performed on all encoded variable groups and encoded associations to obtain the compressed data of the current data.

9. The data compression method as described in claim 8, characterized in that, When the encoding method is determined to be dictionary encoding, the step of bit-packing the encoded variable groups and the encoded associations to obtain the compressed data of the current data includes: For each piece of encoded data, perform the following processing, where each piece of encoded data is either a group of encoded variables or a group of encoded relationships: Get the byte width of each element in the dictionary of the currently encoded data; When the byte width is greater than 8, the smallest multiple of 8 that is greater than the byte width is determined as the number of bits occupied when the element is packed. When the byte width is less than or equal to 8, the smallest power of 2 that is greater than the byte width is determined as the number of bits occupied when the element is packed. Based on the number of bits, the current encoded data is bit-packed, wherein, during bit packing, any empty bits in each element are padded with 0s.

10. A data decompression method, characterized in that, include: For each data item in the data to be decompressed, perform the following processing: Decompress the compressed data to obtain the association relationship of the current data and a variable group with at least one key after processing; Based on the aforementioned association, the redundant parts removed from the variable group of at least one key after recovery processing are obtained to obtain the variable group of at least one key after recovery. The current data is obtained based on a variable group with at least one key after recovery.

11. A data compression device, characterized in that, include: The first processing unit is configured to perform compression processing on each piece of data in the log data to be compressed through the following sub-units: The parsing subunit is configured to parse the current data to obtain at least one set of variables with a key, wherein each set of variables contains multiple variables, and one variable is a value of the key corresponding to the set of variables; A determining subunit is configured to determine an association between variable groups of any two of the at least one keys, wherein the association indicates redundant content between the variable groups of the two keys; The redundancy removal subunit is configured to perform redundancy removal processing on the variable group of the at least one key based on the association relationship, to obtain the processed variable group of the at least one key. A compression subunit is configured to compress the variable group of the association relationship and at least one key after processing to obtain the compressed data of the current data.

12. A data decompression apparatus, characterized in that, include: The second processing unit is configured to perform decompression processing on each piece of data in the log data to be decompressed through the following sub-units: The decompression subunit is configured to decompress the compressed data of the current data to obtain the association relationship of the current data and a variable group of at least one key after processing; The recovery subunit is configured to recover the redundant portion removed from the variable group of at least one key after processing, based on the association relationship, to obtain the recovered variable group of at least one key. Obtain the sub-unit, configured as a variable group based on at least one key after recovery, to obtain the current data.

13. An electronic device, characterized in that, include: At least one processor; At least one memory that stores computer-executable instructions. The computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the processor to perform the method as described in any one of claims 1 to 10.

15. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method as described in any one of claims 1 to 10.

Images