Data compression method and device, equipment, medium and product

Abstract

The invention provides a data compression method and device, equipment, a medium and a product, and relates to the field of data transmission. Comprising the following steps: acquiring transmission data transmitted in a single batch; if the size of the transmission data is greater than a preset first compression threshold value, querying corresponding meta-information; performing classification processing on fields in the transmission data to obtain a classification data set; performing screening processing according to a preset second compression threshold value to obtain a compression target set; collecting feature information of each compressed target, and calculating according to a preset feature set and the feature information to obtain a feature value set of the compressed targets; performing matching processing according to a preset algorithm matching strategy and the characteristic value set, and determining a candidate compression algorithm set of the compression target; and according to a preset algorithm selection strategy, determining an optimal compression algorithm of the compression target from the candidate compression algorithm set. According to the method and the device, the technical problem that the data compression efficiency is relatively low due to the fact that the prior art is easily influenced by mixed data characteristics and the compression algorithm selection is single is solved.

Description

Technical Field

[0001] This application relates to the field of data transmission, and more particularly to a data compression method, apparatus, device, medium, and product. Background Technology

[0002] In distributed database systems, cloud database services, and large-scale data migration scenarios, the data transmission efficiency between the database front-end and back-end or between nodes directly impacts the overall system performance. Traditional data transmission methods, when bandwidth is insufficient, rely on costly hardware upgrades or network optimizations with limited effectiveness. In such cases, reducing transmission volume through data compression becomes a key method for improving transmission efficiency.

[0003] Currently, data compression is mainly achieved by pre-configuring fixed compression algorithms for specific types of data or by dynamically adjusting compression algorithms based on network latency or bandwidth fluctuations.

[0004] However, existing technologies are susceptible to the characteristics of mixed data and the selection of a single compression algorithm results in low data compression efficiency. Summary of the Invention

[0005] This application provides a data compression method, apparatus, device, medium, and product to solve the problem that the prior art is easily affected by mixed data characteristics and has a single compression algorithm, resulting in low data compression efficiency.

[0006] In a first aspect, this application provides a data compression method, comprising:

[0007] Obtain the transmission data for a single batch of transmissions;

[0008] If the size of the transmitted data exceeds the preset first compression threshold, then the corresponding metadata is queried in the preset database based on the transmitted data.

[0009] Based on the metadata, the fields in the transmitted data are classified to obtain a set of classified data; the classified data includes classified field data of multiple preset field types.

[0010] Based on a preset second compression threshold, the categorized field data is filtered to obtain a set of compression targets; wherein the set of compression targets contains at least one compression target.

[0011] Collect feature information for each compression target, and calculate the feature value set of the compression target based on the preset feature set and feature information;

[0012] The matching process is performed based on the preset algorithm matching strategy and the feature value set to determine the candidate compression algorithm set for the compression target; wherein, the candidate compression algorithm set includes at least one compression algorithm;

[0013] Based on a preset algorithm selection strategy, the optimal compression algorithm for the compression target is determined from the set of candidate compression algorithms.

[0014] In one possible design, the categorical field data is filtered according to a preset second compression threshold to obtain a compression target set, including:

[0015] If the size of the categorization field data is not less than the preset second compression threshold;

[0016] Categorical field data are used as compression targets and included in the compression target set; there is no less than one compression target in the compression target set.

[0017] In one possible design, the categorical field data is filtered according to a preset second compression threshold to obtain the compression target set, and the design also includes:

[0018] If the size of all categorized field data is less than the preset second compression threshold;

[0019] Classification datasets are combined as compression targets and included in the compression target set.

[0020] In one possible design, in each set of candidate compression algorithms, the compression algorithms are arranged in a preset order according to a preset algorithm selection strategy.

[0021] Based on a preset algorithm selection strategy, the optimal compression algorithm for the compression target is determined from the set of candidate compression algorithms, including:

[0022] Determine whether the compression processing performed on the corresponding preset field type of the current compression target is the first batch;

[0023] If it is the first batch, the compression target will be compressed once according to each compression algorithm in the candidate compression algorithm set to obtain the latest historical algorithm performance set;

[0024] The weight set is obtained by selecting a strategy based on a preset algorithm.

[0025] The final score for each compression algorithm is calculated based on the weight set and the latest historical algorithm performance set.

[0026] If the scores of each final rating are not exactly the same, the rating with the highest score among all final ratings will be selected as the optimal rating.

[0027] The compression algorithm corresponding to the optimal score is the optimal compression algorithm;

[0028] If there are multiple optimal scores, the compression algorithm ranked first is selected as the optimal compression algorithm.

[0029] Update the latest historical algorithm performance set to obtain the first batch of updated historical algorithm performance sets.

[0030] In one possible design, the optimal compression algorithm for the compression target is determined from a set of candidate compression algorithms according to a preset algorithm selection strategy, and further includes:

[0031] If it is not the first batch, obtain the update history algorithm performance set, weight set and optimal compression algorithm of the previous batch;

[0032] The current final score is calculated based on the previous batch's updated historical algorithm performance set, weight set, and optimal compression algorithm.

[0033] The lowest expected score is obtained by selecting a strategy based on a preset algorithm.

[0034] Compare the current final rating with the lowest expected rating;

[0035] If the current final score is not less than the minimum expected score, then the current final score will be taken as the optimal score.

[0036] The optimal compression algorithm of the previous batch is the optimal compression algorithm of the current batch;

[0037] Update the historical algorithm performance set of the previous batch, and obtain the historical algorithm performance set of the current batch.

[0038] In one possible design, the optimal compression algorithm for the compression target is determined from a set of candidate compression algorithms according to a preset algorithm selection strategy, and further includes:

[0039] If the current final score is less than the minimum expected score, the compression target will be compressed once according to each compression algorithm in the candidate compression algorithm set to obtain the latest historical algorithm performance set for the current batch.

[0040] The final score for each compression algorithm is calculated based on the weight set of the previous batch and the latest historical algorithm performance set of the current batch.

[0041] The highest score among all final scores is selected as the optimal score;

[0042] The compression algorithm corresponding to the best score is the best compression algorithm for the current batch;

[0043] Update the latest historical algorithm performance set for the current batch, and obtain the updated historical algorithm performance set for the current batch.

[0044] In one possible design, the preset feature set includes entropy features, highest repetition rate features, value range features, variance features, and average field length features.

[0045] Secondly, this application provides a data compression apparatus, comprising:

[0046] The acquisition module is used to acquire the transmission data of a single batch of transmissions;

[0047] The query module is used to query the corresponding metadata in the preset database based on the transmitted data if the size of the transmitted data is greater than the preset first compression threshold.

[0048] The classification module is used to classify the fields in the transmitted data according to the metadata to obtain a classified data set; the classified data includes classified field data of multiple preset field types;

[0049] The filtering module is used to filter the categorized field data according to a preset second compression threshold to obtain a set of compression targets; wherein the set of compression targets contains at least one compression target.

[0050] The calculation module is used to collect feature information of each compression target and calculate the feature value set of the compression target based on the preset feature set and feature information.

[0051] The matching module is used to perform matching processing with the feature value set according to the preset algorithm matching strategy to determine the candidate compression algorithm set of the compression target; wherein, the candidate compression algorithm set includes at least one compression algorithm;

[0052] The determination module is used to determine the optimal compression algorithm for the compression target from the set of candidate compression algorithms based on a preset algorithm selection strategy.

[0053] Thirdly, this application provides a data compression device, including: a memory and a processor;

[0054] The memory stores instructions that the computer executes;

[0055] The processor executes computer execution instructions stored in memory, causing the processor to perform a data compression method as described in the first aspect of the invention.

[0056] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a data compression method as described in the first aspect of the invention.

[0057] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements a data compression method according to the first aspect of the invention.

[0058] This application provides a data compression method, apparatus, device, medium, and product, comprising: acquiring transmission data in a single batch; if the size of the transmission data exceeds a preset first compression threshold, querying corresponding metadata in a preset database based on the transmission data; classifying fields in the transmission data according to the metadata to obtain a set of categorized data; filtering the categorized field data according to a preset second compression threshold to obtain a set of compression targets; collecting feature information of each compression target, and calculating a set of feature values ​​for the compression target based on a preset feature set and feature information; performing matching processing with the feature value set according to a preset algorithm matching strategy to determine a set of candidate compression algorithms for the compression target; and determining the optimal compression algorithm for the compression target from the set of candidate compression algorithms according to a preset algorithm selection strategy. Compared to existing technologies that are easily affected by mixed data characteristics and have a single compression algorithm selection, resulting in low data compression efficiency, this application achieves adaptive selection and optimization of compression algorithms by dynamically analyzing data types and characteristics, combining configurable weights and historical algorithm performance, thereby maximizing compression ratio and transmission efficiency under limited CPU resources, thus improving data compression efficiency. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A schematic diagram of a data compression method provided in this application embodiment;

[0061] Figure 2 A schematic flowchart of a data compression method provided in this application embodiment. Figure 1 ;

[0062] Figure 3 A schematic flowchart of a data compression method provided in this application embodiment. Figure 2 ;

[0063] Figure 4 A schematic flowchart of a data compression method provided in this application embodiment. Figure 3 ;

[0064] Figure 5 This is a schematic diagram of the operation flow of the data compression system provided in the embodiments of this application;

[0065] Figure 6 A schematic diagram of the structure of the data compression device provided in the embodiments of this application;

[0066] Figure 7 This is a schematic diagram of a data compression device provided in an embodiment of this application. Detailed Implementation

[0067] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0068] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, nor do they necessarily imply difference. It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more.

[0069] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not specifically limit this. Furthermore, the data compression method provided in the embodiments of this application is merely an example; data compression methods may also include more or less content.

[0070] In distributed database systems, cloud database services, and large-scale data migration scenarios, the data transmission efficiency between the database front-end and back-end or between nodes directly impacts the overall system performance. For example, in scenarios such as cross-regional transaction data synchronization in the financial industry, massive sensor data uploads from IoT devices, and cross-institutional medical record sharing in the healthcare system, the data volume is often large and network bandwidth is limited, while issues such as high latency or network instability also exist. Traditional data transmission methods, when bandwidth is insufficient, rely on costly hardware upgrades or network optimization with limited effectiveness. In these situations, reducing transmission volume through data compression becomes a key means of improving transmission efficiency.

[0071] However, different types of data in databases (such as integers, floating-point numbers, strings, and binary data) have different mathematical characteristics (such as repetition rate, value range, and entropy). Existing compression algorithms (such as LZ4, Zstd, and RLE) show significant differences in compression ratio and performance on data with different characteristics. Using a single fixed algorithm or a coarse selection strategy based on network conditions may lead to low compression efficiency, or even increase CPU resource consumption due to inappropriate algorithm selection, thereby exacerbating the system burden.

[0072] Specifically, when it comes to data compression, different database types may be suitable for different compression algorithms due to their different characteristics. Even data of the same database type may have different mathematical characteristics depending on the business scenario. If only a single compression algorithm is used, it is difficult to guarantee that the data will be compressed in a truly effective way.

[0073] Specifically, existing compression solutions either use a fixed compression algorithm for a certain type of file, resulting in a relatively limited range of choices; or they use compression calculations to assist in network latency, transmitting compressed data suitable for the current network transmission each time. However, there are not many choices for compression algorithms, and the characteristics of the same type of data may change continuously. In scenarios with high network latency and large amounts of data to be transmitted, it is impossible to guarantee that CPU resources can be used to achieve lower network transmission time.

[0074] The existing technical solutions have the following limitations:

[0075] Optionally, existing technologies do not distinguish between data types (such as integers and strings) and characteristics (such as repetition rate and entropy value), resulting in a mismatch between the selection of compression algorithms and the actual characteristics of the data, making it impossible to achieve a balance between compression rate and CPU consumption.

[0076] Optionally, existing technologies directly mix different types of data for compression, causing the algorithm to fail due to feature confounding. For example, when mixing integer and string data, the algorithm optimized for integers may fail. The algorithm may experience a decrease in compression efficiency due to the low repetition rate of the strings.

[0077] Optionally, data characteristics may change dynamically in business scenarios (e.g., the repetition rate of a certain type of data increases significantly within a specific time period). Existing technologies lack a dynamic adjustment mechanism based on historical algorithm performance and real-time feature sampling, making it difficult to optimize compression strategies in a timely manner.

[0078] Optionally, existing technologies do not balance resource consumption and transmission performance through configurable weights (such as compression ratio, compression speed, and CPU usage), resulting in an inability to effectively improve data transmission efficiency in bandwidth-constrained scenarios.

[0079] To address the aforementioned problems, the inventors, during their research on the low data compression efficiency, discovered that existing technologies are susceptible to the influence of mixed data characteristics and rely on a single compression algorithm, resulting in low data compression efficiency. Therefore, the inventors considered dynamically analyzing data types and characteristics, combining configurable weights with historical algorithm performance, to achieve adaptive selection and optimization of the compression algorithm. This maximizes compression ratio and transmission efficiency with limited CPU resources, thereby improving data compression efficiency. Based on this, embodiments of this application provide a data compression method, apparatus, device, medium, and product, applicable to the field of data transmission, aiming to solve the problem of low data compression efficiency in existing technologies.

[0080] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0081] Figure 1 This is a schematic diagram of the system architecture for a data compression method provided in an embodiment of this application. The data compression system is a computer device. Figure 1 In the above architecture, at least one of data acquisition device 101, processing device 102 and display device 103 is included.

[0082] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the processing system architecture of the data compression method. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or divide some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0083] In the specific implementation process, the data acquisition device 101 may include an input / output interface or a communication interface. The data acquisition device 101 can be connected to the processing device through the input / output interface or the communication interface to acquire the transmission data of a single batch.

[0084] The processing device 102 can determine the optimal compression algorithm for compression based on the transmitted data in a single batch.

[0085] The display device 103 can also be a touch screen or the screen of a terminal device, used to receive user commands while displaying the above-mentioned content, so as to realize interaction with the user.

[0086] It should be understood that the aforementioned processing device can be implemented by a processor reading instructions from memory and executing those instructions, or it can be implemented by a chip circuit.

[0087] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0088] The technical solution of this application will be described in detail below with reference to specific embodiments:

[0089] Figure 2 A schematic flowchart of a data compression method provided in this application embodiment. Figure 1 ,like Figure 2 As shown, the method includes:

[0090] S201. Obtain the transmission data for a single batch of transmissions.

[0091] S202. If the size of the transmitted data is greater than the preset first compression threshold, then query the corresponding metadata in the preset database based on the transmitted data.

[0092] For example, the first compression threshold is the total data compression threshold compressMax: compression is performed when the data transmitted in a single transaction exceeds this size.

[0093] Specifically, if the size of the data transmitted in a single batch exceeds compressMax, subsequent processing and compression will be initiated; otherwise, to avoid compressing smaller data, which would lead to high CPU resource consumption and time consumption while the data cannot be effectively compressed.

[0094] S203. Based on the metadata, classify the fields in the transmitted data to obtain a classified data set.

[0095] The categorized data includes categorized field data of multiple preset field types.

[0096] For example, to avoid the mixing of different types of field data affecting the selection of the optimal compression algorithm, fields of the same type in the data transmitted this time are merged together.

[0097] The database side possesses the metadata of the transmitted data, which allows it to directly determine the type of each field. The front-end driver side can determine the type of each field in the data being sent by caching the metadata of the database table.

[0098] To give a further example, the structure before merging is shown in Table 1, which may have multiple fields, each of which may have duplicates or different types.

[0099] Table 1

[0100]

[0101] Furthermore, an example of the merged structure is shown in Table 2. To ensure that the data can be correctly split into the original fields on the other end, the basis for splitting each field needs to be retained in the new fields. Each new field needs to store a small amount of additional information, including the number of old fields contained in the new field, i.e., the number of data items (each data item is an old field in the original data), the sequential array of each data item in the original data, the length subarray of each data item, then all the ordered data items, and the data combinations under each type field (as shown in the figure below). If the fieldMax size is met, all will be treated as independent compression targets for subsequent processing.

[0102] Table 2

[0103]

[0104] S204. Based on the preset second compression threshold, the classification field data is filtered to obtain the compression target set.

[0105] The set of compression targets includes at least one compression target.

[0106] S205. Collect feature information of each compression target, and calculate the feature value set of the compression target based on the preset feature set and feature information.

[0107] The preset feature set includes entropy features, highest repetition rate features, value range features, variance features, and average field length features.

[0108] For example, the feature information of the current data (data combinations that meet the requirements) is obtained through the sampling module, and the feature value set of the compression target is calculated based on the preset feature set.

[0109] The preset feature set is shown in Table 3:

[0110] Table 3

[0111]

[0112] In one possible implementation, a multi-level feature sampling strategy is designed to perform hierarchical sampling of data units (such as combining coarse-grained and fine-grained sampling) to capture data features more comprehensively. For example, coarse-grained sampling (calculating global entropy and repetition rate) is first performed on integer data units, and then fine-grained sampling (analyzing local patterns) is performed on high-entropy sub-blocks, thereby more accurately matching algorithms (such as using Zstd for high-entropy sub-blocks and using [other algorithms] for low-entropy sub-blocks). ).

[0113] Optional, tiered sampling mechanism: coarse-grained sampling is used for fast candidate screening algorithms, and fine-grained sampling is used for refined selection;

[0114] Optional, local feature analysis: perform fine-grained feature calculations (such as local repetition rate and variance) on sub-blocks of data units to avoid global features masking local characteristics.

[0115] Optional, sub-block-level algorithm allocation: Divide the data unit into multiple sub-blocks and select an algorithm independently for each sub-block, improving compression flexibility.

[0116] By employing multi-level feature sampling and sub-block-level algorithm allocation, the system addresses the algorithm selection bias problem caused by a single feature dimension in existing technologies. For example, in mixed feature data units (such as integer data containing locally high repetition regions), the system can select the optimal algorithm for different sub-blocks (e.g., using RLE for high repetition regions and using [other algorithms] for low repetition regions). This improves the overall compression rate. Furthermore, the hierarchical sampling mechanism reduces the resource consumption of fine-grained feature computation, balancing accuracy and efficiency.

[0117] S206. Perform matching processing based on the preset algorithm matching strategy and feature value set to determine the candidate compression algorithm set for the compression target.

[0118] The candidate compression algorithm set includes at least one compression algorithm.

[0119] For example, the common compression scenarios for each available compression algorithm are shown in Table 4:

[0120] Table 4

[0121]

[0122] For example, the set of candidate compression algorithms that can be selected for various types of data when certain characteristics are met is shown in Table 5:

[0123] Table 5

[0124]

[0125] It should be noted that the compression algorithm provided in this embodiment is not necessarily optimal, and the mathematical feature judgment boundary is not necessarily the best. The key lies in the selection of the scheme and the processing flow, namely the processing field, the selection of candidate algorithms based on the features of the data of the same type, and the subsequent algorithm scoring idea. In addition to the compression algorithm mentioned in this embodiment, there are many other compression algorithms that may only have better expectations for a certain type of data. When none of them meet the requirements or the feature boundary is ambiguous, {LZ4, Zstd} is used by default as the candidate algorithm set.

[0126] In one possible implementation, a machine learning model is introduced to predict the optimal algorithm and weight configuration for the current data unit by training on the relationship between historical data features and algorithm performance. For example, when the system detects that the entropy value of a certain type of string data is high, the model can automatically recommend using the Brotli algorithm and adjust the weight parameters to prioritize compression ratio.

[0127] Optional, feature-algorithm mapping model: train supervised learning model (such as random forest, XGBoost), input is data features (entropy, repetition rate, etc.), output is optimal algorithm and weight parameters;

[0128] Optional, adaptive weight adjustment: dynamically adjust the user-configured weights based on the model prediction results (e.g., adjust Ws from the default value of 0.5 to 0.3) to balance resource consumption and compression effect;

[0129] Optional, online learning mechanism: The model updates the training data after each compression to adapt to the long-term changing trends of the data features.

[0130] In particular, the predictive capabilities of machine learning models address the subjectivity issues of manual weight configuration and algorithm selection in existing technologies. For example, in complex business scenarios (such as the transmission of mixed data types), the model can automatically identify the optimal algorithm and weight configuration, avoiding suboptimal choices due to insufficient human experience. Furthermore, the online learning mechanism enables the system to adapt to the long-term evolution of data characteristics (such as a certain type of data gradually changing from low repetition rate to high repetition rate), further improving the accuracy and adaptability of algorithm selection.

[0131] 207. Based on the preset algorithm selection strategy, determine the optimal compression algorithm for the compression target from the set of candidate compression algorithms.

[0132] In one possible embodiment, a real-time feature monitoring module is introduced to continuously collect features of the current data unit (such as entropy value and repetition rate) during data transmission and dynamically adjust the candidate algorithm set. For example, when the repetition rate of a certain integer data unit is detected to increase from 0.4 to 0.6 during transmission, the candidate algorithm set is switched from {LZ4, Zstd} to {RLE} in real time to adapt to the feature change.

[0133] Optional, real-time feature acquisition: Continuously monitor the features of data units through a lightweight sampling module (such as sliding window sampling) to avoid the resource consumption of full computation.

[0134] Optional, dynamic algorithm set update: adjust the candidate algorithm set according to real-time feature changes (such as dynamic adjustment of the repetition rate threshold), rather than relying on statically configured feature boundaries.

[0135] Optional, incremental algorithm scoring: Calculate incremental scores based on historical performance and real-time features to avoid re-evaluating all candidate algorithms each time.

[0136] Specifically, by employing real-time feature monitoring and dynamic algorithm set adjustment, the system addresses the lag in algorithm selection caused by static feature thresholds in existing technologies. For instance, in scenarios involving sudden changes in data features (such as a sudden increase in the repetition rate of data from a certain type of sensor), the system can immediately switch to a better algorithm (such as RLE), avoiding a decrease in compression efficiency due to the lag caused by feature changes. Furthermore, the incremental scoring mechanism reduces the computational overhead of algorithm evaluation, further improving processing efficiency.

[0137] This embodiment provides a data compression method, comprising: acquiring transmission data in a single batch; if the size of the transmission data exceeds a preset first compression threshold, querying corresponding metadata in a preset database based on the transmission data; classifying fields in the transmission data according to the metadata to obtain a set of categorized data; filtering the categorized field data according to a preset second compression threshold to obtain a set of compression targets; collecting feature information of each compression target, and calculating a set of feature values ​​for the compression target based on a preset feature set and feature information; performing matching processing with the feature value set according to a preset algorithm matching strategy to determine a set of candidate compression algorithms for the compression target; and determining the optimal compression algorithm for the compression target from the set of candidate compression algorithms according to a preset algorithm selection strategy. Compared to existing technologies that are easily affected by mixed data characteristics and have a single compression algorithm selection, resulting in low data compression efficiency, this application achieves adaptive selection and optimization of compression algorithms by dynamically analyzing data types and features, combining configurable weights and historical algorithm performance, thereby maximizing compression ratio and transmission efficiency under limited CPU resources, thus improving data compression efficiency.

[0138] Figure 3 A schematic flowchart of a data compression method provided in this application embodiment. Figure 2 As shown in the figure, step S204 specifically includes:

[0139] S301. If the size of the categorized field data is not less than the preset second compression threshold.

[0140] For example, the second compression threshold is the field data compression threshold fieldMax: If a row of data has multiple fields, in order to avoid the mixing of field data of different types affecting the selection of the optimal compression algorithm, the fields of the same type are merged, the size of the merged field of that type is determined, and only the field data exceeding the fieldMax size is compressed; otherwise, if all fields are smaller than the size, the data sent this time is treated as a whole and the default compression algorithm is directly used.

[0141] S302. Take the categorized field data as the compression target and include it in the compression target set.

[0142] Among them, there is no less than one compression target in the compression target set.

[0143] For example, after merging fields of various types, it is determined whether the size of each merged field reaches the threshold fieldMax. If so, the field is then subjected to a separate subsequent compression process.

[0144] S303. If the size of all category field data is less than the preset second compression threshold.

[0145] For example, if none of the fields reach the threshold, the transmitted data will be treated as a whole and the default LZ4 compression algorithm will be used directly.

[0146] The data is a mix of different types with unclear features. Even with Zstd, which has a good default compression rate, it is difficult to guarantee good results. Therefore, LZ4, which has the fastest compression time, is used to minimize the total time consumption.

[0147] S304. Combine the classification datasets as compression targets and include them in the compression target set.

[0148] In this embodiment, intelligent filtering of compression targets is achieved by dynamically identifying the relationship between the categorized field data and the preset second compression threshold: when a single field data reaches the threshold, it is compressed separately to retain the granularity of key information; when all field data does not reach the threshold, it is compressed as a whole to reduce redundant storage. Ultimately, a balance and optimization is achieved between compression efficiency and data integrity, effectively improving the resource utilization rate of data storage and processing.

[0149] Figure 4 A schematic flowchart of a data compression method provided in this application embodiment. Figure 3 As shown in the figure, in each candidate compression algorithm set, the compression algorithms are arranged in a preset order according to a preset algorithm selection strategy; therefore, the specific implementation steps of S207 above include:

[0150] S401. Determine whether the compression processing performed on the corresponding preset field type of the current compression target is the first batch.

[0151] S402. If it is the first batch, compress the target once according to each compression algorithm in the candidate compression algorithm set, and obtain the latest historical algorithm performance set.

[0152] The first compression of a field of a certain type will use all algorithms in the candidate algorithm list for compression and update the historical algorithm performance of the current data type.

[0153] Among them, the historical algorithm performance includes Cr compression ratio, Cs compression speed, and Cc CPU consumption.

[0154] S403. Obtain the weight set by selecting a strategy according to the preset algorithm.

[0155] Among them, there are corresponding weights Wr, Ws, and Wc according to user expectations.

[0156] Specifically, if server CPU resources are also scarce, then configure Ws to be higher and Wc and Wr to be lower. If CPU resources are sufficient, the network is poor, and good transmission efficiency is desired, then configure Wr to be the highest and the others to be moderate.

[0157] For example, the weight set includes weights Wr, Ws, and Wc: which correspond to the historical average compression ratio, compression speed, and CPU usage, respectively. The weights are adjusted to select the algorithm that best meets the user's desired weight from the candidate algorithms that meet the available features. The fixed weights configured by this connection can be specified individually or fixedly used each time the algorithm is executed through the execution interface.

[0158] S404. Calculate the final score for each compression algorithm based on the weight set and the latest historical algorithm performance set.

[0159] For example, for a set of compression algorithms An, the final score for each algorithm is:

[0160]

[0161] S405. If the scores of each final score are not exactly the same, the score with the highest score among all final scores shall be selected as the optimal score.

[0162] S406, The compression algorithm corresponding to the optimal score is the optimal compression algorithm.

[0163] S407. If there are multiple optimal scores, the compression algorithm ranked first shall be selected as the optimal compression algorithm.

[0164] Specifically, when there are multiple identical highest scores in the final score, the compression algorithm ranked first is selected according to the pre-arranged order.

[0165] S408. Update the latest historical algorithm performance set and obtain the first batch of updated historical algorithm performance sets.

[0166] Specifically, if they are the same, the algorithm that appears earlier in the algorithm set is selected, and the historical average values ​​of Cr, Cs, and Cc for that type of algorithm are updated.

[0167] In one possible implementation, if it is not the first batch, the update history algorithm performance set, weight set and optimal compression algorithm of the previous batch are obtained.

[0168] Furthermore, the current final score is calculated based on the previous batch of updated historical algorithm performance set, weight set, and optimal compression algorithm.

[0169] Furthermore, a strategy is selected based on a preset algorithm to obtain the lowest expected score.

[0170] Among them, the lowest expected score line: if the final algorithm score is lower than this score, other compression algorithms will be used to select the one with the highest score.

[0171] Specifically, if the score of this algorithm is lower than the line, other algorithms in the candidate set are selected for compression and the scores of all candidate algorithms are obtained. The compression algorithm with the highest score is selected, and the average values ​​of historical performance Cr, Cs, and Cc are updated.

[0172] Furthermore, compare the current final score with the lowest expected score.

[0173] Furthermore, if the current final score is not less than the minimum expected score, then the current final score will be taken as the optimal score.

[0174] Furthermore, the optimal compression algorithm of the previous batch becomes the optimal compression algorithm for the current batch.

[0175] Specifically, during subsequent compression, the candidate compression algorithm with the highest score is calculated based on the current weights and historical average performance Cr, Cs, Cc, and the current weights, and then compression is performed.

[0176] Furthermore, update the set of algorithm performance data from the previous batch and obtain the set of algorithm performance data from the current batch.

[0177] In one possible embodiment, if the current final score is less than the minimum expected score, the compression target is compressed once according to each compression algorithm in the candidate compression algorithm set to obtain the latest historical algorithm performance set for the current batch.

[0178] Furthermore, based on the weight set of the previous batch and the latest historical algorithm performance set of the current batch, the final score of each compression algorithm is calculated.

[0179] Furthermore, the score with the highest value is selected from all the final scores as the optimal score.

[0180] Furthermore, the compression algorithm corresponding to the optimal score is the optimal compression algorithm for the current batch.

[0181] Furthermore, update the latest historical algorithm performance set for the current batch, and obtain the updated historical algorithm performance set for the current batch.

[0182] In this embodiment, intelligent adaptation and continuous optimization of compression algorithms are achieved through a batch dynamic evaluation mechanism and a historical data-driven algorithm selection strategy. The first batch establishes a benchmark scoring system through full algorithm testing, while subsequent batches are dynamically verified or re-evaluated based on historical performance and the lowest expected score. This ensures the priority selection of high-scoring algorithms and the sequential fallback in scenarios with the same score. Furthermore, the iterative update of the historical algorithm performance set improves the accuracy of subsequent decisions. Ultimately, a closed-loop optimization is formed between compression efficiency, algorithm adaptability, and resource utilization efficiency, effectively improving the processing robustness and overall performance in multi-batch data compression scenarios.

[0183] This application also provides a possible embodiment. Figure 5 This is a schematic diagram of the operation flow of the data compression system provided in the embodiments of this application, such as... Figure 5 As shown, starting with the data size, when the transmitted data exceeds compressMax, fields of the same type are first merged. When processing the merged fields, if the field size is not greater than fieldMax, the process ends directly; otherwise, data is sampled and features are calculated. After generating a list of candidate compression algorithms based on the features, the system adopts different strategies depending on whether it is the first time compressing this type of data—for the first compression, all candidate algorithms are used for compression and the historical average performance is updated, and the scores of all algorithms are calculated; for non-first compression, the current weight and historical average performance are combined to calculate the score and update the historical performance. Finally, the system determines whether to re-select algorithms or select the optimal algorithm to complete the compression by comparing the highest score with the preset threshold Line.

[0184] The system employs a two-tiered filtering mechanism using data size thresholds (compressMax) and field size thresholds (fieldMax), forming a closed-loop logic of "data preprocessing - feature calculation - algorithm selection - score evaluation - threshold verification." The initial compression uses a full-scale algorithm test mode to establish benchmark performance. Subsequent compressions incorporate historical performance weights for dynamic optimization, with feedback adjustments based on the highest score and the line score. This ensures that the compression algorithm selection aligns with current data characteristics while also considering historical experience, creating an adaptive algorithm selection logic.

[0185] In this embodiment, redundant data is reduced by merging fields, compression algorithms are accurately matched by sampling and calculating features, and algorithm selection is continuously optimized by combining historical performance with current weights. Finally, the optimal compression scheme is selected by comparing the highest score with a threshold, thereby improving compression efficiency while ensuring compression effect, effectively saving storage space and improving data transmission performance.

[0186] Figure 6 A schematic diagram of the data compression device provided in the embodiments of this application is shown below. Figure 6 As shown, the device includes: an acquisition module 61, a query module 62, a classification module 63, a filtering module 64, a calculation module 65, a matching module 66, and a determination module 67.

[0187] The acquisition module 61 is used to acquire the transmission data of a single batch of transmissions;

[0188] The query module 62 is used to query the corresponding metadata in the preset database based on the transmitted data if the size of the transmitted data is greater than the preset first compression threshold.

[0189] The classification module 63 is used to classify the fields in the transmitted data according to the metadata to obtain a classified data set; wherein the classified data includes classified field data of multiple preset field types;

[0190] The filtering module 64 is used to filter the categorized field data according to a preset second compression threshold to obtain a compression target set; wherein the compression target set contains at least one compression target.

[0191] The calculation module 65 is used to collect feature information of each compression target and calculate the feature value set of the compression target based on the preset feature set and feature information.

[0192] The matching module 66 is used to perform matching processing with the feature value set according to the preset algorithm matching strategy to determine the candidate compression algorithm set of the compression target; wherein, the candidate compression algorithm set includes at least one compression algorithm;

[0193] The determination module 67 is used to determine the optimal compression algorithm for the compression target from the set of candidate compression algorithms according to a preset algorithm selection strategy.

[0194] In one possible design, the categorical field data is filtered according to a preset second compression threshold to obtain a compression target set, including:

[0195] The filtering module 64 is also used to filter data if the size of the classification field data is not less than a preset second compression threshold.

[0196] Categorical field data are used as compression targets and included in the compression target set; there is no less than one compression target in the compression target set.

[0197] In one possible design, the categorical field data is filtered according to a preset second compression threshold to obtain the compression target set, and the design also includes:

[0198] The filtering module 64 is also used if the size of all category field data is less than a preset second compression threshold;

[0199] Classification datasets are combined as compression targets and included in the compression target set.

[0200] In one possible design, in each set of candidate compression algorithms, the compression algorithms are arranged in a preset order according to a preset algorithm selection strategy.

[0201] Based on a preset algorithm selection strategy, the optimal compression algorithm for the compression target is determined from the set of candidate compression algorithms, including:

[0202] The determination module 67 is also used to determine whether the compression processing performed on the corresponding preset field type of the current compression target is the first batch;

[0203] If it is the first batch, the compression target will be compressed once according to each compression algorithm in the candidate compression algorithm set to obtain the latest historical algorithm performance set;

[0204] The weight set is obtained by selecting a strategy based on a preset algorithm.

[0205] The final score for each compression algorithm is calculated based on the weight set and the latest historical algorithm performance set.

[0206] If the scores of each final rating are not exactly the same, the rating with the highest score among all final ratings will be selected as the optimal rating.

[0207] The compression algorithm corresponding to the optimal score is the optimal compression algorithm;

[0208] If there are multiple optimal scores, the compression algorithm ranked first is selected as the optimal compression algorithm.

[0209] Update the latest historical algorithm performance set to obtain the first batch of updated historical algorithm performance sets.

[0210] In one possible design, the optimal compression algorithm for the compression target is determined from a set of candidate compression algorithms according to a preset algorithm selection strategy, and further includes:

[0211] The determination module 67 is also used to obtain the set of algorithm performance, weight set and optimal compression algorithm of the previous batch if it is not the first batch.

[0212] The current final score is calculated based on the previous batch's updated historical algorithm performance set, weight set, and optimal compression algorithm.

[0213] The lowest expected score is obtained by selecting a strategy based on a preset algorithm.

[0214] Compare the current final rating with the lowest expected rating;

[0215] If the current final score is not less than the minimum expected score, then the current final score will be taken as the optimal score.

[0216] The optimal compression algorithm of the previous batch is the optimal compression algorithm of the current batch;

[0217] Update the historical algorithm performance set of the previous batch, and obtain the historical algorithm performance set of the current batch.

[0218] In one possible design, the optimal compression algorithm for the compression target is determined from a set of candidate compression algorithms according to a preset algorithm selection strategy, and further includes:

[0219] The determination module 67 is also used to perform compression processing on the compression target according to each compression algorithm in the candidate compression algorithm set if the current final score is less than the minimum expected score, so as to obtain the latest historical algorithm performance set of the current batch.

[0220] The final score for each compression algorithm is calculated based on the weight set of the previous batch and the latest historical algorithm performance set of the current batch.

[0221] The highest score among all final scores is selected as the optimal score;

[0222] The compression algorithm corresponding to the best score is the best compression algorithm for the current batch;

[0223] Update the latest historical algorithm performance set for the current batch, and obtain the updated historical algorithm performance set for the current batch.

[0224] In one possible design, the preset feature set includes entropy features, highest repetition rate features, value range features, variance features, and average field length features.

[0225] The data compression device provided in this embodiment can execute a data compression method of the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0226] In a specific implementation of the aforementioned data compression method, each module can be implemented as a processor. The processor can execute computer execution instructions stored in the memory, thereby enabling the processor to execute the aforementioned data compression method.

[0227] Figure 7 This is a schematic diagram of a data compression device provided in an embodiment of this application. Figure 7 As shown, the data compression device 70 includes at least one processor 71 and a memory 72. The data compression device 70 also includes a communication component 73. The processor 71, memory 72, and communication component 73 are connected via a bus 74.

[0228] In a specific implementation, at least one processor 71 executes computer execution instructions stored in memory 72, causing at least one processor 71 to execute a data transmission method executed by the compression device side of the above data.

[0229] The specific implementation process of processor 71 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0230] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0231] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0232] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0233] The functions implemented by the data compression device and the main control device described above illustrate the solutions provided by the embodiments of the present invention. It is understood that, in order to achieve the above functions, the data compression device or the main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.

[0234] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method in the field of data transmission described above.

[0235] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0236] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in a data compression device or a host device.

[0237] This application also provides a computer program product, comprising: a computer program stored in a readable storage medium, at least one processor of a data compression device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the data compression device to perform the scheme provided in any of the above embodiments.

[0238] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0239] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data compression method, characterized in that, include: Obtain the transmission data for a single batch of transmissions; If the size of the transmitted data is greater than a preset first compression threshold, then the corresponding metadata is queried in a preset database based on the transmitted data. Based on the metadata, the fields in the transmitted data are classified to obtain a classified data set; wherein, the classified data includes classified field data of multiple preset field types; The categorized field data is filtered according to a preset second compression threshold to obtain a set of compression targets; wherein the set of compression targets contains at least one compression target. Collect feature information of each compression target, and calculate the feature value set of the compression target based on a preset feature set and the feature information; The set of feature values ​​is matched with a preset algorithm matching strategy to determine a set of candidate compression algorithms for the compression target; wherein the set of candidate compression algorithms includes at least one compression algorithm. According to a preset algorithm selection strategy, the optimal compression algorithm for the compression target is determined from the set of candidate compression algorithms.

2. The method according to claim 1, characterized in that, The step of filtering the categorized field data according to a preset second compression threshold to obtain a compression target set includes: If the size of the classification field data is not less than the preset second compression threshold; The categorized field data is used as the compression target and included in the compression target set; wherein there is at least one compression target in the compression target set.

3. The method according to claim 1, characterized in that, The step of filtering the categorized field data according to a preset second compression threshold to obtain a compression target set further includes: If the size of all the categorized field data is less than the preset second compression threshold; The classified dataset is combined into the compression target and included in the compression target set.

4. The method according to claim 3, characterized in that, In each set of candidate compression algorithms, the compression algorithms are arranged in a preset order according to the preset algorithm selection strategy. The step of determining the optimal compression algorithm for the compression target from the set of candidate compression algorithms according to a preset algorithm selection strategy includes: Determine whether the compression processing performed on the preset field type corresponding to the current compression target is the first batch; If it is the first batch, the compression target is compressed once according to each of the compression algorithms in the candidate compression algorithm set to obtain the latest historical algorithm performance set; The weight set is obtained according to the preset algorithm selection strategy; Based on the weight set and the latest historical algorithm performance set, the final score of each compression algorithm is calculated; If the scores of each of the final scores are not exactly the same, the score with the highest score among all the final scores shall be selected as the optimal score. The compression algorithm corresponding to the optimal score is the optimal compression algorithm; If there are multiple optimal scores, the compression algorithm ranked first is selected as the optimal compression algorithm. Update the latest historical algorithm performance set to obtain the first batch of updated historical algorithm performance sets.

5. The method according to claim 3, characterized in that, The step of determining the optimal compression algorithm for the compression target from the set of candidate compression algorithms according to a preset algorithm selection strategy further includes: If it is not the first batch, obtain the update history algorithm performance set, weight set and the optimal compression algorithm of the previous batch; The current final score is calculated based on the previous batch's updated historical algorithm performance set, the weight set, and the optimal compression algorithm. The lowest expected score is obtained by selecting a strategy based on the preset algorithm. Compare the current final score with the lowest expected score; If the current final score is not less than the minimum expected score, then the current final score is taken as the optimal score; The optimal compression algorithm of the previous batch is the optimal compression algorithm of the current batch; Update the update history algorithm performance set of the previous batch, and obtain the update history algorithm performance set of the current batch.

6. The method according to claim 5, characterized in that, The step of determining the optimal compression algorithm for the compression target from the set of candidate compression algorithms according to a preset algorithm selection strategy further includes: If the current final score is less than the minimum expected score, then the compression target is compressed once according to each of the compression algorithms in the candidate compression algorithm set to obtain the latest historical algorithm performance set of the current batch. The final score of each compression algorithm is calculated based on the weight set of the previous batch and the latest historical algorithm performance set of the current batch. The highest score among all the final scores is selected as the optimal score. The compression algorithm corresponding to the optimal score is the optimal compression algorithm for the current batch; Update the latest historical algorithm performance set for the current batch, and obtain the updated historical algorithm performance set for the current batch.

7. The method according to claim 6, characterized in that, The preset feature set includes entropy features, highest repetition rate features, value range features, variance features, and average field length features.

8. A data compression device, characterized in that, include: The acquisition module is used to acquire the transmission data of a single batch of transmissions; The query module is used to query the corresponding metadata in a preset database based on the transmitted data if the size of the transmitted data is greater than a preset first compression threshold. The classification module is used to classify the fields in the transmitted data according to the metadata to obtain a classified data set; wherein the classified data includes classified field data of multiple preset field types; The filtering module is used to filter the categorized field data according to a preset second compression threshold to obtain a set of compression targets; wherein the set of compression targets contains at least one compression target; The calculation module is used to collect feature information of each of the compression targets and calculate the feature value set of the compression targets based on the preset feature set and the feature information. A matching module is used to perform matching processing with the feature value set according to a preset algorithm matching strategy to determine a set of candidate compression algorithms for the compression target; wherein, the set of candidate compression algorithms includes at least one compression algorithm; The determination module is used to determine the optimal compression algorithm for the compression target from the set of candidate compression algorithms according to a preset algorithm selection strategy.

9. A data compression device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images