A data compression method, apparatus, device, storage medium, and program product.

By clustering data in the time domain, frequency domain, and with anomaly features, a personalized compression strategy is generated, which solves the problem of poor data compression performance caused by static compression methods and achieves fast and lossless data compression.

CN122512928BActive Publication Date: 2026-11-17CHINA MOBILE INFORMATION TECHNOLOGY CO LTD +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202610975719.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-07-02
Publication Date
2026-11-17
Estimated Expiration
2046-07-02

AI Technical Summary

Technical Problem

In existing technologies, static compression methods result in poor data compression performance, with some data being lost or compression taking a long time.

Method used

By extracting time-domain features, frequency-domain features, and anomaly features from multiple data sets, clustering is performed, and a compression strategy is generated for each cluster to compress the data in each cluster.

Benefits of technology

It achieves fast and lossless data compression, improving compression efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122512928B_ABST
    Figure CN122512928B_ABST
Patent Text Reader

Abstract

This invention provides a data compression method, apparatus, device, storage medium, and program product, relating to the field of data processing technology. The method includes: acquiring multiple data sets from different data sources; extracting at least one of time-domain features, frequency-domain features, and anomaly features from the acquired data sets, where the time-domain features characterize the changes in the data over time, the frequency-domain features represent the data at different frequency components, and the anomaly features characterize anomalous data; clustering the data sets based on at least one of the time-domain features, frequency-domain features, and anomaly features to obtain multiple clusters, each cluster including at least one data set; generating a compression strategy corresponding to each cluster; and compressing the at least one data set included in each cluster based on the compression strategy corresponding to each cluster. This invention improves compression performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a data compression method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the rapid development of IoT technology, massive amounts of time-series data are continuously uploaded to servers or the cloud for subsequent real-time analysis, predictive modeling, or data archiving. In related technologies, to alleviate data storage pressure, preset algorithms are typically used to compress the data, reducing the storage resources it occupies. However, these technologies generally employ static compression methods, and due to the different characteristics of various data sets, some data may be lost after compression, or compression of some data may take a long time, resulting in poor compression performance.

[0003] It is evident that static compression methods in related technologies result in poor data compression performance. Summary of the Invention

[0004] This invention provides a data compression method, apparatus, device, storage medium, and program product to solve the problem that static compression methods in related technologies result in poor data compression performance.

[0005] To solve the above problems, the present invention is implemented as follows: In a first aspect, embodiments of the present invention provide a data compression method, comprising: Collect multiple data points, which are data from different data sources; Extract at least one of the time-domain features, frequency-domain features, and anomaly features from multiple collected data. The time-domain features are used to characterize the changes of the multiple data over time, the frequency-domain features are used to represent the multiple data at different frequency components, and the anomaly features are used to characterize the anomaly data among the multiple data. The multiple data are clustered based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain multiple clusters, each cluster including at least one data point; Generate a compression strategy corresponding to each cluster; Based on the compression strategy corresponding to each cluster, at least one piece of data included in each cluster is compressed.

[0006] In a second aspect, embodiments of the present invention provide a data compression apparatus, comprising: The data acquisition module is used to acquire multiple data points, which are data from different data sources. An extraction module is used to extract at least one of time-domain features, frequency-domain features, and anomaly features from multiple collected data. The time-domain features are used to characterize the changes of the multiple data over time, the frequency-domain features are used to represent the multiple data at different frequency components, and the anomaly features are used to characterize the abnormal data among the multiple data. The clustering module is used to cluster the multiple data based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain multiple clusters, each cluster including at least one data point; The generation module is used to generate the compression strategy corresponding to each cluster. The compression module is used to compress at least one piece of data included in each cluster based on the compression strategy corresponding to each cluster.

[0007] Thirdly, embodiments of the present invention also provide an electronic device, including a transceiver and a processor. The processor is used to collect multiple data points, which are data from different data sources; The processor is further configured to extract at least one of time-domain features, frequency-domain features, and anomaly features from the acquired multiple data sets. The time-domain features are used to characterize the changes of the multiple data sets over time, the frequency-domain features are used to represent the multiple data sets at different frequency components, and the anomaly features are used to characterize the anomaly data among the multiple data sets. The processor is further configured to cluster the plurality of data based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain a plurality of clusters, each cluster including at least one piece of data; The processor is also used to generate a compression strategy corresponding to each cluster. The processor is further configured to compress at least one piece of data included in each cluster based on the compression strategy corresponding to each cluster.

[0008] Fourthly, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data compression method described in the first aspect.

[0009] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data compression method described in the first aspect.

[0010] In a sixth aspect, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the data compression method described in the first aspect.

[0011] In this embodiment of the invention, multiple data sets are collected from different data sources. At least one of the following is extracted from the collected data sets: time-domain features, frequency-domain features, and anomaly features. The time-domain features characterize the temporal variation of the data sets, the frequency-domain features represent the data sets at different frequency components, and the anomaly features characterize anomalous data. The data sets are then clustered based on at least one of these features to obtain multiple clusters, each cluster containing at least one data set. A compression strategy is generated for each cluster. The at least one data set included in each cluster is then compressed based on the compression strategy. Thus, by clustering multiple data sets using at least one of the time-domain features, frequency-domain features, and anomaly features to obtain different clusters, and then generating a compression strategy for each cluster, compression can be performed on at least one data set within each cluster based on the corresponding compression strategy. This allows for fast and lossless compression of each data set, effectively improving the compression performance. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a data compression method provided in an embodiment of the present invention; Figure 2 This is a structural diagram of a data compression device provided in an embodiment of the present invention; Figure 3 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Please see Figure 1 , Figure 1 This is a flowchart of a data compression method provided in an embodiment of the present invention, such as... Figure 1 As shown, it includes the following steps: Step 101: Collect multiple data points, which are data from different data sources.

[0016] The above data come from different data sources. By collecting data from different data sources, we can extract the characteristics of different data, and then determine the compression strategy based on the characteristics of the data.

[0017] For example, data from different data sources can be time-series data from different sources such as IoT gateways and third-party platforms (such as industrial systems and meteorological monitoring platforms).

[0018] Step 102: Extract at least one of the time-domain features, frequency-domain features, and anomaly features of the acquired multiple data. The time-domain features are used to characterize the changes of the multiple data over time, the frequency-domain features are used to represent the multiple data at different frequency components, and the anomaly features are used to characterize the abnormal data in the multiple data.

[0019] The aforementioned time-domain features are used to characterize the changes of the multiple data over time, and may include at least one of the following parameters: mean (arithmetic mean of sampled values ​​within a sliding window), standard deviation (square root of variance of sampled values ​​within a sliding window), kurtosis (calculated based on fourth moment), skewness (calculated based on third moment), and trend slope (slope calculated from data within a linear regression fitting window).

[0020] The aforementioned frequency domain features are used to represent the situation of the multiple data at different frequency components, and may include at least one of the following parameters: frequency domain energy distribution (calculating the energy ratio of each frequency component after converting the time domain data to the frequency domain based on FFT), dominant frequency amplitude (extracting the amplitude of the three frequency components with the highest energy ratio).

[0021] The aforementioned abnormal features are used to characterize the abnormal data in the multiple data sets, and may include at least one of the following parameters: outlier ratio (calculated after detecting outliers based on the isolated forest algorithm, the proportion of outliers to the total number of points in the window), maximum amplitude difference (the difference between the maximum and minimum values ​​within the window).

[0022] It should be understood that different features can reflect different situations of data, such as the overall distribution level and dispersion of data, the steepness and asymmetry of data distribution, the linear trend of data, the periodicity and intensity of data, the frequency and range of data fluctuations, etc. In this embodiment, at least one of the time domain features, frequency domain features and anomaly features of multiple data is extracted so that the data compression strategy can be determined subsequently through at least one of the time domain features, frequency domain features and anomaly features.

[0023] Step 103: Cluster the multiple data based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain multiple clusters, each cluster including at least one data.

[0024] The above clusters are obtained by clustering multiple data based on features. Different clusters include data with different features, so different compression strategies need to be selected; the data in the same cluster have similar features, so the same compression strategy can be selected.

[0025] In some implementations, after the first clustering to obtain clusters, the data can be divided into different clusters according to the characteristics of the data, and the center of the clusters can be updated to achieve clustering of different data.

[0026] Specifically, in the clustering and grouping stage, incremental MiniBatch KMeans clustering can be used to optimize the problem that the traditional KMeans algorithm needs to retrain the entire dataset. The initial clustering is based on historical data (such as data from the previous 7 days) to train the KMeans model. The number of clusters K is determined by the elbow rule (default K=5-8) to generate the initial clusters. During incremental updates, only the feature vectors of the new data segment whose distance from the existing cluster center is greater than the threshold (threshold = average distance of the initial cluster × 1.2) are updated. When the proportion of samples whose distance from the nearest cluster center is greater than the threshold in the new feature vector is ≥20%, the number of clusters K+1 is automatically increased to generate new clusters to adapt to the new data source features. The clustering update time is reduced by 60% compared to the traditional method.

[0027] Step 104: Generate the compression strategy corresponding to each cluster.

[0028] The above compression strategy is a strategy for compressing the data included in the cluster. Different compression strategies include different compression parameters. By generating a compression strategy corresponding to each cluster, the data included in the cluster can be compressed through the compression strategy.

[0029] Since the data in the same cluster have similar characteristics, the same compression strategy can achieve similar compression results. Furthermore, since the compression strategy is generated specifically for the cluster, it can achieve better compression results, thus enabling each data to be compressed quickly and losslessly, effectively improving the compression effect.

[0030] Step 105: Compress at least one piece of data included in each cluster based on the compression strategy corresponding to each cluster.

[0031] In this embodiment of the invention, multiple data sets are collected from different data sources. At least one of the following is extracted from the collected data sets: time-domain features, frequency-domain features, and anomaly features. The time-domain features characterize the temporal variation of the data sets, the frequency-domain features represent the data sets at different frequency components, and the anomaly features characterize anomalous data. The data sets are then clustered based on at least one of these features to obtain multiple clusters, each cluster containing at least one data set. A compression strategy is generated for each cluster. The at least one data set included in each cluster is then compressed based on the compression strategy. Thus, by clustering multiple data sets using at least one of the time-domain features, frequency-domain features, and anomaly features to obtain different clusters, and then generating a compression strategy for each cluster, compression can be performed on at least one data set within each cluster based on the corresponding compression strategy. This allows for fast and lossless compression of each data set, effectively improving the compression performance.

[0032] In one embodiment, the collection of multiple data points includes: Obtain the sampling frequency for each data source; The window length of each data source is calculated based on the sampling frequency and preset time interval of each data source. Data is collected from each data source based on the window length of each data source to obtain the multiple data sets.

[0033] It should be noted that, due to the different data characteristics (such as generation frequency) of different data sources, different acquisition windows are required to collect data from different data sources. Therefore, in this embodiment of the invention, the sampling frequency of each data source is obtained; the window length of each data source is calculated based on the sampling frequency and a preset time interval; and data from each data source is collected based on the window length of each data source to obtain multiple data sets. In this way, the window length is calculated using the acquisition frequency of each data source, and the window length is matched with the data source, thereby enabling data collection from different data sources through the window length, thus achieving the acquisition of multiple data sets.

[0034] The formula for calculating the window length for each data source is: Window length = Sampling frequency × Preset time interval. For example, the window length for high-frequency data (such as industrial equipment vibration monitoring, with a sampling frequency of 100Hz) is 100Hz × 300s = 30,000 sampling points, and the window length for low-frequency data (such as ambient temperature monitoring, with a sampling frequency of 1 time / minute) is 1 time / minute × 5 minutes = 5 sampling points.

[0035] In some implementations, users can manually adjust the window threshold based on business needs (e.g., the minimum window length for critical business data is ≥100 sampling points) to ensure that window division balances data integrity and processing efficiency.

[0036] In some implementations, load balancing scheduling can be based on the Spark / Flink distributed computing framework, which distributes data segments to different computing nodes based on both data volume and computational complexity. By pre-calculating the number of sampling points (data volume) and the estimated time for feature extraction (computational complexity) for each data segment, a greedy algorithm is used to achieve a node load deviation of ≤10%, avoiding processing delays caused by excessive data volume on a single node. The batch processing efficiency is improved by more than 30% compared to the traditional "file-level" scheduling.

[0037] In one embodiment, the method further includes: The plurality of data are preprocessed to obtain the preprocessed plurality of data; The extraction of at least one of the time-domain features, frequency-domain features, and anomaly features from the multiple data obtained includes: Extract at least one of the time-domain features, frequency-domain features, and anomaly features from the preprocessed data; The preprocessing includes at least one of the following: Standardize the fields in the multiple data sets; Convert the format of the multiple data items; Delete data with abnormal values ​​from the multiple datasets.

[0038] In this embodiment of the invention, preprocessing of multiple data is achieved by standardizing the fields in the multiple data, converting the format of the multiple data, and deleting data with abnormal values ​​in the multiple data, so that feature extraction can be performed on the preprocessed data.

[0039] For example, the mapping relationship between each data source field and the standardized field can be defined through a JSON configuration file (such as mapping "device number" to "sensor identifier" and "acquisition time" to "timestamp"), supporting automatic parsing of multiple mainstream time series data formats (such as JSON, CSV, Protocol Buffers).

[0040] In some implementations, data can be uniformly converted into the columnar storage format Parquet. Parquet's column-level compression feature can be used to reduce the redundancy of the original data in advance, and a directory structure can be built according to the data source identifier / time period, such as / data / iot / sensor_001 / 20240520 / 00_12. Finally, dirty data with abnormal sample values ​​is filtered based on the Z-Score algorithm (threshold is configurable, default ±3σ). The filtered data is written to the distributed data lake (Hudi), and a data cleaning log (including abnormal data identifier and filtering reason) is recorded for traceability.

[0041] Furthermore, a time-layered index can be embedded in the data storage path, with the first-level index representing the time period and the second-level index representing the data segment identifier. A mapping table from the data source to the data segment is established and stored in a metadata database (such as MySQL). This supports quick location of data segments by data source and time range, with an index query response time of ≤100ms, thereby solving the problem of low efficiency in locating massive amounts of data in the cloud.

[0042] In one embodiment, the plurality of clusters includes a first cluster, and generating a compression strategy corresponding to each cluster includes: Obtain data parameters of at least one data included in the first cluster, wherein the data parameters include at least one of the following: abnormal data ratio, trend slope, main frequency energy ratio, and maximum amplitude difference; The feature type of the first cluster is determined based on the data parameters in the first mapping relationship set, wherein the first mapping relationship set includes the mapping relationship between different data parameters and different feature types; The compression strategy for the first cluster is determined based on the feature type of the first cluster in the second mapping relationship set, wherein the second mapping relationship set includes the mapping relationship between different feature types and different compression strategies.

[0043] The first cluster mentioned above is one of multiple clusters. In this embodiment, the first cluster is used as an example. Each of the multiple clusters can determine the compression strategy in the same way as the first cluster.

[0044] The first mapping relationship set mentioned above is a pre-configured relationship set between feature types and data parameters. By configuring the first mapping relationship set, the feature types corresponding to different data parameters can be determined.

[0045] The aforementioned second mapping relationship set represents the pre-configured relationship between feature types and compression strategies. After determining the feature type of a cluster through the first preset relationship set, the compression strategy corresponding to the feature type can be determined through the second mapping relationship set, thereby realizing the determination of the compression strategy corresponding to the cluster.

[0046] In some implementations, the first mapping set and the second mapping set can be configured as a single mapping set, which includes multiple mapping relationships. Each mapping relationship is a mapping relationship between data parameters, feature types, and compression strategies, thereby determining the compression strategy corresponding to the cluster through a single mapping set.

[0047] For example, data parameters with outlier ratio <5% and absolute trend slope <0.1 are defined as stationary linear trend type. The compression strategy for stationary linear trend type is as follows: linear prediction residual encoding and vectorized LZ4 lossless compression are adopted. First, a linear prediction model x_pred = a×t + b (a is the trend slope and b is the intercept) is fitted based on the first N sampling points in the window (N = window length × 20%) and the residual res = x_actual - x_pred between the actual sampled value and the predicted value is calculated. Then, the residual is differentially encoded (diff_res = res_i - res_{i-1}) and compressed by LZ4 vectorized encoding (using the CPU SIMD instruction set). The compression speed is 40% faster than traditional LZ4 and the lossless compression ratio can reach 1:8-1:12.

[0048] For example, if the data parameter has a main frequency energy ratio of ≥60%, it is set as a periodic feature type. The compression strategy of the periodic feature type is as follows: DFT lossy compression is adopted. First, DFT is performed on the data in the window to obtain the complex form of the frequency component F(k). Only the first M frequency components with the energy ratio are retained (M is configured based on the business accuracy requirements, and the default M=5-10). Adaptive quantization is adopted for the retained amplitude A(k) (accuracy range 8-16 bits, the larger the amplitude, the higher the quantization accuracy), and phase θ(k) is quantized with 8 bits to ensure that the quantization error is ≤3% of the original amplitude. The lossy compression ratio can reach 1:15-1:25.

[0049] For example, data parameters such as outlier rate ≥ 20% and maximum amplitude difference > 20% of the original range can be defined as abnormal fluctuation type. The compression strategy for abnormal fluctuation type is as follows: use RLE and difference preservation lightweight lossless compression, perform run-length encoding on continuous identical sampled values ​​(such as fixed values ​​when equipment fails) (e.g., "5,10" means 5 appears 10 times consecutively), and store only the difference between non-continuous values ​​and the previous value (e.g., x_1, x_2-x_1, x_3-x_2), avoid storing the original data, and the compression ratio can reach 1:3-1:5, while ensuring 100% retention of outliers, meeting the needs of fault diagnosis scenarios.

[0050] In this embodiment of the invention, data parameters of at least one data element included in the first cluster are obtained. These data parameters include at least one of the following: outlier proportion, trend slope, main frequency energy proportion, and maximum amplitude difference. Based on the data parameters in a first mapping relationship set, the feature type of the first cluster is determined. This first mapping relationship set includes mapping relationships between different data parameters and different feature types. Based on the feature type of the first cluster in a second mapping relationship set, a compression strategy for the first cluster is determined. This second mapping relationship set includes mapping relationships between different feature types and different compression strategies. Thus, by determining the compression strategy for the first cluster based on the data parameters of the data included in the first cluster, the compression strategy can achieve fast and lossless compression of at least one data element included in the first cluster, effectively improving the compression effect.

[0051] In one embodiment, the plurality of clusters includes a first cluster, and the method further includes: The sampling size is calculated based on the number of at least one data point included in the first cluster. Based on the sampling quantity, at least one data point included in the compressed first cluster is sampled to obtain sampling compression parameters, which include at least one of compression ratio, compression error, compression speed and decompression speed. An evaluation result is generated based on the sampling compression parameters. The evaluation result is used to characterize whether the compression of at least one data included in the first cluster has achieved a preset compression result. If the evaluation result is used to characterize the failure to achieve the preset compression result, the compression strategy corresponding to the first cluster is adjusted. At least one piece of data included in the first cluster is compressed based on the adjusted compression strategy.

[0052] It should be noted that after determining the compression strategy corresponding to the cluster, since the compression strategy is pre-configured, it is also necessary to determine the actual compression effect of the compression strategy and adjust the compression strategy according to the actual effect, so as to further improve the compression quality of the data.

[0053] The sampling size mentioned above can be obtained based on a preset ratio. For example, the sampling size is the product of the number of at least one data point and the preset ratio. In some implementations, the preset ratio can be dynamically adjusted according to the number of at least one data point. For example, the preset ratio is 20% when the data volume is less than 1000 windows, and 10% when the data volume is greater than or equal to 1000 windows, thereby ensuring that the sampling size is greater than or equal to 100.

[0054] The above sampling compression parameters are parameters for compressing the data included in the cluster, including at least one of compression ratio, compression error, compression speed, and decompression speed. Specifically, compression ratio = original data size / compressed data size; compression error varies under different compression conditions; in lossless compression scenarios, it can be verifying data integrity (i.e., MD5 checksum consistency), while in lossy compression scenarios, it is calculating RMSE and peak error (PE = max|x_actual - x_restored|); compression speed = compressed data size / compression time; decompression speed = decompressed data size / decompression time.

[0055] In some implementations, evaluation results are generated based on sampling compression parameters. These results characterize whether the compression of at least one data point within the first cluster achieves a preset compression result, and the data compression effectiveness can be determined through the evaluation results. The evaluation results may include parameters such as evaluation metrics for each strategy group and identifiers of substandard data (e.g., windows where RMSE > threshold).

[0056] The above-mentioned preset compression results are the results of pre-configuration for different sampling compression parameters. If the evaluation result reaches the preset compression result, it is considered that the current compression strategy can effectively compress the data and no adjustment is needed; if the evaluation result does not reach the preset compression result, it is considered that the current compression strategy cannot effectively compress the data, and the compression strategy corresponding to the first cluster needs to be adjusted.

[0057] In some implementations, the compression strategy corresponding to the first cluster is adjusted according to pre-configured rules. For example, if the DFT compression RMSE exceeds the standard, the number of main frequency retentions M is increased (+2 each time, maximum M=20). If the compression speed of the linear prediction residual is not up to standard, the linear prediction order is reduced (e.g., from order 2 to order 1). After adjustment, the samples that do not meet the standard are re-evaluated for compression until the index meets the standard, ensuring that the compression quality is controllable.

[0058] In this embodiment of the invention, a sampling quantity is calculated based on the number of at least one data point included in the first cluster; the at least one data point included in the compressed first cluster is sampled based on the sampling quantity to obtain sampling compression parameters, which include at least one of compression ratio, compression error, compression speed, and decompression speed; an evaluation result is generated based on the sampling compression parameters, which characterizes whether the compression of at least one data point included in the first cluster achieves a preset compression result; if the evaluation result indicates that the preset compression result has not been achieved, the compression strategy corresponding to the first cluster is adjusted; the at least one data point included in the first cluster is compressed based on the adjusted compression strategy. Thus, by obtaining the evaluation result through sampling, the compression effect is evaluated; and by adjusting the compression strategy based on the preset compression result and the evaluation result, the compression quality of the data is further improved.

[0059] In one embodiment, the clustering of the plurality of data based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain a plurality of clusters includes: Based on at least one of the time-domain features, the frequency-domain features, and the anomaly features, and a preset parameter, the multiple data are clustered to obtain multiple clusters; The method further includes: Acquire multiple historical data of a first compression strategy, wherein the first compression strategy is a strategy for compressing at least one data included in a first cluster, and each historical data includes compression parameters and clustering parameters, wherein the compression parameters include at least one of compression ratio, compression range and compression speed, and the clustering parameters include at least one of cluster center, window length and cluster distance; A compression score is calculated for each historical data point based on the compression parameters included in each historical data point. The preset parameters are optimized based on the compression score and clustering parameters of each historical data point.

[0060] The aforementioned preset parameters are those used during clustering, such as cluster radius. It should be noted that since the same clustering strategy is applied to all data within a cluster in this invention, it is necessary to ensure, as far as possible, that the characteristics of data within the same cluster are similar after clustering. However, since the preset parameters are set in advance based on experience, inaccurate preset parameters may lead to inaccurate clustering, affecting the final data compression effect. Therefore, this embodiment introduces historical data and compression scores to optimize the preset parameters.

[0061] The aforementioned historical data can be stored in a historical database. Each time data compression is performed, the cluster-related parameters obtained from clustering are stored in the historical database, including compression parameters and clustering parameters, so that historical data can be retrieved from the historical database later.

[0062] The aforementioned compression parameters include at least one of compression ratio, compression range, and compression speed. A compression score is calculated for each historical data point based on the compression parameters included in each historical data point. This can be achieved by weighting at least one of the compression ratio, compression range, and compression speed. For example, compression score = α × compression ratio + β × (1 - RMSE / range) + γ × compression speed, where α, β, and γ are weighting coefficients. These weighting coefficients can be configured according to business requirements, such as α = 0.4, β = 0.4, and γ = 0.2.

[0063] Furthermore, after obtaining the compressed score for each historical data point, the preset parameters can be optimized based on the compressed score and clustering parameters. For example, when the compressed score is higher than the set score threshold, the distance threshold of incremental clustering and the compression parameters (such as the default M value of DFT) can be adjusted to improve the strategy score by ≥5%.

[0064] In some implementations, in terms of scheduling, the system connects to a cloud-based scheduling platform (such as Airflow) to support two methods: scheduled scheduling (execute compression tasks according to a preset cycle, such as at midnight every day, to avoid peak business periods) and event-triggered scheduling (automatically trigger compression tasks when the amount of new data added to the data lake is greater than or equal to a threshold, such as 100GB), to ensure timely data processing.

[0065] In this embodiment of the invention, a compression score is calculated for each historical data point based on the compression parameters included in each historical data point; the preset parameters are then optimized based on the compression score and clustering parameters of each historical data point. In this way, by optimizing the preset parameters through the compression score and clustering parameters, the clustering accuracy is improved, enabling the use of more accurate compression strategies to compress the data and thus improving the quality of data compression.

[0066] In some implementations, the compression system can output information such as processing results, evaluation results, and compression strategy mapping relationships as metadata in JSON / YAML format for use by the data lake metadata catalog system or interface. The system supports integration with cloud platform scheduling services (such as Airflow and AWS Glue) to run compression tasks on a scheduled basis or trigger compression based on data update events.

[0067] Please see Figure 2 , Figure 2 This is a structural diagram of a data compression device provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the data compression device 200 includes: The acquisition module 201 is used to acquire multiple data points, which are data from different data sources; Extraction module 202 is used to extract at least one of time-domain features, frequency-domain features and anomaly features from multiple collected data. The time-domain features are used to characterize the changes of the multiple data over time, the frequency-domain features are used to represent the multiple data at different frequency components, and the anomaly features are used to characterize the abnormal data in the multiple data. Clustering module 203 is used to cluster the plurality of data based on at least one of the time domain features, the frequency domain features and the anomaly features to obtain a plurality of clusters, each cluster including at least one data; Generation module 204 is used to generate a compression strategy corresponding to each cluster. Compression module 205 is used to compress at least one piece of data included in each cluster based on the compression strategy corresponding to each cluster.

[0068] In one embodiment, the acquisition module 201 includes: The first acquisition unit is used to acquire the sampling frequency of each data source; The first calculation unit is used to calculate the window length of each data source based on the sampling frequency and preset time interval of each data source; The acquisition unit is used to acquire data from each data source based on the window length of each data source, thereby obtaining the multiple data sets.

[0069] In one embodiment, the data compression device 200 further includes: The processing module is used to preprocess the plurality of data to obtain the preprocessed plurality of data; The extraction module 202 includes: An extraction unit is used to extract at least one of the time-domain features, frequency-domain features, and anomaly features of the preprocessed plurality of data; The preprocessing includes at least one of the following: Standardize the fields in the multiple data sets; Convert the format of the multiple data items; Delete data with abnormal values ​​from the multiple datasets.

[0070] In one embodiment, the plurality of clusters includes a first cluster, and the generation module 204 includes: The second acquisition unit is used to acquire data parameters of at least one data included in the first cluster, wherein the data parameters include at least one of abnormal data ratio, trend slope, main frequency energy ratio and maximum amplitude difference; The first determining unit is configured to determine the feature type of the first cluster based on the data parameters in the first mapping relationship set, wherein the first mapping relationship set includes mapping relationships between different data parameters and different feature types. The second determining unit is used to determine the compression strategy of the first cluster based on the feature type of the first cluster in the second mapping relationship set, wherein the second mapping relationship set includes the mapping relationship between different feature types and different compression strategies.

[0071] In one embodiment, the plurality of clusters includes a first cluster, and the data compression device 200 further includes: The second calculation unit is used to calculate the sampling quantity based on the number of at least one data included in the first cluster; A sampling unit is used to sample at least one data included in the compressed first cluster based on the sampling quantity to obtain sampling compression parameters, wherein the sampling compression parameters include at least one of compression ratio, compression error, compression speed and decompression speed. A generation unit is used to generate an evaluation result based on the sampling compression parameters. The evaluation result is used to characterize whether the compression of at least one data included in the first cluster has achieved a preset compression result. An adjustment unit is used to adjust the compression strategy corresponding to the first cluster when the evaluation result is used to characterize that the preset compression result has not been achieved. A compression unit is used to compress at least one piece of data included in the first cluster based on an adjusted compression strategy.

[0072] In one embodiment, the clustering module 203 includes: A clustering unit is used to cluster the multiple data based on at least one of the time-domain features, the frequency-domain features, and the anomaly features and a preset parameter to obtain multiple clusters. The data compression device 200 further includes: The acquisition module is used to acquire multiple historical data of a first compression strategy. The first compression strategy is a strategy for compressing at least one piece of data included in a first cluster. Each piece of historical data includes compression parameters and clustering parameters. The compression parameters include at least one of compression ratio, compression range, and compression speed. The clustering parameters include at least one of cluster center, window length, and cluster distance. The calculation module is used to calculate the compression score of each historical data based on the compression parameters included in each historical data. The optimization module is used to optimize the preset parameters based on the compression score and clustering parameters of each historical data.

[0073] The data processing apparatus provided in this embodiment of the invention can implement each process of the above-described data compression method, with one-to-one correspondence of technical features and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0074] It should be noted that the data compression device in the embodiments of the present invention can be a device, or it can be a component, integrated circuit, or chip in an electronic device.

[0075] This invention also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the above-described functionality. Figure 1 The various processes of the data compression method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0076] For details, see Figure 3 As shown, this embodiment of the invention also provides an electronic device, including a bus 301, a transceiver 302, an antenna 303, a bus interface 304, a processor 305, and a memory 306.

[0077] The processor 305 is used to collect multiple data, which are data from different data sources; The processor 305 is further configured to extract at least one of time-domain features, frequency-domain features, and anomaly features from the acquired multiple data. The time-domain features are used to characterize the changes of the multiple data over time, the frequency-domain features are used to represent the multiple data at different frequency components, and the anomaly features are used to characterize the abnormal data among the multiple data. The processor 305 is further configured to cluster the plurality of data based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain a plurality of clusters, each cluster including at least one piece of data; The processor 305 is also used to generate a compression strategy corresponding to each cluster. The processor 305 is further configured to compress at least one piece of data included in each cluster based on the compression strategy corresponding to each cluster.

[0078] In one embodiment, the collection of multiple data points includes: Obtain the sampling frequency for each data source; The window length of each data source is calculated based on the sampling frequency and preset time interval of each data source. Data is collected from each data source based on the window length of each data source to obtain the multiple data sets.

[0079] In one embodiment, the processor 305 is further configured to preprocess the plurality of data to obtain the preprocessed plurality of data; The extraction of at least one of the time-domain features, frequency-domain features, and anomaly features from the multiple data obtained includes: Extract at least one of the time-domain features, frequency-domain features, and anomaly features from the preprocessed data; The preprocessing includes at least one of the following: Standardize the fields in the multiple data sets; Convert the format of the multiple data items; Delete data with abnormal values ​​from the multiple datasets.

[0080] In one embodiment, the plurality of clusters includes a first cluster, and generating a compression strategy corresponding to each cluster includes: Obtain data parameters of at least one data included in the first cluster, wherein the data parameters include at least one of the following: abnormal data ratio, trend slope, main frequency energy ratio, and maximum amplitude difference; The feature type of the first cluster is determined based on the data parameters in the first mapping relationship set, wherein the first mapping relationship set includes the mapping relationship between different data parameters and different feature types; The compression strategy for the first cluster is determined based on the feature type of the first cluster in the second mapping relationship set, wherein the second mapping relationship set includes the mapping relationship between different feature types and different compression strategies.

[0081] In one embodiment, the plurality of clusters includes a first cluster, and the processor 305 is further configured to calculate the number of samples based on the number of at least one data included in the first cluster; The processor 305 is further configured to sample at least one data included in the compressed first cluster based on the sampling quantity to obtain sampling compression parameters, wherein the sampling compression parameters include at least one of compression ratio, compression error, compression speed and decompression speed. The processor 305 is further configured to generate an evaluation result based on the sampling compression parameters, the evaluation result being used to characterize whether the compression of at least one data included in the first cluster has achieved a preset compression result. The processor 305 is further configured to adjust the compression strategy corresponding to the first cluster when the evaluation result is used to characterize that the preset compression result has not been achieved; The processor 305 is further configured to compress at least one piece of data included in the first cluster based on an adjusted compression strategy.

[0082] In one embodiment, the clustering of the plurality of data based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain a plurality of clusters includes: Based on at least one of the time-domain features, the frequency-domain features, and the anomaly features, and a preset parameter, the multiple data are clustered to obtain multiple clusters; The method further includes: Acquire multiple historical data of a first compression strategy, wherein the first compression strategy is a strategy for compressing at least one data included in a first cluster, and each historical data includes compression parameters and clustering parameters, wherein the compression parameters include at least one of compression ratio, compression range and compression speed, and the clustering parameters include at least one of cluster center, window length and cluster distance; A compression score is calculated for each historical data point based on the compression parameters included in each historical data point. The preset parameters are optimized based on the compression score and clustering parameters of each historical data point.

[0083] exist Figure 3 In this context, a bus architecture (represented by bus 301) is used. Bus 301 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 305 and memory represented by memory 306. Bus 301 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 304 provides an interface between bus 301 and transceiver 302. Transceiver 302 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 305 is transmitted over a wireless medium via antenna 303, which further receives data and transmits it to processor 305.

[0084] Processor 305 manages bus 301 and general processing, and also provides various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 306 can be used to store data used by processor 305 during operation.

[0085] Optionally, the processor 305 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU).

[0086] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the above-described functions. Figure 1 The various processes corresponding to the data compression method embodiments achieve the same technical effect, and will not be described again here to avoid repetition. The computer-readable storage medium mentioned includes, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0087] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The various processes of the corresponding data compression method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0088] In the embodiments of this invention, the terms "first," "second," etc., are used to distinguish similar object parameters and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected object parameters, such as A and / or B and / or C, representing seven possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0089] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of this application.

[0091] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data compression method, characterized in that, include: Collect multiple data points, which are data from different data sources; Extract at least one of the time-domain features, frequency-domain features, and anomaly features from multiple collected data. The time-domain features are used to characterize the changes of the multiple data over time, the frequency-domain features are used to represent the multiple data at different frequency components, and the anomaly features are used to characterize the anomaly data among the multiple data. The multiple data are clustered based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain multiple clusters, each cluster including at least one data point; Generate a compression strategy corresponding to each cluster; Based on the compression strategy corresponding to each cluster, at least one piece of data included in each cluster is compressed; The data are clustered based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain multiple clusters, including: Based on at least one of the time-domain features, the frequency-domain features, and the anomaly features, and a preset parameter, the multiple data are clustered to obtain multiple clusters, wherein the preset parameter is the parameter used during clustering. The method further includes: Acquire multiple historical data of a first compression strategy, wherein the first compression strategy is a strategy for compressing at least one data included in a first cluster, and each historical data includes compression parameters and clustering parameters, wherein the compression parameters include at least one of compression ratio, compression range and compression speed, and the clustering parameters include at least one of cluster center, window length and cluster distance; A compression score is calculated for each historical data point based on the compression parameters included in each historical data point. The preset parameters are optimized based on the compression score and clustering parameters of each historical data point; The compression parameters include compression ratio, compression range, and compression speed. The compression score is calculated as α × compression ratio + β × (1 - RMSE / range) + γ × compression speed, where α, β, and γ are weighting coefficients.

2. The method as described in claim 1, characterized in that, The collection of multiple data points includes: Obtain the sampling frequency for each data source; The window length of each data source is calculated based on the sampling frequency and preset time interval of each data source. Data is collected from each data source based on the window length of each data source to obtain the multiple data sets.

3. The method as described in claim 2, characterized in that, The method further includes: The plurality of data are preprocessed to obtain the preprocessed plurality of data; The extraction of at least one of the time-domain features, frequency-domain features, and anomaly features from the multiple data obtained includes: Extract at least one of the time-domain features, frequency-domain features, and anomaly features from the preprocessed data; The preprocessing includes at least one of the following: Standardize the fields in the multiple data sets; Convert the format of the multiple data items; Delete data with abnormal values ​​from the multiple datasets.

4. The method according to any one of claims 1 to 3, characterized in that, The plurality of clusters includes a first cluster, and the generation of a compression strategy corresponding to each cluster includes: Obtain data parameters of at least one data included in the first cluster, wherein the data parameters include at least one of the following: abnormal data ratio, trend slope, main frequency energy ratio, and maximum amplitude difference; The feature type of the first cluster is determined based on the first mapping relationship set and the data parameters. The first mapping relationship set includes the mapping relationship between different data parameters and different feature types. The compression strategy for the first cluster is determined based on the second mapping relationship set and the feature type of the first cluster. The second mapping relationship set includes the mapping relationship between different feature types and different compression strategies.

5. The method according to any one of claims 1 to 3, characterized in that, The plurality of clusters includes a first cluster, and the method further includes: The sampling size is calculated based on the number of at least one data point included in the first cluster. Based on the sampling quantity, at least one data point included in the compressed first cluster is sampled to obtain sampling compression parameters, which include at least one of compression ratio, compression error, compression speed and decompression speed. An evaluation result is generated based on the sampling compression parameters. The evaluation result is used to characterize whether the compression of at least one data included in the first cluster has achieved a preset compression result. If the evaluation result is used to characterize the failure to achieve the preset compression result, the compression strategy corresponding to the first cluster is adjusted. At least one piece of data included in the first cluster is compressed based on the adjusted compression strategy.

6. A data compression device, characterized in that, include: The data acquisition module is used to acquire multiple data points, which are data from different data sources. An extraction module is used to extract at least one of time-domain features, frequency-domain features, and anomaly features from multiple collected data. The time-domain features are used to characterize the changes of the multiple data over time, the frequency-domain features are used to represent the multiple data at different frequency components, and the anomaly features are used to characterize the abnormal data among the multiple data. The clustering module is used to cluster the multiple data based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain multiple clusters, each cluster including at least one data point; The generation module is used to generate the compression strategy corresponding to each cluster. A compression module is used to compress at least one piece of data included in each cluster based on the compression strategy corresponding to each cluster. The clustering module includes: A clustering unit is used to cluster the multiple data based on at least one of the time-domain features, the frequency-domain features, and the anomaly features, and a preset parameter, to obtain multiple clusters, wherein the preset parameter is a parameter used during clustering. The data compression device further includes: The acquisition module is used to acquire multiple historical data of a first compression strategy. The first compression strategy is a strategy for compressing at least one piece of data included in a first cluster. Each piece of historical data includes compression parameters and clustering parameters. The compression parameters include at least one of compression ratio, compression range, and compression speed. The clustering parameters include at least one of cluster center, window length, and cluster distance. The calculation module is used to calculate the compression score of each historical data based on the compression parameters included in each historical data. An optimization module is used to optimize the preset parameters based on the compression score and clustering parameters of each historical data. The compression parameters include compression ratio, compression range, and compression speed. The compression score is calculated as α × compression ratio + β × (1 - RMSE / range) + γ × compression speed, where α, β, and γ are weighting coefficients.

7. An electronic device, characterized in that, Including transceivers and processors, The processor is used to collect multiple data points, which are data from different data sources; The processor is further configured to extract at least one of time-domain features, frequency-domain features, and anomaly features from the acquired multiple data sets. The time-domain features are used to characterize the changes of the multiple data sets over time, the frequency-domain features are used to represent the multiple data sets at different frequency components, and the anomaly features are used to characterize the anomaly data among the multiple data sets. The processor is further configured to cluster the plurality of data based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain a plurality of clusters, each cluster including at least one piece of data; The processor is also used to generate a compression strategy corresponding to each cluster. The processor is further configured to compress at least one piece of data included in each cluster based on the compression strategy corresponding to each cluster; The data are clustered based on at least one of the time-domain features, the frequency-domain features, and the anomaly features to obtain multiple clusters, including: Based on at least one of the time-domain features, the frequency-domain features, and the anomaly features, and a preset parameter, the multiple data are clustered to obtain multiple clusters, wherein the preset parameter is the parameter used during clustering. The processor is also used for: Acquire multiple historical data of a first compression strategy, wherein the first compression strategy is a strategy for compressing at least one data included in a first cluster, and each historical data includes compression parameters and clustering parameters, wherein the compression parameters include at least one of compression ratio, compression range and compression speed, and the clustering parameters include at least one of cluster center, window length and cluster distance; A compression score is calculated for each historical data point based on the compression parameters included in each historical data point. The preset parameters are optimized based on the compression score and clustering parameters of each historical data point; The compression parameters include compression ratio, compression range, and compression speed. The compression score is calculated as α × compression ratio + β × (1 - RMSE / range) + γ × compression speed, where α, β, and γ are weighting coefficients.

8. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data compression method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data compression method as described in any one of claims 1 to 5.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the data compression method as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Carrier communication data acquisition and data optimization method

    CN117082153A

  • Multi-source heterogeneous data compression and transmission method and system for intelligent fusion terminal

    CN120812139A

  • Data compression storage method and device and electronic equipment

    CN121749992A

  • Query method and device for time sequence database and medium

    CN122285753A