Data compression method and device

Through the resampling method of feature analysis of time-series data and the sampling step size determination, the problem of data distortion in time-series data compression is solved, efficient data compression and resource saving is achieved, while retaining data characteristics.

CN113849505BActive Publication Date: 2024-11-26LENOVO (BEIJING) LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111076416.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-14
Publication Date
2024-11-26
Estimated Expiration
2041-09-14

AI Technical Summary

Technical Problem

When the prior art compresses time series data, it is difficult to retain the original data characteristics, resulting in distortion of compressed data and unable to truly reflect the data characteristics.

Method used

By performing data analysis on the data to be processed, the data to be compressed is determined, and the sampling step is determined according to the characteristics of each data to resample data to realize data compression.

Benefits of technology

Without reducing the quality of data analysis, the data transmission volume and resource consumption are reduced, ensuring that compressed data can truly reflect the characteristics of the original data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113849505B_ABST
    Figure CN113849505B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a data compression method and device, wherein the method includes: obtaining at least two data to be processed; determining at least one data to be compressed from the at least two data to be processed by performing data analysis on each of the data to be processed; performing feature analysis on each of the data to be compressed to determine a sampling step length of a service corresponding to each of the data to be compressed; and using the sampling step length to resample data corresponding to the at least one data to be compressed to obtain at least one compressed data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of big data, and are related to but not limited to a data compression method and device. Background Art

[0002] In the data information age, most information exists in the form of time series data. When processing and analyzing time series data, a large number of data curves may be presented in a graph at the same time. When a graph is drawn with many curves, the amount of network transmission data will be huge, and the front-end page will consume more resources when presenting data.

[0003] Related technologies reduce the amount of data transmission by compressing all time series data, but this will cause data distortion. Therefore, how to provide a data compression method that can retain the characteristics of the original data is a problem that needs to be solved urgently. Summary of the invention

[0004] Based on the problems in the related art, the embodiments of the present application provide a data compression method and device.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a data compression method, including:

[0007] Obtain at least two pieces of data to be processed;

[0008] By performing data analysis on each of the data to be processed, at least one data to be compressed is determined from the at least two data to be processed;

[0009] Performing feature analysis on each of the data to be compressed, and determining a sampling step length of a service corresponding to each of the data to be compressed;

[0010] The sampling step length is used to resample the data of the service corresponding to the at least one data to be compressed to obtain at least one compressed data.

[0011] The present application provides a data compression device, including:

[0012] An acquisition module, used for acquiring at least two pieces of data to be processed;

[0013] A data analysis module, configured to determine at least one to-be-compressed data from among the at least two to-be-processed data by performing data analysis on each of the to-be-processed data;

[0014] A feature analysis module, used to perform feature analysis on each of the data to be compressed, and determine a sampling step length of a service corresponding to each of the data to be compressed;

[0015] The resampling module is used to resample data corresponding to at least one of the services to be compressed by adopting the sampling step length to obtain at least one compressed data.

[0016] The present application provides a data compression device, including:

[0017] The memory is used to store executable instructions; the processor is used to implement the above-mentioned data compression method when executing the executable instructions stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-mentioned data compression method.

[0019] The data compression method and device provided in the embodiment of the present application, by performing data analysis on at least two acquired data to be processed, determining at least one data to be compressed that needs to be compressed, performing feature analysis on each data to be compressed, determining the sampling step corresponding to each data to be compressed when performing data compression, and resampling the data corresponding to at least one data to be compressed by the determined sampling step to obtain compressed data. In this way, the embodiment of the present application determines the data to be compressed according to the characteristics of each data to be processed, and resamples different data to be compressed using different step sizes, so that the embodiment of the present application compresses the data without reducing the quality of data analysis, reduces the amount of data transmission, and reduces the consumption of resources and memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of an application scenario of the data compression method provided in an embodiment of the present application;

[0021] Figure 2 It is a flowchart of a data compression method provided in an embodiment of the present application;

[0022] Figure 3 It is a flowchart of a data compression method provided in an embodiment of the present application;

[0023] Figure 4 It is a flowchart of a data compression method provided in an embodiment of the present application;

[0024] Figure 5 It is a flowchart of a data compression method provided in an embodiment of the present application;

[0025] Figure 6 It is a flowchart of a data compression method provided in an embodiment of the present application;

[0026] Figure 7 It is a flowchart of a data compression method provided in an embodiment of the present application;

[0027] Figure 8 It is a flowchart of a data compression method provided in an embodiment of the present application;

[0028] Fig. 9 It is a flowchart of a data compression method provided in an embodiment of the present application;

[0029] Fig.10 It is a schematic diagram of the structure of the data compression device provided in an embodiment of the present application;

[0030] Fig.11 It is a schematic diagram of the composition structure of the data compression device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the technical field of the embodiments of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0033] In the process of analyzing different time series data, the density of the time series data can be observed to qualitatively judge the quality of data capture and the time interval of data loss. Through the histogram of the standard deviation of the data center monitoring indicators, it is found that most monitoring curves have small fluctuations, and only a small amount of monitoring data of a small number of indicators have certain fluctuations. By observing the time series data visualization components related to the monitoring system (for example, visualization components such as Zabbix or Grafana), the related technology finds that many tools can dynamically generate resampling steps, but such settings are for all transmission curves, not for different curves with different data characteristics. Therefore, the compression data method of the related technology will cause the compressed data to be distorted and cannot truly reflect the characteristics of the original data.

[0034] Based on the problems in the related art, an embodiment of the present application provides a data compression method, which performs data analysis on at least two acquired data to be processed, determines at least one data to be compressed that needs to be compressed, performs feature analysis on each data to be compressed, determines the sampling step corresponding to each data to be compressed when performing data compression, and resamples data corresponding to at least one data to be compressed by the determined sampling step to achieve data compression. In this way, the embodiment of the present application determines the data to be compressed according to the characteristics of each data to be processed, and resamples different data to be compressed using different step sizes, so that the embodiment of the present application compresses the data without reducing the quality of data analysis, reduces the amount of data transmission, and reduces the consumption of resources and memory.

[0035] Figure 1 is a schematic diagram of an application scenario of the data compression method provided in the embodiment of the present application, such as Figure 1 As shown, the data compression system for implementing the data compression method includes a terminal 10, a network 20 and a server 30. At least two time series data can be displayed on the terminal 10. The server 30 obtains a data compression request sent by the terminal 10 through the network 20. The server 30 performs data analysis on at least two data to be processed, determines at least one data to be compressed that needs to be compressed, performs feature analysis on each data to be compressed, determines the sampling step corresponding to each data to be compressed when performing data compression, resamples data for a service corresponding to at least one data to be compressed through the determined sampling step, obtains compressed data, and sends the compressed data to the terminal 10 through the network 20. The terminal 10 visualizes the compressed data, and the user can directly display the received compressed data on the current interface 10-1.

[0036] The data compression method provided by the embodiment of the present application will be described below in conjunction with the exemplary application and implementation of the server provided by the embodiment of the present application. Figure 2 , Figure 2 is a flow chart of the data compression method provided in the embodiment of the present application, which will be combined with Figure 2 The steps shown are explained.

[0037] Step S201: Obtain at least two data to be processed.

[0038] In some embodiments, the data to be processed may be data to be analyzed, and the data to be analyzed may be a data column recorded in chronological order such as time series data.

[0039] Step S202: performing data analysis on each of the data to be processed, and determining at least one data to be compressed from the at least two data to be processed.

[0040] In the embodiment of the present application, by monitoring the data to be processed, it can be known that the fluctuation of most monitoring curves is very small, and only a small amount of monitoring data of a small number of indicators has certain fluctuations. Therefore, performing data analysis on the data to be processed can refer to performing data fluctuation analysis on the data to be processed, for example, performing difference analysis between extreme points of the data, or calculating the standard deviation of each data to be processed, and determining whether the data to be processed needs to be compressed by the result of the fluctuation analysis. Through data analysis, the data to be compressed that needs to be compressed can be quickly determined from a large amount of data to be processed.

[0041] Here, the extreme value difference result or standard deviation result can be used to determine whether the fluctuation of each data to be processed reaches the preset fluctuation threshold, so as to determine whether the data to be processed needs to be compressed. When the extreme value difference result of any data is less than the preset fluctuation threshold, it means that the fluctuation of the data is small. After compressing the data, the compressed data will not be distorted, and the compressed data can still truly reflect the data characteristics; when the extreme value difference result of a certain data is greater than the preset fluctuation threshold, it means that the fluctuation of the data is large. After compressing the data, the compressed data will not be able to present the real data characteristics. Therefore, it is also necessary to analyze the degree of data dispersion, such as standard deviation analysis, to determine whether to compress the data.

[0042] Step S203: Perform feature analysis on each of the data to be compressed, and determine a sampling step length of a service corresponding to each of the data to be compressed.

[0043] In some embodiments, after the data to be compressed is determined, the sampling step length when compressing the data to be compressed is determined by performing feature analysis on the data to be compressed. Here, feature analysis may refer to performing sampling interval analysis on the compressed data according to the features of the compressed data to obtain the sampling step length of each data when compressing the data.

[0044] Step S204: Resample the data of the service corresponding to the at least one data to be compressed by using the sampling step length to obtain at least one compressed data.

[0045] In the embodiment of the present application, each data to be processed corresponds to a service. The service corresponding to each data to be compressed is resampled by using the sampling step corresponding to each data to be compressed to obtain compressed data.

[0046] By performing data analysis on at least two acquired data to be processed, at least one data to be compressed that needs to be compressed is determined, and feature analysis is performed on each data to be compressed, and the sampling step corresponding to each data to be compressed when data compression is performed is determined. By using the determined sampling step, data resampling is performed on the service corresponding to at least one data to be compressed to achieve data compression. In this way, the embodiment of the present application determines the data to be compressed according to the features of each data to be processed, and resamples different data to be compressed using different step sizes, so that the data compression method provided by the embodiment of the present application can reduce the amount of data transmission and save resources while compressing the data, so that the compressed data can truly reflect the features of the original data.

[0047] In some embodiments, the data to be compressed may be determined by performing fluctuation analysis on the data to be processed. Figure 3 is a flow chart of a data compression method provided in an embodiment of the present application, such as Figure 3 As shown, in some embodiments, step S202 may be implemented by the following steps:

[0048] Step S301: Perform fluctuation analysis on each of the data to be processed to obtain a fluctuation difference corresponding to each of the data to be processed.

[0049] In some embodiments, fluctuation analysis may be an analysis method such as extreme value analysis that can determine data volatility. The extreme value difference corresponding to each data to be processed, that is, the fluctuation difference, can be determined through extreme value analysis, and the data fluctuation of each data to be processed can be determined through the extreme value difference.

[0050] Step S302: classify the data to be processed according to the fluctuation difference, and divide the data to be processed into first data to be compressed or second data to be compressed.

[0051] In some embodiments, step S302 may be implemented by the following steps:

[0052] Step S3021: If the fluctuation difference is less than or equal to a preset fluctuation threshold, the data to be processed is determined as the first data to be compressed.

[0053] Here, the fluctuation threshold can be pre-set by the technician. If the fluctuation difference of a certain data to be processed is less than or equal to the preset fluctuation threshold, it means that the fluctuation of the data to be processed is small and the data can be compressed. The data to be processed is then determined as the first data to be compressed. Therefore, the first data to be compressed refers to the data to be processed with smaller fluctuations.

[0054] It should be noted that the extreme value analysis of the data to be processed can be performed by calculating the difference between each adjacent group of maximum and minimum values ​​in the data, or it can be performed by dividing the data into different intervals and determining whether to determine the data to be processed as the first data to be compressed by calculating the difference between the maximum and minimum values ​​of each interval.

[0055] Step S3022: If the fluctuation difference is greater than the fluctuation threshold, determine the discrete value of the data to be processed.

[0056] In some embodiments, when the extreme value difference of the data to be processed is greater than a preset fluctuation threshold, it indicates that the fluctuation of the data to be processed is large, and it is necessary to determine whether data compression is required based on the degree of discreteness of the data to be processed. Here, determining the discrete value of the data to be processed may be calculating the standard deviation or variance of the data to be processed, and the obtained standard deviation or variance value is the discrete value of the data to be processed. The discrete value corresponding to each data to be processed is used to determine the degree of discreteness of the data.

[0057] Step S3023: If it is determined that the discrete value is less than or equal to a preset discrete threshold, the data to be processed is determined as second data to be compressed.

[0058] In some embodiments, the discrete threshold can be preset by a technician, for example, it can be marked by a tool such as a histogram. When the standard deviation or variance value of the data to be processed is less than or equal to the preset discrete threshold, it means that the difference between most of the values ​​in the data to be processed and the average value of all the values ​​is small, that is, the discrete degree of the data to be processed is small, and it can be considered that the data to be processed can reflect the characteristics of the original data after compression, and the data to be processed can be determined as the second data to be compressed. The second data to be compressed refers to the data to be processed with large fluctuations but small data discreteness.

[0059] In some embodiments, when the standard deviation or variance value of the data to be processed is greater than a preset discrete threshold value, it means that the difference between most of the values ​​in the data to be processed and the average value of all the values ​​is large, that is, the discrete degree of the data to be processed is large. If the processed data is resampled and compressed, most of the feature points in the data to be processed cannot be collected, resulting in the compressed data being unable to reflect the characteristics of the original data. Therefore, when the standard deviation or variance value of the data to be processed is greater than the preset discrete threshold value, the data to be processed will not be resampled, and the data can be stored and displayed on the terminal interface together with other compressed data.

[0060] Step S303: determine the first data to be compressed and the second data to be compressed as the data to be compressed.

[0061] The embodiment of the present application performs fluctuation analysis and discrete analysis on the data to be processed, and accurately and quickly determines the data to be compressed from the data to be processed, so that the compressed data not only reduces the amount of data transmitted at the front end, but also data with more features will not be compressed, and the problem of compressed data being unable to reflect the characteristics of the original data will not occur.

[0062] In some embodiments, since the first data to be compressed and the second data to be compressed represent different types of data, different methods are used to determine the corresponding sampling step lengths for the first data to be compressed and the second data to be compressed. Figure 4 is a flow chart of a data compression method provided in an embodiment of the present application, such as Figure 4 As shown, in some embodiments, step S203 may be implemented by the following steps:

[0063] Step S401: Perform sampling interval analysis on the first data to be compressed to obtain a visualized sampling interval of the first data to be compressed.

[0064] In some embodiments, the first data to be compressed refers to data to be processed with small fluctuations. The sampling interval analysis of the first data to be compressed can be to analyze the interval that can be queried when the first data to be compressed is visualized, and use the interval of the first data to be compressed once visualized as the visualization sampling interval of the first data to be compressed.

[0065] In some embodiments, since the fluctuation of the first data to be compressed is small, the data of the start time point and the end time point of the first data to be compressed can also be directly used as the compressed data of the first data to be compressed, that is, when the first data to be compressed is resampled, only the data of the start time point and the end time point are collected.

[0066] Step S402: Determine the duration corresponding to the visualization sampling interval as the sampling step length of the service corresponding to the first data to be compressed.

[0067] Here, each visualization sampling interval has a duration, which can be used as the sampling step of the service corresponding to the first data to be compressed during data compression. For example, the total duration of the first data to be compressed is 60 minutes, and after sampling interval analysis, the visualization sampling interval is 1 minute. When resampling the service corresponding to the first compressed data, data is collected once every 1 minute from the beginning of data collection until data collection is completed to form compressed data.

[0068] In some embodiments, for the second data to be compressed, step S203 may also be implemented by the following steps:

[0069] Step S403: Obtain an information table, wherein the information table at least includes: a resampling step corresponding to each service.

[0070] Here, the information table may be a resampling step corresponding to each business set by a technician according to the characteristics of each business. The characteristics of a business may be functions of the business, for example, an attribute may be functions such as information transmission or storage.

[0071] Step S404: Perform feature analysis on each of the second data to be compressed, and obtain corresponding attributes of each of the second data to be compressed.

[0072] In some embodiments, the second data to be compressed refers to data to be processed with large fluctuations but small data discreteness. Such data can be subjected to feature analysis to obtain data attributes. The business corresponding to each second data to be compressed can be determined by the attribute corresponding to each second data to be compressed.

[0073] Step S405: Determine the service corresponding to each of the second data to be compressed according to the attribute of each of the second data to be compressed.

[0074] Step S406: Determine a sampling step length of the service corresponding to each piece of the second data to be compressed according to the information table and the service corresponding to each piece of the second data to be compressed.

[0075] In some embodiments, the information table includes the resampling step length of each service. After the service corresponding to the data is determined according to the service attributes, the sampling step length of the service corresponding to the second data to be compressed can be determined by a table lookup method.

[0076] The embodiment of the present application determines the sampling step size corresponding to the data to be compressed by using the characteristics of different data to be compressed, so that the data compression method provided by the embodiment of the present application can set different resampling step sizes according to the characteristics of different data, so that the compressed data will not significantly reduce the data analysis quality.

[0077] In some embodiments, before determining whether to-be-processed data needs to be compressed, it is necessary to determine whether the to-be-processed data is missing. Based on the above embodiments, Figure 5 is a flow chart of a data compression method provided in an embodiment of the present application, such as Figure 5 As shown, in some embodiments, the data compression method may further include the following steps:

[0078] Step S501: perform integrity check on each of the data to be processed.

[0079] Here, the integrity check may be to check whether the curve of the data to be processed is continuous. If there are no values ​​at some time points in the curve of the data to be processed, it may be indicated that the data to be processed is missing.

[0080] Step S502: If it is determined that there is data missing in any of the data to be processed, the corresponding data to be processed is saved in a preset database.

[0081] Here, the preset database is used to store data that does not need to be compressed. When data is transmitted or presented, the compressed data is transmitted or presented together with the data in the preset database.

[0082] Step S503: Determine that there is no missing data in the data to be processed, and then perform the data analysis on the data to be processed.

[0083] In the embodiment of the present application, the data missing may be caused by system downtime, process restart or monitoring data loss from the perspective of operation and maintenance. Such data is generally not resampled and compressed, and the missing data is saved in a preset database. When there is no missing data to be processed, step S202 is executed to perform the data analysis on the data to be processed and determine at least one data to be compressed.

[0084] The data compression method provided in the embodiment of the present application avoids resampling of missing data and reduces resource consumption.

[0085] In some embodiments, each data has a static threshold, and if the threshold is exceeded, the data can be judged to be abnormal. Figure 6 is a flow chart of a data compression method provided in an embodiment of the present application, such as Figure 6 As shown, in some embodiments, the embodiment of the present application may also have the following steps:

[0086] Step S601: perform abnormal monitoring on each of the data to be processed.

[0087] Step S602: Determine whether the time series value in any data to be processed is greater than the static threshold.

[0088] In some embodiments, the data to be processed is time series data, including at least one time series value, and abnormal monitoring of the data to be processed refers to monitoring whether the value of each business data exceeds a preset static threshold.

[0089] Step S603: If it is determined that at least one time series value in any data to be processed is greater than the static threshold, the data to be processed is saved in a preset database.

[0090] Step S604: If it is determined that each time series value in any data to be processed is less than or equal to the static threshold, the data analysis is performed on the data to be processed.

[0091] In some embodiments, for the data to be processed whose time series value is greater than the static threshold, the data interval whose time series value is greater than the static threshold may not be resampled, or the data to be processed whose time series value is greater than the static threshold may be saved in a preset database, and the entire data to be processed is not resampled and compressed. If it is determined that all the time series values ​​in any data to be processed are less than or equal to the static threshold, step S202 is executed to perform the data analysis on the data to be processed to determine at least one data to be compressed.

[0092] In some embodiments, after the compressed data is resampled and compressed to form compressed data, the data can be saved, transmitted or visualized. Figure 7 is a flow chart of a data compression method provided in an embodiment of the present application, such as Figure 7 As shown, in some embodiments, the embodiment of the present application may also have the following steps:

[0093] Step S701: Save at least one compressed data into the preset database.

[0094] Step S702: Transmit at least one of the compressed data and the data to be processed in the preset database to a data processing device, so that the data processing device processes the compressed data.

[0095] In some embodiments, the data transmission device may be a data analysis device, through which a technician receives compressed data and uncompressed data to be processed and performs data analysis on all the data.

[0096] Step S703, aggregate the compressed data and the data to be processed in the preset database to form at least one data to be displayed; wherein the compressed data at least includes: a start timestamp, an end timestamp, a sampling step, and a timing value corresponding to each sampling step starting from the start timestamp.

[0097] Here, the data to be displayed includes all compressed data and data to be processed in a preset database.

[0098] Step S704: Perform display feature processing on each of the to-be-displayed data, so that each of the to-be-displayed data is displayed with a corresponding display feature.

[0099] In some embodiments, performing display feature processing on each data to be displayed refers to adding features to each data to be displayed, for example, the visualization color of each display data is different, and each data to be displayed is displayed in a different color during display.

[0100] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0101] Figure 8 is a flow chart of a data compression method provided in an embodiment of the present application, such as Figure 8 As shown, the data compression method provided in the embodiment of the present application can be implemented by the following steps:

[0102] Step S801, obtaining time series data.

[0103] Step S802: Determine whether there is missing data in the time series data.

[0104] In some embodiments, after obtaining the time series data curve that needs to be analyzed (i.e., the data to be processed), it is identified whether there is missing data in the curve. If there is missing data, from the operation and maintenance perspective, it may be a system crash, process restart, or monitoring data loss. Based on operation and maintenance experience, such curves are generally not resampled and compressed.

[0105] In some embodiments, when there is missing data in the time series data, step S804 is executed, and when there is no missing data in the time series data, step S803 is executed.

[0106] Step S803: Calculate the extreme value difference of the time series data.

[0107] Step S804: store the time series data in a database.

[0108] Step S805: Determine whether the extreme value difference is less than the extreme value threshold.

[0109] In some embodiments, some monitoring data exists in the form of counters, that is, the slope of each time point of the time series data is the same. If the time interval of the collected data is the same, the data compression method provided in the embodiment of the present application can still be used.

[0110] In some embodiments, if the extreme value difference is less than the extreme value threshold, step S806 is executed, and if the extreme value difference is greater than the extreme value threshold, step S807 is executed.

[0111] Step S806: resample the time series data according to the visualization query interval.

[0112] In some embodiments, if the difference is 0 or less than a certain threshold, the interval of the visualization query is used as the resampling step size for calculation, or the data of the start time point and the end time point are directly used.

[0113] Step S807: Calculate the standard deviation of the time series data.

[0114] Step S808: If the standard deviation of the time series data is greater than the standard deviation threshold, the time series data is stored in a database.

[0115] In some embodiments, the standard deviation threshold may be obtained by manual labeling (for example, by marking with tools such as observing a histogram) or by some anomaly detection algorithm, such as iForest.

[0116] Step S809: If the standard deviation of the time series data is less than the standard deviation threshold, determine the sampling step of each time series data according to the table lookup method, and resample according to the sampling step.

[0117] In some embodiments, the range of data fluctuation can be obtained by table lookup, and if it exceeds the standard deviation threshold, the original data is used. If it is lower than a certain standard deviation, the resampling step length is obtained by table lookup, and resampling is performed according to the sampling step length to obtain compressed data.

[0118] Step S810: Store, transmit and visualize the data in the database and the resampled data.

[0119] In some embodiments, the data transmission includes metadata such as the start time point, the end time point, and the resampling frequency, which will help restore the data to perform the required further processing, or perform the correct resampling or time series data processing in the restored data. The data in the database and the resampled data can also be sent to a visualization program for visualization.

[0120] In some embodiments, in many time series curves, the data fluctuation is large during a part of the time, while the data fluctuation is small during the rest of the time. Therefore, different data resampling steps may be used for different time windows. Fig. 9 is a flow chart of a data compression method provided in an embodiment of the present application, such as Fig. 9 As shown, the embodiment of the present application may also have the following steps:

[0121] Step S901: perform outlier analysis on time series data.

[0122] Step S902: Add fixed intervals before and after the outlier to form an outlier range interval.

[0123] In some embodiments, after the outlier range interval is determined, the time series data is divided into the outlier range interval and an interval other than each outlier range interval.

[0124] Step S903: Obtain data of the outlier range interval.

[0125] Step S904: Merge the data in the outlier range interval and the resampled time series data.

[0126] Step S905: Calculate the standard deviation of the intervals except for each outlier range interval.

[0127] Step S906: perform standardization on each interval and resample different intervals.

[0128] Step S907: Generate resampled time series data.

[0129] Step S908: Transmit, store and visualize the merged data.

[0130] In some embodiments, first, anomaly detection is performed on the time series data to obtain the time range in which the outliers occur. Typical anomaly detection methods include: horizontal fluctuation anomaly detection, adjacent data point difference anomaly detection, anomaly detection based on the residual of the predicted value and the actual value, and other anomaly detection algorithms, or the aggregation results of several anomaly detection algorithm results. Secondly, the intervals before and after the abnormal data are marked as abnormal intervals, and the data within the abnormal interval uses the original data without resampling calculations. For data outside the abnormal interval, the standard deviation of data in different time periods is calculated again, and the corresponding resampling step size is obtained by table lookup, and the data is resampled. Finally, the data in different intervals in a sequence are processed according to different resampling methods and returned to the relevant transmission, storage or application.

[0131] In some embodiments, a static threshold is set for the data used to monitor the alarm, and the data exceeding the static threshold is more important for engineers to analyze the anomaly. In an embodiment of the present application, the aforementioned compression method is performed on the data within the static threshold range. For indicators or data intervals exceeding the static threshold, the aforementioned compression method is not performed, and the original data is used.

[0132] The data compression method provided in the embodiment of the present application significantly reduces the amount of data transmitted and stored in time-series data, and retains relatively sensitive data intervals in the operation and maintenance business, and will not cause losses to the operation and maintenance business during the compression process.

[0133] Fig.10 is a schematic diagram of the structure of the data compression device provided in the embodiment of the present application, such as Fig. 9 As shown, the data compression device 100 includes:

[0134] The acquisition module 101 is used to acquire at least two data to be processed; the data analysis module 102 is used to determine at least one data to be compressed from the at least two data to be processed by performing data analysis on each of the data to be processed; the feature analysis module 103 is used to perform feature analysis on each of the data to be compressed and determine the sampling step of the service corresponding to each of the data to be compressed; the resampling module 104 is used to use the sampling step to resample the data corresponding to at least one of the data to be compressed to obtain at least one compressed data.

[0135] In some embodiments, the data analysis module 102 is further used to perform fluctuation analysis on each of the data to be processed to obtain a fluctuation difference corresponding to each of the data to be processed; classify the data to be processed according to the fluctuation difference to divide the data to be processed into first data to be compressed or second data to be compressed; and determine the first data to be compressed and the second data to be compressed as the data to be compressed.

[0136] In some embodiments, the data analysis module 102 is also used to determine the data to be processed as the first data to be compressed if the fluctuation difference is less than or equal to a preset fluctuation threshold; determine the discrete value of the data to be processed if the fluctuation difference is greater than the fluctuation threshold; and determine that the discrete value is less than or equal to the preset discrete threshold, then determine the data to be processed as the second data to be compressed.

[0137] In some embodiments, the feature analysis module 103 is further configured to perform sampling interval analysis on the first data to be compressed to obtain a visualized sampling interval of the first data to be compressed; and determine the duration corresponding to the visualized sampling interval as the sampling step length of the service corresponding to the first data to be compressed.

[0138] In some embodiments, the feature analysis module 103 is also used to obtain an information table, which includes at least: a resampling step corresponding to each business; performing feature analysis on each of the second data to be compressed to obtain the attributes of each of the second data to be compressed; determining the business corresponding to each of the second data to be compressed based on the attributes of each of the second data to be compressed; and determining the sampling step of the business corresponding to each of the second data to be compressed based on the information table and the business corresponding to each of the second data to be compressed.

[0139] In some embodiments, the data compression device 100 also includes: a detection module, used to perform integrity detection on each of the data to be processed; a first determination module, used to determine if there is data missing in any of the data to be processed, and then save the corresponding data to be processed into a preset database; a second determination module, used to determine if there is data missing in the data to be processed, and then perform the data analysis on the data to be processed.

[0140] In some embodiments, the data compression device 100 also includes: an abnormality monitoring module for performing abnormal monitoring on each of the data to be processed; a third determination module for determining whether the time series value in any of the data to be processed is greater than the static threshold; a first saving module for saving the data to be processed into a preset database if it is determined that at least one time series value in any of the data to be processed is greater than the static threshold; and an analysis module for performing the data analysis on the data to be processed if it is determined that each time series value in any of the data to be processed is less than or equal to the static threshold.

[0141] In some embodiments, the data compression device 100 also includes: a second saving module, used to save at least one of the compressed data into the preset database; a transmission module, used to transmit at least one of the compressed data and the data to be processed in the preset database to a data processing device, so that the data processing device processes the compressed data.

[0142] In some embodiments, the data compression device 100 also includes: an aggregation module, used to aggregate the compressed data and the data to be processed in the preset database to form at least one data to be displayed; wherein the compressed data at least includes: a start timestamp, an end timestamp, a sampling step and a timing value corresponding to each sampling step from the start timestamp; a display feature processing module, used to perform display feature processing on each of the data to be displayed, so that each of the data to be displayed is displayed with corresponding display features.

[0143] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0144] It should be noted that in the embodiment of the present application, if the above-mentioned data compression method is implemented in the form of a software function module and sold or used as an independent commodity, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be embodied in the form of a software commodity in essence or in other words, the part that contributes to the relevant technology can be embodied in the form of a software commodity, and the computer software commodity is stored in a storage medium, including a number of instructions for a terminal to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read Only Memory), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0145] Correspondingly, an embodiment of the present application provides a data compression device, Fig.11 is a schematic diagram of the structure of the data compression device provided in the embodiment of the present application, such as Fig.11 As shown, the data compression device 110 at least includes: a processor 111 and a computer-readable storage medium 112 configured to store executable instructions, wherein the processor 111 generally controls the overall operation of the data compression device. The computer-readable storage medium 112 is configured to store instructions and applications executable by the processor 111, and can also cache data to be processed or processed by the processor 111 and each module in the data compression device 110, which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0146] The present application embodiment provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided by the present application embodiment, for example, Figure 2 The method shown.

[0147] In some embodiments, the storage medium can be a computer-readable storage medium, for example, a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a compact disk read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above memories.

[0148] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0149] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). As an example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network.

[0150] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

[0151] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0152] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, an element defined by the statement "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed.

[0153] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data compression method, It is characterized in that include: Obtain at least two pieces of data to be processed; Performing fluctuation analysis on each of the data to be processed to obtain a fluctuation difference corresponding to each of the data to be processed; If the fluctuation difference is less than or equal to a preset fluctuation threshold, determining the data to be processed as first data to be compressed; If the fluctuation difference is greater than the fluctuation threshold, determining a discrete value of the data to be processed; If it is determined that the discrete value is less than or equal to a preset discrete threshold, the data to be processed is determined as second data to be compressed; Performing feature analysis on each data to be compressed, and determining a sampling step length of a service corresponding to each data to be compressed; the data to be compressed includes the first data to be compressed and the second data to be compressed; The sampling step size is used to resample data corresponding to at least one data to be compressed, so as to obtain at least one compressed data; wherein different sampling step sizes are used to resample different data to be compressed.

2. The method according to claim 1, It is characterized in that The performing feature analysis on each of the data to be compressed to determine the sampling step length of the service corresponding to each of the data to be compressed includes: Performing sampling interval analysis on the first data to be compressed to obtain a visualized sampling interval of the first data to be compressed; The duration corresponding to the visualization sampling interval is determined as the sampling step length of the service corresponding to the first data to be compressed.

3. The method according to claim 1, It is characterized in that The performing feature analysis on each of the data to be compressed to determine the sampling step length of the service corresponding to each of the data to be compressed includes: Obtaining an information table, the information table at least including: a resampling step length corresponding to each service; Performing feature analysis on each of the second data to be compressed, and correspondingly obtaining an attribute of each of the second data to be compressed; According to the attribute of each of the second data to be compressed, correspondingly determining the service corresponding to each of the second data to be compressed; A sampling step length of the service corresponding to each second data to be compressed is determined according to the information table and the service corresponding to each second data to be compressed.

4. The method according to claim 1, It is characterized in that The method further comprises: Performing integrity check on each of the data to be processed; If it is determined that there is data missing in any of the data to be processed, the corresponding data to be processed is saved in a preset database; If it is determined that there is no missing data in the data to be processed, data analysis is performed on the data to be processed.

5. The method according to claim 1, It is characterized in that Each of the data to be processed has a static threshold; The data to be processed includes: at least one time series value; The method further comprises: Performing abnormal monitoring on each of the data to be processed; Determine whether the time series value in any data to be processed is greater than the static threshold; If it is determined that at least one time series value in any of the data to be processed is greater than the static threshold, the data to be processed is saved in a preset database; If it is determined that each time series value in any data to be processed is less than or equal to the static threshold, data analysis is performed on the data to be processed.

6. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: saving at least one of the compressed data into a preset database; or, At least one of the compressed data and the data to be processed in the preset database is transmitted to a data processing device, so that the data processing device processes the compressed data.

7. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: Aggregating the compressed data and the data to be processed in the preset database to form at least one data to be displayed; wherein the compressed data at least includes: a start timestamp, an end timestamp, a sampling step, and a time series value corresponding to each sampling step from the start timestamp; Display feature processing is performed on each of the to-be-displayed data respectively, so that each of the to-be-displayed data is displayed with a corresponding display feature.

8. A data compression device, It is characterized in that include: An acquisition module, used for acquiring at least two pieces of data to be processed; A data analysis module, used for performing fluctuation analysis on each of the data to be processed to obtain a fluctuation difference corresponding to each of the data to be processed; If the fluctuation difference is less than or equal to a preset fluctuation threshold, determining the data to be processed as first data to be compressed; If the fluctuation difference is greater than the fluctuation threshold, determining the discrete value of the data to be processed; if the discrete value is less than or equal to a preset discrete threshold, determining the data to be processed as second data to be compressed; A feature analysis module, used to perform feature analysis on each data to be compressed, and determine a sampling step length of a service corresponding to each data to be compressed; the data to be compressed includes the first data to be compressed and the second data to be compressed; The resampling module is used to use the sampling step size to resample the data corresponding to at least one of the services to be compressed to obtain at least one compressed data; wherein different sampling step sizes are used to resample different data to be compressed.