Time series data compression processing method and device, storage medium and electronic equipment

Abstract

The invention provides a time series data compression processing method and device, a storage medium and electronic equipment, and belongs to the technical field of data processing. The method comprises the steps of obtaining original time series data to be compressed, wherein the original time series data comprises a numerical value list of original data and a time list used for reflecting time information of numerical values of the original data; key numerical value points are sampled from the numerical value list; acquiring a time point corresponding to the key numerical point from the time list; forming data pairs by using the sampled key numerical points and time points; determining a time function matched with the data pair; and storing the data pair and the time function, so that the confidence coefficient of fitting data fitted according to the time function and the data pair relative to the original time sequence data is not lower than a confidence coefficient threshold value. According to the invention, the occupation of the time sequence data on the storage space can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, storage medium and electronic device for compressing time series data. Background Art

[0002] Time series data (abbreviated as "time series data") is data formed in chronological order. Each piece of data contains a numerical value and the corresponding time. It is widely present in various fields, such as the Internet of Things, industrial control, financial markets, meteorological monitoring, data centers, etc. In some fields, such as data centers, in order to achieve safe production, a large number of sensors are installed inside the data center. The data collected by these sensors will be continuously reported, which requires a large amount of data storage space.

[0003] Take the data collection of the environmental monitoring system of the data center room as an example. The collected data includes HVAC data, power data, etc., such as time series data on the air conditioning controller, time series data on the cabinet, temperature and humidity data in the room, etc. A data center generally has more than 300,000 collection points. Calculated at a sampling rate of once every 5 seconds, the data center will collect at least 5184,000,000 data records in one day of operation. Due to the large amount of time series data generated, effective management is required during long-term storage to avoid data redundancy and waste of storage space.

[0004] If all the data is simply saved, it will take up a lot of physical storage space and be difficult to retrieve and use. Most of these time series data usually reflect the normal data when the equipment or environment is running normally, and there is a lot of redundancy. From a business perspective, the relevant operation and maintenance personnel are not concerned with the massive amount of ordinary data, but hope to see the trends shown by the data and the time series data (abnormal data) detected when there are abnormalities in the production process.

[0005] For the storage of this type of time series data, if simple full storage or lossless compression storage is used, a large amount of invalid data (data of no concern) will be stored, which will result in high storage costs and waste. On the other hand, although the current common lossy data compression storage, such as rotational internal compression method, steady-state threshold method, linear extrapolation method, etc., can reduce the data size to a certain extent, it cannot distinguish between invalid data and abnormal data reflecting abnormal production status (or so-called "key valid data"), nor can it perform targeted processing on abnormal data, resulting in incompatibility with some characteristics of the monitoring system, which may cause the loss of key valid data.

[0006] Therefore, in the monitoring and management of time series data, how to combine the specific characteristics of the data and business needs to formulate a suitable compression strategy to efficiently compress the time series data for long-term storage in order to maximize the business effectiveness of the data is still a challenging task. Summary of the invention

[0007] Based on this, the purpose of the present application is to provide a compression processing method, device, storage medium and electronic device for time series data, so as to retain the key information of business data as much as possible while meeting the requirements for time series data compression.

[0008] In a first aspect, the present application provides a method for compressing time series data, the method comprising:

[0009] Acquire original time series data to be compressed, the original time series data comprising a value list of the original data and a time list for reflecting time information of the value of the original data;

[0010] Sampling key value points from the value list;

[0011] Obtaining the time point corresponding to the key value point from the time list;

[0012] The key numerical points and time points of the sampling are formed into data pairs;

[0013] determining a time function matching the data pair;

[0014] The data pair and the time function are stored so that the confidence of the fitted data fitted according to the time function and the data pair relative to the original time series data is not less than a confidence threshold.

[0015] Optionally, the sampling of key numerical points from the numerical list includes: extracting multiple first key points that meet configuration rules from the numerical list; obtaining corresponding multiple second key points from the numerical list by quartile statistics; and using one or more of the first key points and the second key points as the key numerical points.

[0016] Optionally, the configuration rule includes one or more of the following rules:

[0017] 1) The initial value point, maximum value point and minimum value point in the numerical list;

[0018] 2) the alarm value points for triggering an alarm and the alarm recovery value points for triggering an alarm recovery in the value list;

[0019] 3) Reference value points captured from the value list by a preset lossy compression algorithm;

[0020] 4) If the absolute value of the difference between the currently extracted numerical point and the first extracted key point exceeds a preset difference threshold, the currently extracted numerical point is used as the first key point;

[0021] 5) If the time interval between the currently extracted numerical point and the first extracted key point exceeds a preset time difference, the currently extracted numerical point is used as the first key point.

[0022] Optionally, extracting multiple first key points that meet the configuration rules from the value list includes: taking the initial value point in the value list as the extracted first key point; extracting the unextracted value points from the value list in sequence, and comparing whether the currently extracted value point meets the configuration rules with the last extracted first key point; if so, taking the currently extracted value point as the first key point, updating the last extracted first key point, and returning to execute the extraction of the unextracted value points from the value list in sequence.

[0023] Optionally, the obtaining the corresponding plurality of second key points from the value list by using the quartile statistical method includes:

[0024] Calculate the corresponding upper quartile Q3 and lower quartile Q1 from the numerical list;

[0025] Based on the upper quartile Q3 and the lower quartile Q1, the corresponding upper limit value P1 and lower limit value P2 are calculated, P1 = Q3 + k × (Q3 - Q1), P2 = Q1 - k × (Q3 - Q1), k is the IQR coefficient;

[0026] One or more of the numerical points in the numerical list that are greater than the upper limit value P1 and less than the lower limit value P2 are used as the second key point.

[0027] Optionally, taking one or more of the first key point and the second key point as the key numerical point includes: taking the union of a first set formed by the first key point and a second set formed by the second key point; detecting whether the number of data in the union exceeds a first quantity threshold, and if so, sub-sampling the data in the union, and taking the sub-sampled data as the key numerical points, so that the number of the sub-sampled key numerical points does not exceed the first quantity threshold.

[0028] Optionally, the configuration rules include multiple ones; the sub-sampling of the data in the union and using the sub-sampled data as the key value points includes:

[0029] An intersection is calculated for a first set formed by the first key points and a second set formed by the second key points, and key points in the intersection are used as key numerical points obtained by secondary sampling.

[0030] Optionally, the sub-sampling of the data in the union and using the sub-sampled data as the key numerical point includes: using the first key point extracted by at least two configuration rules as the key numerical point obtained by sub-sampling.

[0031] Optionally, the sub-sampling the data in the union and using the sub-sampled data as the key value points includes:

[0032] Determining the number of data to be deleted according to the first number threshold and the number of data in the union;

[0033] Calculate the time difference and value difference between adjacent data in the union;

[0034] The key value points of the deletion quantity are deleted based on the time difference and the value difference, and the retained data are used as the key value points, wherein the smaller the time difference and / or the smaller the value difference, the higher the priority of being deleted.

[0035] Optionally, after sampling the key numerical points from the numerical list, it also includes: detecting whether the number of data in the set is less than a second quantity threshold, and if so, returning to execute the method of obtaining the corresponding multiple second key points from the numerical list through the quartile statistics method, so as to increase the number of second key points obtained.

[0036] Optionally, determining a time function that matches the data pair includes: obtaining a data type and / or data attribute to which a key numerical point in the data pair belongs; and determining a time function that matches the data type and / or data attribute.

[0037] Optionally, determining a time function that matches the data pair includes:

[0038] Calling multiple candidate time functions to fit the data pair to form corresponding candidate fitting data;

[0039] Calculating the confidence between each candidate fitting data and the original time series data;

[0040] The candidate time function with the highest confidence is taken as the time function matching the data pair.

[0041] Optionally, after storing the data pair and the time function, the method further includes: receiving a display instruction for the data pair; and calling the time function to fit the data pair according to the display instruction to form and display the fitting data.

[0042] Optionally, before obtaining the original time series data to be compressed, the method further includes:

[0043] Acquire collected data collected in real time by at least one sensor in the data center to form a collected data set;

[0044] Segmenting the collected data set according to a preset time length;

[0045] The segmented collected data is used as the original time series data; the collected data includes state quantity data and analog quantity data.

[0046] In a second aspect of the present application, a time series data compression processing device is provided, the device comprising:

[0047] A time series data acquisition module, used to acquire original time series data to be compressed, wherein the original time series data includes a value list of the original data and a time list for reflecting time information of the value of the original data;

[0048] A key data sampling module is used to sample key value points from the value list; obtain the time points corresponding to the key value points from the time list; form data pairs with the sampled key value points and time points; and determine a time function that matches the data pair;

[0049] A data storage module is used to store the data pair and the time function so that the confidence of the fitted data fitted according to the time function and the data pair relative to the original time series data is not less than a confidence threshold.

[0050] In a third aspect of the present application, a computer-readable storage medium is provided, on which executable instructions are stored. When the executable instructions are executed by a processor, the processor executes the method described in any embodiment of the present application.

[0051] In a fourth aspect of the present application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method described in any embodiment of the present application.

[0052] The compression processing method, device, storage medium and electronic device of the time series data in the present application can effectively reduce the storage space occupied by the time series data by extracting the key points in the original time series data, and further setting a time function for fitting the original time series data according to the key points, and storing the key points and the time function together. At the same time, since the key numerical points and the corresponding time points, as well as the time function that describes the law of data change are retained, the main characteristics and trends of the original time series data can be accurately restored when needed, which is of great significance for the storage of data such as data center monitoring data, data collected by Internet of Things devices, and monitoring data of industrial control systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope of the present application.

[0054] Figure 1 1 is a flow chart of a method for compressing time series data in one embodiment;

[0055] Figure 2 A schematic diagram of a process of extracting a plurality of first key points that meet the configuration rule from a value list in one embodiment;

[0056] Figure 3 is a schematic diagram of time series data, key points and time functions in the first embodiment;

[0057] Figure 4 is a schematic diagram of time series data, key points and time functions in the second embodiment;

[0058] Figure 5 is a schematic diagram of time series data, key points and time functions in the third embodiment;

[0059] Figure 6 is a schematic diagram of time series data, key points and time functions in the fourth embodiment;

[0060] Figure 7 A schematic diagram of a process of performing sub-sampling on the data in a union and using the sub-sampled data as key value points in an embodiment;

[0061] Figure 8 is a schematic diagram of the structure of a device for compressing and processing time series data in one embodiment;

[0062] Fig. 9 is a schematic diagram of the structure of a device for compressing and processing time series data in another embodiment;

[0063] Fig.10FIG. 4 is a structural block diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] All terms (including technical and scientific terms) used in this application have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used here should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0066] For example, the terms "first", "second", etc. used in this application can be used to describe various elements in this article, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, the first key point can be referred to as the second key point, and similarly, the second key point can be referred to as the first key point. Both the first key point and the second key point are key points, but they are not the same key point.

[0067] For another example, the terms “include”, “comprising”, etc. used in this application indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0068] In one embodiment, a method for compressing time series data is provided. Figure 1 As shown, the method includes:

[0069] Step 102: Obtain original time series data to be compressed.

[0070] In this embodiment, the original time series data may be data collected by relevant sensors, and the original time series data includes both the value of the collection object and the collection time of the corresponding value.

[0071] Specifically, the original time series data may be data collected within a certain time period, such as 1 hour, 5 hours, 12 hours, 24 hours, one week, one month, or any other suitable time period. Alternatively, the original time series data may be uncompressed data accumulated to a certain amount of data, such as any suitable data amount value, such as 1TB, 10TB, or any other suitable value.

[0072] The original time series data may be data collected in the above-mentioned Internet of Things, industrial control, financial market, weather monitoring, data center and other dry operation scenarios. For example, it may be data collected by relevant sensors in the data center at a certain collection frequency. For example, it may be HVAC data, power data, temperature and humidity data, etc. collected by one or more sensors deployed in the computer room of the data center, and collected at a frequency of once every 5 seconds, and the data collected within 24 hours is used as the original time series data to be compressed.

[0073] The collected time series data can be stored in real time, and the system can periodically extract a certain amount or a certain time period of data from the stored time series data as the original time series data to be compressed.

[0074] The original time series data includes a numerical list of the original data and a time list used to reflect the time information of the numerical value of the original data. That is, each time series data has a corresponding numerical value and a collection time corresponding to the data. The electronic device can form at least two lists of the original time series data according to the two dimensions of numerical value and time, one column representing the corresponding numerical value and the other column representing the time corresponding to the numerical value. For example, a time series data representing the ambient temperature collected by a temperature sensor has a temperature value of 35.0 (in ° C), and the corresponding collection time is 09:53:01 on August 28, 2024.

[0075] In one embodiment, before step 102, it also includes: acquiring collected data collected in real time by at least one sensor in the data center to form a collected data set; segmenting the collected data set according to a preset time length; using the segmented collected data as original time series data; the collected data includes data types such as state quantity data and analog quantity data, and may also include one or more data attributes such as data collection frequency, data source, data physical properties or data units.

[0076] In this embodiment, each sensor in the data center can collect relevant monitoring data according to the corresponding set collection frequency, and report and store the collected value points and the corresponding collection time to form a corresponding collection data set.

[0077] Among them, the data types and / or data attributes of the data collected by each sensor are not necessarily the same, and the data types may include state quantity data and analog quantity data. The data type represents the type divided according to the nature of the data, and the data type may include state quantity data, analog quantity data and other types. Among them, analog quantity data is used to reflect the quantitative characteristics of the data, such as temperature, length, area, volume, weight, etc., and the value in the time dimension is usually a gradual and continuous change; state quantity data is used to reflect the state or qualitative characteristics of the corresponding data, such as the switch state ("on" or "off"), etc., and the state quantity data usually has only a limited number of states, and the value in the time dimension is usually a jump change.

[0078] Data attributes refer to the characteristics or descriptive information of the data in the numerical list. Data attributes can reflect one or more of the data collection frequency, data source, data physical meaning (such as temperature, pressure, flow, etc.), data unit, etc.

[0079] Specifically, data can be stored in units of sensors, and each sensor forms a corresponding collection data set. For the collection data set formed by each sensor, corresponding data division can be performed to form data segments, and the data segments are the corresponding original time series data. Among them, the division of the collection data set can be divided in units of a preset time length, and the preset time length can be a fixed time length, or a custom time length set according to the amount of accumulated uncompressed time series data. For example, the preset time length can be 24 hours. When the span of the collection time of the data in the collection data set collected by the sensor reaches or exceeds 24 hours, the data is segmented in units of 24 hours, and the time span of the data in the original time series data formed by each segment is the 24 hours. It can be understood that the division time lengths corresponding to different sensors are not necessarily the same.

[0080] By segmenting data by time length, electronic devices can periodically compress collected data. For example, they can choose to perform data compression when the system is relatively idle, or when the stored collected data set reaches a certain amount of data, they can compress it, which can increase the flexibility of data compression time.

[0081] Optionally, the data can be divided according to a preset data volume. For example, if the data volume is 1TB, it means that whenever the accumulated data volume in the collected data set uploaded by the sensor reaches or exceeds 1TB, it is divided according to the data volume of 1TB, so that the data volume of each divided original time series data is 1TB.

[0082] Step 104: sampling key value points from the value list.

[0083] In this embodiment, key value points are important value points that are identified and extracted in the time series data according to specific business rules, data characteristics or analysis requirements. These points usually represent the beginning or end of a significant change, anomaly, trend in the data, or are values ​​that are particularly meaningful for understanding and analyzing the overall data. In the processing and analysis of time series data, key value points can help compress and store data more efficiently while retaining the main features and trends of the data.

[0084] The electronic device is pre-configured with recognition rules for key value points according to one or more of corresponding business rules, data characteristics, and analysis requirements, and recognizes key value points according to the recognition rules, thereby extracting the recognized key value points.

[0085] For example, the initial value point, end point, minimum value point, maximum value point, warning value point, data mutation point, inflection point, extreme value point, etc. in the value list can be identified to reflect the important changes or trends of the data.

[0086] In addition, key numerical points can be identified and extracted according to relevant mathematical statistical algorithms or key point recognition models, and the identified numerical points can be used as key numerical points. The mathematical statistical algorithm can be, for example, quartile statistics, standard deviation, percentile, box plot and other statistical algorithms; the recognition model can include clustering analysis models such as K-means, DBSCAN, or related machine learning models such as support vector machines, random forests, neural networks, etc.

[0087] Step 106, obtaining the time point corresponding to the key value point from the time list.

[0088] In this embodiment, after obtaining the corresponding key numerical point, the time information corresponding to the key numerical point can be obtained to obtain the corresponding time point. Among them, the time point can be a specific time, such as the above-mentioned 09:53:01 on August 28, 2024; it can also be a time difference or time offset relative to a certain time, such as the time offset relative to the time of the first data in the original time series data. Based on the time offset and the time of the first data, the time of the corresponding numerical point can be determined. For example, if the time offset is 5 (in seconds), it means 5 seconds after the time of the first data.

[0089] Step 108, forming data pairs from the sampled key numerical points and time points.

[0090] In this embodiment, for the determined sampled time series data, the corresponding numerical points and time points can be combined to form corresponding data pairs for storage. The data pairs can be expressed as [time point, numerical point]. By forming data pairs, the numerical value and the corresponding time of the sampled data can be known. When there are N key numerical points, the corresponding data pairs include N groups.

[0091] For the determined data pairs, a corresponding buffer library may be set. For example, an influxdb buffer library may be used to store the data pairs in the buffer library.

[0092] Step 110, determining a time function that matches the data pair.

[0093] In this embodiment, a corresponding time function is further set for the sampled data pairs. The time function is a function model used to represent the law of time series data changing over time. Based on the time function, the main features and trends of the time series data can be simulated.

[0094] Among them, the time function can be of various types, for example, it can include a combination of one or more of the step function, the broken line function, the trigonometric function, the exponential function, the logarithmic function, the polynomial function, etc. Different time functions are suitable for corresponding to different data pairs. For example, for the data pair of the above-mentioned state quantity type, the time function matching it can be a step function; for a data pair of a certain analog quantity type, the time function matching it can be a broken line function.

[0095] In one embodiment, step 110 includes: obtaining the data type and / or data attribute to which the key value point in the data pair belongs; and determining a time function that matches the data type and / or data attribute.

[0096] Time series data has corresponding data types and / or data attributes. When the corresponding original time series data is obtained, the data types and / or data attributes corresponding to the key value points in the data pair can be learned.

[0097] The electronic device may pre-establish a correspondence between different data types and / or data attributes and time functions. After the corresponding correspondence is established, the corresponding time function may be queried according to the data type and / or data attribute to which the data pair belongs, and the queried time function may be used as the matching time function.

[0098] In another embodiment, step 110 includes: calling multiple candidate time functions to fit the data pair to form corresponding candidate fitting data; calculating the confidence between each candidate fitting data and the original time series data; and taking the candidate time function with the highest confidence as the time function that matches the data pair.

[0099] It is also possible to directly select one or more time functions from the multiple time functions that are set to fit the data pair to form fitting data. The fitting data reflects the degree of restoration of the original time series data. The electronic device can determine a time function as a matching time function based on the degree of restoration of the original time series data by each time function. For example, the time function with the highest degree of restoration can be selected as the matching time function. The data pairs involved in the fitting can be all or part of the extracted data pairs. By selecting part of the data pairs for fitting, the workload of fitting can be reduced and the fitting efficiency can be improved.

[0100] for example Figures 3 to 6 As shown in the figure, the horizontal axis represents the corresponding time offset, the vertical axis represents the value of the corresponding data point, the gray dots represent the data points in the value list, the red dots represent the extracted key value points, and the red dotted line represents the fitting curve fitted by the corresponding time function based on the key value points. Figure 3 There are 4 key numerical points involved in the fitting, and the time function corresponding to the fitting curve is a broken line function; Figure 4 There are 8 key numerical points involved in the fitting, and the time function corresponding to the fitting curve is a step function; Figure 5 There are 3 key numerical points involved in the fitting, and the time function corresponding to the fitting curve is a broken line function; Figure 6 There are 3 key numerical points involved in the fitting, and the time function corresponding to the fitting curve is a step function.

[0101] The degree of restoration can be reflected by the confidence level. The higher the confidence level, the higher the degree of restoration. That is, the closer the fitted data is to the original time series data, the smaller the error between the fitted data and the original time series data.

[0102] In the process of matching the time function, the time function to be selected (i.e., the candidate time function) is called to fit the data pair to form fitting data (i.e., the candidate fitting data), and the confidence of the fitting data and the original time series data is calculated. When the confidence is not lower than the preset confidence threshold, the time function to be selected is used as the matching time function. Otherwise, other functions are continuously replaced for confidence verification until the selected confidence reaches the corresponding confidence threshold.

[0103] Optionally, the confidence threshold may be a preset fixed value, such as uniformly setting the confidence threshold to 95%. In addition, it may also be a value adaptively determined according to different data types and / or data attributes, that is, the confidence thresholds corresponding to the original time series data of different data types and / or data attributes are not necessarily the same. For example, some data have higher requirements for confidence, and some data have lower requirements for confidence. The electronic device can set different confidence thresholds according to relevant requirements.

[0104] Step 112, storing the data pairs and the time function, so that the confidence of the fitted data fitted according to the time function and the data pairs relative to the original time series data is not less than the confidence threshold.

[0105] The confidence level and confidence level threshold in this embodiment have the same meaning as the confidence level and confidence level threshold in the above text, wherein the confidence level is used to measure the degree of restoration or similarity between the fitted data and the corresponding original time series data, and the confidence level threshold is used to measure whether the fitted data meets the critical value required for restoring the original time series data.

[0106] After determining the data pairs and the corresponding time functions, while saving the sampled data pairs, the matched time functions are also saved, so that when the data is subsequently reviewed in detail, in addition to being able to view the corresponding data pairs, the time functions and data pairs can also be fitted, so that the fitted data has a higher confidence level than the original time series data.

[0107] Specifically, a corresponding relational database may be selected to store the data pair and the time function for a long term, wherein the corresponding data pair may be composed into a linked list and converted into binary together with the corresponding time function and stored in the relational database.

[0108] The compression processing method of time series data in the present application can effectively reduce the storage space occupied by time series data by extracting key numerical points in the original time series data, and further setting a time function for fitting the original time series data according to the key numerical points, and storing the key numerical points and the time function together. At the same time, since the key numerical points and the corresponding time points, as well as the time function that describes the law of data change, are retained, the main features and trends of the original time series data can be accurately restored when needed, which has important practical applications for data storage such as data center monitoring data, data collected by Internet of Things devices, and monitoring data of industrial control systems.

[0109] In one embodiment, step 104 includes: extracting multiple first key points that meet the configuration rules from the numerical list; obtaining multiple corresponding second key points from the numerical list by quartile statistics; and using one or more of the first key points and the second key points as key numerical points.

[0110] In this embodiment, a variety of configuration rules are pre-set, and key points can be screened according to the configuration rules, and the numerical points in the numerical list that meet the configuration rules are used as the first key points. Among them, the configuration rules can be rules configured for the above-mentioned business rules, data characteristics, and analysis requirements. There can be multiple configuration rules, and the configuration rules corresponding to the data reported by different sensors are not necessarily the same. The electronic device determines the configuration rules that match the data type and data attributes corresponding to the numerical list, and screens the first key point that meets the corresponding configuration rules.

[0111] The corresponding configuration rules for different data types and / or data attribute values ​​are not necessarily the same. For example, for the value of the temperature analog quantity, the corresponding configuration rules include taking the value that reflects the temperature change rate exceeding the change rate threshold as the first key point, and taking the value that exceeds the temperature threshold as the first key point. For the value of the switch state quantity, the corresponding configuration rule is to take the value that changes for the first time as the first key point.

[0112] In addition to extracting key points based on configuration rules, the quartile statistical method is also used to obtain key points (recorded as "second key points"). Specifically, all values ​​in the value list are sorted from small to large, and the first quartile (Q1, also called the "lower quartile"), the second quartile (median, Q2) and the third quartile (Q3, also called the "upper quartile") are calculated. Then, according to the IQR (Interquartile Range, i.e. Q3-Q1), the points that belong to the corresponding outliers or drastic changes in the value list are determined, and these points are used as the second key points. For example, we can regard numerical points below Q1-1.5IQR or above Q3+1.5IQR as outliers and use them as the second key points.

[0113] After obtaining the first key point and the second key point, one or more of them are extracted as key numerical points according to actual needs. For example, both the first key point and the second key point can be used as key numerical points, or some of the numerical points can be used as key numerical points. It is understandable that there may be repeated points in the first key point and the second key point. The first key point and the second key point can be intersected or unioned, and the points formed can be used as key numerical points.

[0114] In one embodiment, the electronic device can set the number of key numerical points that need to be stored, and the number can be a corresponding specific value, or a numerical range. The size of the number can be determined according to one or more factors such as the size of the storage space in the data and the data type and / or data attribute of the original time series data. For example, it can be determined that the number that needs to be stored can be a number interval between a first number threshold and a second number threshold, and the first number threshold is greater than the second number threshold. The electronic device extracts a corresponding appropriate number of numerical points from the first key point and the second key point as the key numerical point.

[0115] In one embodiment, the configuration rules include one or more of the following rules: 1) initial value points, maximum value points, and minimum value points in the numerical list; 2) alarm value points for triggering alarms and alarm recovery value points for triggering alarm recovery in the numerical list; 3) reference numerical points captured from the numerical list by a preset lossy compression algorithm; 4) if the absolute value of the difference between the currently extracted numerical point and the first extracted key point exceeds a preset difference threshold, the currently extracted numerical point is used as the first key point; 5) if the time interval between the currently extracted numerical point and the first extracted key point exceeds a preset time difference, the currently extracted numerical point is used as the first key point.

[0116] In this embodiment, the electronic device can traverse the value list and use the value point at the initial position, the last value point, the maximum value point, and the minimum value point in the list as the corresponding first key point.

[0117] Different data types and / or data attributes are correspondingly set with alarm thresholds for triggering alarms and triggering alarm recovery. The electronic device can query the value point corresponding to the alarm threshold in the value list, and use the identified alarm value point and alarm recovery value point as the first key point. Optionally, the value corresponding to the alarm time or the alarm recovery time generated by the system can be used as the alarm value point, and the value corresponding to the alarm recovery time can be used as the alarm recovery value point.

[0118] The electronic device is further provided with one or more lossy compression algorithms, such as one or more compression algorithms such as rotational internal compression method, steady-state threshold method, linear extrapolation method, etc., and a suitable compression rate is set to compress the data in the numerical list, so as to obtain a number of numerical points that are suitable for the compression rate. The numerical points selected by each compression algorithm during compression are detected, and one or more of the selected numerical points are used as the first key point. For example, the number of times each numerical value in the numerical list is captured by the lossy compression algorithm can be counted, and the numerical point whose number of captures exceeds the corresponding number threshold can be used as the first key point, or the number to be selected can be set, and the numerical points that are at the front of the required number of sorting can be selected as the first key point.

[0119] In addition, when the electronic device extracts the first key point, it can also extract the value points in the value list in sequence, and for the currently extracted value point, calculate the time interval between the time point corresponding to the value point and the time point between the most recently extracted first key point. When the time interval is greater than the preset time difference, it means that no corresponding value point has been extracted in a time period within the time difference, and the currently extracted value point can be set as the first key point, which is helpful to capture the long-term trend change point in the data. The time interval can be calculated based on information such as the acquisition frequency of the original time series data corresponding to the value point.

[0120] Similarly, the difference in numerical value between the currently extracted numerical point and the most recently extracted first key point can also be calculated. When the absolute value of the difference exceeds the preset difference threshold, the currently extracted numerical point can also be used as the first key point, which helps to capture the points of drastic changes in the data.

[0121] By setting multiple configuration rules, key value points can be sampled from the value list more comprehensively and accurately. These key value points include both special values ​​and alarm-related value points of the data, reference value points extracted by lossy compression algorithms, and key points based on value differences and time interval changes, so as to more comprehensively reflect the main characteristics and trends of the original value list, and provide strong support for subsequent data compression storage and data analysis.

[0122] In one embodiment, multiple first key points that meet the configuration rules are extracted from a value list, including: step 1: taking the initial value point in the value list as the extracted first key point; step 2: extracting the unextracted value points from the value list in order, and comparing whether the currently extracted value point meets the configuration rules with the last extracted first key point; step 3: if so, taking the currently extracted value point as the new extracted first key point, and storing the last extracted first key point; step 4: returning to execute steps 2 and 3 until the value points in the value list are extracted in order.

[0123] In this embodiment, a sampling result list v1 for storing the first key point may be pre-set. By performing a cyclic query on the value points in the value list, each value point is detected to see whether it meets the corresponding configuration rule, and the value points that meet the corresponding configuration rule are stored in the sampling result list v1. Specifically, the value point currently extracted in the value list may be detected to see whether it meets any of the above five configuration rules. If it meets, the currently extracted value point is stored in the sampling result list v1. If it does not meet, the next value point is determined until all the value points in the value list are detected.

[0124] For example, for a certain numerical point, the numerical difference and time interval between the numerical point and the first key point most recently stored in the sampling result list v1 can be detected. If the time interval exceeds the corresponding time difference, or the absolute value of the numerical difference exceeds the preset difference threshold, the numerical point is stored in the sampling result list v1 and used as the latest first key point in the sampling result list v1. Alternatively, if the numerical point meets the corresponding alarm change condition, the numerical point is also added to the sampling result list v1.

[0125] In one embodiment, Figure 2 As shown, the corresponding multiple second key points are obtained from the value list by using the quartile statistical method, including:

[0126] Step 202, calculating the corresponding upper quartile Q3 and lower quartile Q1 from the value list.

[0127] In this embodiment, the numerical points in the numerical list can be sorted in ascending order of numerical value, and the numerical point at the 1 / 4 position is selected as the upper quartile Q3; the numerical point at the 3 / 4 position is selected as the lower quartile Q1.

[0128] Step 204 , calculating the corresponding upper limit value P1 and lower limit value P2 based on the upper quartile Q3 and the lower quartile Q1 .

[0129] In this embodiment, P1 = Q3 + k × (Q3-Q1), P2 = Q1-k × (Q3-Q1), where k is the IQR coefficient (IQRrate), and the value of k can be a preset fixed value, or a value set according to different data types and / or data attributes. For example, the value of k can be set to any suitable value such as 1.5 or 1.0.

[0130] Step 206: One or more of the value points in the value list that are greater than the upper limit value P1 and less than the lower limit value P2 are used as second key points.

[0131] After the corresponding P1 and P2 are calculated, one or more of the numerical points in the numerical list that are not in the interval [P2, P1] can be extracted as the second key point.

[0132] The number of extracted points can be determined according to the set k value. By setting a suitable k value, a suitable number of second key points can be obtained.

[0133] In one embodiment, one or more of the first key point and the second key point are used as key numerical points, including: taking the union of a first set formed by the first key point and a second set formed by the second key point; detecting whether the number of data in the union exceeds a first quantity threshold, and if so, sub-sampling the data in the union, and using the sub-sampled data as key numerical points, so that the number of sub-sampled key numerical points does not exceed the first quantity threshold.

[0134] In this embodiment, by summarizing the first key point and the second key point, comparing whether the number of summarized key points (i.e., the number of data in the union) meets the expected number of key value points, when the summarized number exceeds the first number threshold, it means that the number of key points obtained is large, and further sampling is still required so that the number of extracted key points does not exceed the first number threshold. Specifically, key points of the first number threshold can be extracted as key value points.

[0135] Optionally, the extraction method may be random sampling, stratified sampling or other suitable sampling methods. Specifically, for example, the numerical points that are extracted as key points (including the first key point and the second key point) more often in the union data may be extracted as key numerical points.

[0136] In one embodiment, the intersection of the first set formed by the first key points and the second set formed by the second key points is calculated, and the key points in the intersection are used as the key value points obtained by subsampling.

[0137] In one embodiment, the first key point extracted by at least two configuration rules is used as the key value point obtained by subsampling.

[0138] Optionally, when the same numerical point is used as both the first key point and the second key point, the numerical point can be directly used as the key numerical point. If the number of key numerical points obtained by taking the intersection still does not meet the required number (for example, less than the second number threshold), the numerical points with the difference in number can be further extracted from the first key point and the second key point as the key numerical points. The difference in number refers to the difference between the second number threshold and the number of key numerical points obtained by taking the intersection.

[0139] In another embodiment, Figure 7 As shown, the data in the union is subsampled, and the subsampled data is used as key numerical points, including:

[0140] Step 702: Determine the number of data to be deleted based on the first number threshold and the number of data in the union.

[0141] Step 704, calculating the time difference and the value difference between adjacent data in the union.

[0142] Step 706, based on the time difference and the value difference, the key value points of the deletion quantity are deleted, and the retained data are used as the key value points, wherein the smaller the time difference and / or the smaller the value difference, the higher the priority of being deleted.

[0143] In this embodiment, optionally, the deletion quantity may be the difference between the number of data in the union and a first quantity threshold, or may be the difference between the number of data in the union and a second quantity threshold, or may be any suitable value between the two differences.

[0144] Optionally, the data to be deleted may be any suitable data in the union, such as any suitable first key point or second key point mentioned above. Preferably, the data to be deleted may be extracted from the second key point, specifically, may be extracted from the second key point that does not belong to the intersection, and the numerical points of the required number of deletions are extracted from the second key point that does not belong to the intersection.

[0145] The calculated union can be sorted according to the time sequence corresponding to the numerical points, and the time difference and numerical difference between two adjacent numerical points can be calculated, and numerical points with smaller time difference and smaller numerical difference can be deleted first, so that after the numerical points are deleted, the remaining numerical points meet the corresponding quantity requirements. Among them, the numerical points can be deleted in the second key point that does not belong to the above intersection.

[0146] When the time difference and value difference between two adjacent numerical points are smaller, it means that the characteristics of the original time series data reflected by the two numerical points are similar, and only one of the two numerical values ​​needs to be retained. For example, the corresponding time difference threshold and value difference threshold can be set. When the time difference between two adjacent numerical values ​​is less than the corresponding time difference threshold, and the absolute value of the value difference is also less than the corresponding value difference threshold, one of the two numerical points can be selected for deletion. When the number of numerical points that meet this condition exceeds the number of numerical points to be deleted, the data points are deleted according to the above priority.

[0147] In one embodiment, after sampling key numerical points from the numerical list, it also includes: detecting whether the number of data in the set is less than a second quantity threshold, and if so, returning to execute the quartile statistics method to obtain the corresponding multiple second key points from the numerical list, so that the number of second key points obtained increases.

[0148] In this embodiment, the k value set in the quartile statistics method can be a fixed value, or a value that matches the data type and / or data attribute. When the number of key numerical points obtained according to the set k value is less than the second quantity threshold, the k value is adjusted according to the difference between the k value and the second quantity threshold, and the above step of "obtaining the corresponding multiple second key points from the numerical list through the quartile statistics method" is re-executed to increase the number of second key points obtained, thereby increasing the total number of key numerical points and meeting the corresponding key numerical point quantity requirement.

[0149] In one embodiment, after step 112, the method further includes: receiving a display instruction for the data pair; and calling a time function to fit the data pair according to the display instruction to form and display fitting data.

[0150] In this embodiment, the relevant user can perform relevant operations to trigger the display instruction for the stored data pair. After receiving the display instruction, the electronic device displays the stored data pair, and fits the data pair according to the determined time function matching the data pair, so that when the data pair is displayed, the corresponding fitting data is further displayed, wherein the fitting data can form a corresponding curve according to the time point, for example, it can be displayed as follows Figure 4 The fitting data of the red dotted line in the figure enables relevant personnel to understand the changing characteristics and trends presented by the original time series data.

[0151] In one embodiment, a compression processing device for time series data is provided, such as Figure 8 As shown, the device comprises:

[0152] The time series data acquisition module 802 is used to acquire the original time series data to be compressed, where the original time series data includes a value list of the original data and a time list for reflecting the time information of the value of the original data.

[0153] The key data sampling module 804 is used to sample key value points from the value list; obtain the time points corresponding to the key value points from the time list; form data pairs with the sampled key value points and time points; and determine the time function that matches the data pair.

[0154] The data storage module 806 is used to store the data pairs and the time function so that the confidence of the fitted data fitted according to the time function and the data pairs relative to the original time series data is not less than the confidence threshold.

[0155] In one embodiment, the key data sampling module 804 is also used to extract multiple first key points that meet the configuration rules from the numerical list; obtain the corresponding multiple second key points from the numerical list through the quartile statistics method; and use one or more of the first key points and the second key points as key numerical points.

[0156] In one embodiment, the key data sampling module 804 is also used to use the initial value point in the value list as the extracted first key point; extract the unextracted value points from the value list in sequence, and compare whether the currently extracted value point meets the configuration rules with the last extracted first key point; if so, use the currently extracted value point as the first key point, update the last extracted first key point, and return to execute the extraction of the unextracted value points from the value list in sequence.

[0157] In one embodiment, the key data sampling module 804 is also used to calculate the corresponding upper quartile Q3 and lower quartile Q1 from the numerical list; based on the upper quartile Q3 and the lower quartile Q1, the corresponding upper limit value P1 and lower limit value P2 are calculated, P1 = Q3 + k × (Q3-Q1), P2 = Q1-k × (Q3-Q1), k is the IQR coefficient; one or more of the numerical points in the numerical list that are greater than the upper limit value P1 and less than the lower limit value P2 are used as the second key point.

[0158] In one embodiment, the key data sampling module 804 is also used to take the union of a first set formed by the first key point and a second set formed by the second key point; detect whether the number of data in the union exceeds a first quantity threshold, and if so, perform sub-sampling on the data in the union, and use the sub-sampled data as key numerical points, so that the number of sub-sampled key numerical points does not exceed the first quantity threshold.

[0159] In one embodiment, the configuration rules include multiple types. The key data sampling module 804 is further used to find the intersection of the first set formed by the first key points and the second set formed by the second key points, and use the key points in the intersection as the key value points obtained by secondary sampling.

[0160] In one embodiment, the key data sampling module 804 is further configured to use the first key point extracted by at least two configuration rules as a key value point obtained by secondary sampling.

[0161] In one embodiment, the key data sampling module 804 is also used to determine the number of data to be deleted based on the first quantity threshold and the number of data in the union; calculate the time difference and the numerical difference between adjacent data in the union; delete the key numerical points of the deletion number based on the time difference and the numerical difference, and use the retained data as the key numerical points, wherein the smaller the time difference and / or the smaller the numerical difference, the higher the priority of deletion.

[0162] In one embodiment, the key data sampling module 804 is also used to detect whether the number of data in the set is less than a second quantity threshold. If so, it returns to execute the quartile statistics method to obtain the corresponding multiple second key points from the numerical list, thereby increasing the number of second key points obtained.

[0163] In one embodiment, the key data sampling module 804 is further used to obtain the data type and / or data attribute to which the key value points in the data pair belong; and determine a time function that matches the data type and / or data attribute.

[0164] In one embodiment, the key data sampling module 804 is also used to call multiple candidate time functions to fit the data pair to form corresponding candidate fitting data; calculate the confidence between each candidate fitting data and the original time series data; and use the candidate time function with the highest confidence as the time function that matches the data pair.

[0165] In one embodiment, Fig. 9 As shown, the above-mentioned time series data compression processing device also includes:

[0166] The display module 808 is used to receive a display instruction for a data pair; call a time function to fit the data pair according to the display instruction, and form and display fitting data.

[0167] In one embodiment, the time series data acquisition module 802 is also used to acquire the collected data collected in real time by at least one sensor in the data center to form a collected data set; segment the collected data set according to a preset time length; use the segmented collected data as the original time series data; the collected data includes state quantity data and analog quantity data.

[0168] In one embodiment, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the steps in the above-mentioned method embodiments.

[0169] In one embodiment, an electronic device is also provided, including one or more processors; a memory, in which one or more programs are stored, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the steps in the above-mentioned method embodiments.

[0170] In one embodiment, an electronic device is provided, which may be the above-mentioned CRT measuring instrument. Fig.10 As shown, the electronic device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0171] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0172] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, including a computer-readable medium carrying instructions, in such an embodiment, the instructions can be downloaded and installed from a network through a communication part 1009, and / or installed from a removable medium 1011. When the instructions are executed by a central processing unit (CPU) 1001, the various method steps described in the present invention are executed.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0174] In addition, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present application, and should not be regarded as an admission or in any form to imply that the information constitutes prior art already known to those skilled in the art.

Claims

1. A compression processing method for time series data, characterized in that: The method comprises: Acquire original time series data to be compressed, the original time series data comprising a value list of original data and a time list for reflecting time information of the value of the original data; Sampling key value points from the value list; Obtaining the time point corresponding to the key value point from the time list; The key numerical points and time points of the sampling are formed into data pairs; determining a time function matching the data pair; The data pair and the time function are stored so that the confidence of the fitted data fitted according to the time function and the data pair relative to the original time series data is not less than a confidence threshold.

2. The method according to claim 1, characterized in that The step of sampling key value points from the value list includes: Extracting a plurality of first key points that meet the configuration rule from the value list; Acquire a plurality of corresponding second key points from the value list by using quartile statistics method; One or more of the first key point and the second key point is used as the key value point.

3. The method according to claim 2, characterized in that The configuration rules include one or more of the following rules: 1) The initial value point, maximum value point and minimum value point in the numerical list; 2) the alarm value points for triggering an alarm and the alarm recovery value points for triggering an alarm recovery in the value list; 3) Reference value points captured from the value list by a preset lossy compression algorithm; 4) If the absolute value of the difference between the currently extracted numerical point and the first extracted key point exceeds a preset difference threshold, the currently extracted numerical point is used as the first key point; 5) If the time interval between the currently extracted numerical point and the first extracted key point exceeds a preset time difference, the currently extracted numerical point is used as the first key point.

4. The method according to claim 3, characterized in that The step of extracting a plurality of first key points that meet the configuration rule from the value list includes: Step 1: taking the initial value point in the value list as the first key point extracted; Step 2: extracting unextracted value points from the value list in order, and comparing whether the currently extracted value point satisfies the configuration rule with the last extracted first key point; Step 3: If satisfied, the currently extracted value point is used as the new extracted first key point, and the last extracted first key point is stored; Step 4: Return to execute steps 2 and 3 until all the value points in the value list are extracted in sequence.

5. The method according to claim 2, characterized in that: The obtaining the corresponding plurality of second key points from the value list by using the quartile statistics method includes: Calculate the corresponding upper quartile Q3 and lower quartile Q1 from the numerical list; Based on the upper quartile Q3 and the lower quartile Q1, the corresponding upper limit value P1 and lower limit value P2 are calculated, P1 = Q3 + k × (Q3 - Q1), P2 = Q1 - k × (Q3 - Q1), k is the IQR coefficient; One or more of the numerical points in the numerical list that are greater than the upper limit value P1 and less than the lower limit value P2 are used as the second key point.

6. The method according to claim 2, characterized in that The taking one or more of the first key point and the second key point as the key value point comprises: Taking a union of a first set formed by the first key points and a second set formed by the second key points; Detect whether the number of data in the union exceeds a first quantity threshold. If so, subsample the data in the union and use the subsampled data as the key value points so that the number of subsampled key value points does not exceed the first quantity threshold.

7. The method according to claim 6, characterized in that The configuration rules include multiple types; the sub-sampling of the data in the union and using the sub-sampled data as the key value points includes: finding the intersection of a first set formed by the first key points and a second set formed by the second key points, and using the key points in the intersection as the key numerical points obtained by subsampling; and / or The first key point extracted by at least two configuration rules is used as the key value point obtained by secondary sampling.

8. The method according to claim 6, characterized in that The sub-sampling of the data in the union and using the sub-sampled data as the key value points includes: Determining the number of data to be deleted according to the first number threshold and the number of data in the union; Calculate the time difference and value difference between adjacent data in the union; The key value points of the deletion quantity are deleted based on the time difference and the value difference, and the retained data are used as the key value points, wherein the smaller the time difference and / or the smaller the value difference, the higher the priority of being deleted.

9. The method according to claim 6, characterized in that After sampling the key value points from the value list, the method further includes: Check whether the number of data in the set is less than the second number threshold, and if so, return to execute the method of obtaining the corresponding multiple second key points from the value list by the quartile statistical method, so as to increase the number of the obtained second key points.

10. The method according to claim 1, characterized in that The determining of a time function matching the data pair comprises: Obtaining the data type and / or data attribute of the key value point in the data pair; A temporal function is determined that matches the data type and / or data attribute.

11. The method according to claim 1, characterized in that: The determining of a time function matching the data pair comprises: Calling multiple candidate time functions to fit the data pair to form corresponding candidate fitting data; Calculating the confidence between each candidate fitting data and the original time series data; The candidate time function with the highest confidence is taken as the time function matching the data pair.

12. The method according to claim 1, characterized in that After storing the data pair and the time function, the method further includes: receiving a display instruction for the data pair; The time function is called according to the display instruction to fit the data pair, and the fitting data is formed and displayed.

13. The method according to any one of claims 1 to 9, characterized in that Before obtaining the original time series data to be compressed, the method further includes: Acquire collected data collected in real time by at least one sensor in the data center to form a collected data set; Segmenting the collected data set according to a preset time length; The segmented collected data is used as the original time series data; the collected data includes state quantity data and analog quantity data.

14. A compression processing device for time series data, characterized in that: The device comprises: A time series data acquisition module, used to acquire original time series data to be compressed, wherein the original time series data includes a value list of the original data and a time list for reflecting time information of the value of the original data; A key data sampling module is used to sample key value points from the value list; obtain the time points corresponding to the key value points from the time list; form data pairs with the sampled key value points and time points; and determine a time function that matches the data pair; A data storage module is used to store the data pair and the time function so that the confidence of the fitted data fitted according to the time function and the data pair relative to the original time series data is not less than a confidence threshold.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores executable instructions, which, when executed by a processor, enable the processor to perform the method according to any one of claims 1 to 13.

16. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, causes the one or more processors to perform the method according to any one of claims 1 to 13.

Images