Data compression method and system

Abstract

The present application discloses a data compression method and system, comprising: processing each acquisition point in the data to be compressed according to a time sequence; if a feature point does not yet exist in the data compression system, assigning the current acquisition point to the feature point and the previous point; the previous point is the point immediately before the current acquisition point; if a feature point already exists in the data compression system, determining whether the time difference between the current acquisition point and the current feature point is greater than a maximum time interval; if so, assigning the current acquisition point to the feature point and the previous point; if not, determining whether the previous point needs to be retained according to a revolving door algorithm; if so, storing the previous point and assigning the current acquisition point to the feature point and the previous point; if not, assigning the current acquisition point to the previous point. Lossy compression is performed on data according to the compression target, without requiring the user to set a compression deviation, resulting in high efficiency and convenience.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and specifically to a data compression method and system. Background Art

[0002] Process control equipment and sensor devices in the Industrial Internet of Things (IIoT) generate massive amounts of real-time data daily. By monitoring, controlling, analyzing, and mining this data, faults can be identified to maintain stable production operations. To reduce the storage space occupied by data, the revolving door (SDT) compression algorithm is currently widely used to compress real-time data.

[0003] The revolving door data compression algorithm specifies the amount of data distortion that users can tolerate before compressing time-stamped data. It then includes as many data points as possible within a given data trend. Essentially, the algorithm replaces the remaining data points between two stored data points with the remaining data points, then restores the remaining data points using linear interpolation.

[0004] However, the revolving door data compression algorithm assumes that all data is normal and does not account for data anomalies. Furthermore, the user-specified precision parameter E must be determined in advance. If E is too large, data trends will shift and become distorted. If E is too small, the compression ratio will be very low. Especially in the presence of noise and outliers, the revolving door data compression algorithm will barely compress data, resulting in a very low compression ratio. Summary of the Invention

[0005] To this end, the embodiments of the present application provide a data compression method and system that losslessly compresses data according to a compression target without requiring the user to set the compression deviation themselves, which is efficient and convenient.

[0006] In order to achieve the above objectives, the embodiments of the present application provide the following technical solutions:

[0007] According to a first aspect of an embodiment of the present application, a data compression method is provided, the method comprising:

[0008] Obtaining data to be compressed, and processing the collection points in the data to be compressed one by one according to the time sequence; each collection point carries the value of the collection point and a timestamp;

[0009] If the feature point does not exist in the data compression system, the current acquisition point is assigned to the feature point and the previous point; the previous point is the point at the previous moment of the current acquisition point;

[0010] If a feature point already exists in the data compression system, it is determined whether the time difference between the current acquisition point and the current feature point is greater than the longest time interval. If so, the current acquisition point is assigned to the feature point and the previous point. If not, it is determined whether the previous point needs to be retained according to the revolving door algorithm. If so, the previous point is stored and the current acquisition point is assigned to the feature point and the previous point. If not, the current acquisition point is assigned to the previous point.

[0011] Optionally, determining whether the previous point needs to be retained according to a revolving door algorithm includes:

[0012] A ray is drawn on the coordinate axis of the revolving door algorithm according to the compression deviation and the starting acquisition point; the compression deviation is determined according to the value of the previous point, the value of the current feature point, and the set compression ratio; the opening size of the upper and lower slopes of the ray is determined according to the compression deviation;

[0013] If the current acquisition point is outside the opening enclosed by the rays, it is determined that the previous point needs to be retained;

[0014] If the current acquisition point is within the opening enclosed by the rays, it is determined that the previous point does not need to be retained.

[0015] Optionally, after determining that the previous point does not need to be retained, the method further includes:

[0016] The up and down slopes of the ray in the revolving door algorithm are updated according to the current acquisition point.

[0017] Optionally, after determining that the previous point needs to be retained, the method further includes:

[0018] The current acquisition point is updated as the starting acquisition point of the next compression space.

[0019] Optionally, the maximum time interval is determined based on the inverse of the sensor acquisition frequency and a set compression ratio;

[0020] The set compression ratio is determined based on the data storage space occupied by the to-be-compressed data in one day, the target storage duration of the data, and the disk space.

[0021] According to a second aspect of an embodiment of the present application, a data compression system is provided, the system comprising:

[0022] A data acquisition module is used to acquire the data to be compressed and process the collection points in the data to be compressed one by one according to the time sequence; each collection point carries the value of the collection point and a timestamp;

[0023] An initial feature point determination module is used to assign the current acquisition point to the feature point and the previous point if the feature point does not exist in the data compression system; the previous point is the point at the previous moment of the current acquisition point;

[0024] The data compression module is configured to determine, if a feature point already exists in the data compression system, whether the time difference between the current acquisition point and the current feature point is greater than the longest time interval; if so, assign the current acquisition point to the feature point and the previous point; if not, determine, based on a revolving door algorithm, whether the previous point needs to be retained; if so, store the previous point and assign the current acquisition point to the feature point and the previous point; if not, assign the current acquisition point to the previous point.

[0025] Optionally, the data compression module is specifically configured to:

[0026] A ray is drawn on the coordinate axis of the revolving door algorithm according to the compression deviation and the starting acquisition point; the compression deviation is determined according to the value of the previous point, the value of the feature point, and the set compression ratio; the opening size of the upper and lower slopes of the ray is determined according to the compression deviation;

[0027] If the current acquisition point is outside the opening enclosed by the rays, it is determined that the previous point needs to be retained;

[0028] If the current acquisition point is within the opening enclosed by the rays, it is determined that the previous point does not need to be retained.

[0029] Optionally, the maximum time interval is determined based on the inverse of the sensor acquisition frequency and a set compression ratio;

[0030] The set compression ratio is determined based on the data storage space occupied by the to-be-compressed data in one day, the target storage duration of the data, and the disk space.

[0031] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0032] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the method described in the first aspect above.

[0033] In summary, the embodiments of the present application provide a data compression method and system. The method obtains data to be compressed and processes the collection points in the data to be compressed one by one according to a time sequence. Each collection point carries its value and a timestamp. If a feature point does not yet exist in the data compression system, the current collection point is assigned to the feature point and the previous point. The previous point is the point immediately before the current collection point. If a feature point already exists in the data compression system, the method determines whether the time difference between the current collection point and the current feature point is greater than the longest time interval. If so, the current collection point is assigned to the feature point and the previous point. If not, the method determines whether the previous point needs to be retained according to a revolving door algorithm. If so, the previous point is stored and the current collection point is assigned to the feature point and the previous point. If not, the method assigns the current collection point to the previous point. The method performs lossy compression on data according to the compression target, eliminating the need for the user to set a compression bias. This method is efficient and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0035] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0036] Figure 1 A flowchart of a data compression method provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a complete data compression method embodiment provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a revolving door algorithm provided in an embodiment of the present application;

[0039] Figure 4 Another schematic diagram of the revolving door algorithm provided in an embodiment of the present application;

[0040] Figure 5 Another schematic diagram of the revolving door algorithm provided in an embodiment of the present application;

[0041] Figure 6 A block diagram of a data compression system provided in an embodiment of the present application;

[0042] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown;

[0043] Figure 8 A schematic diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0045] Many scholars have studied improvements to the revolving door algorithm. One proposed an improved revolving door (SDT) algorithm based on dynamic tolerance adjustment. This algorithm dynamically adjusts the tolerance, increases the number of data points within the compression interval that have a significant impact on compression accuracy, and reduces compression errors. Another proposed data compression processing and performance analysis of the ISDT algorithm, which adaptively adjusts the tolerance to detect and remove outliers. Another proposed an improved revolving door compression algorithm. Based on the revolving door compression algorithm, it introduces an automatic threshold adjustment algorithm, enabling the threshold to automatically adjust based on actual values ​​and compression ratios. This algorithm achieves compression results comparable to or similar to manually specifying these parameters, reducing user effort. Furthermore, the compression accuracy parameters can be automatically adjusted based on data changes, ensuring relatively controllable compression ratios and fitting accuracy overall.

[0046] Figure 1 The present invention provides a data compression method flow, which performs lossy compression on data according to the compression target without requiring the user to set the compression deviation. The method is efficient and convenient.

[0047] Step 101: Obtain data to be compressed, and process the collection points in the data to be compressed one by one according to the time sequence; each collection point carries the value of the collection point and a timestamp;

[0048] Step 102: If the feature point does not exist in the data compression system, assign the current acquisition point to the feature point and the previous point; the previous point is the point at the previous moment of the current acquisition point;

[0049] Step 103: If a feature point already exists in the data compression system, determine whether the time difference between the current acquisition point and the current feature point is greater than the longest time interval. If so, assign the current acquisition point to the feature point and the previous point. If not, determine whether the previous point needs to be retained based on a revolving door algorithm. If so, store the previous point and assign the current acquisition point to the feature point and the previous point. If not, assign the current acquisition point to the previous point.

[0050] In a possible implementation, in step 103, determining whether the previous point needs to be retained according to a revolving door algorithm includes:

[0051] A ray is drawn on the coordinate axis of the revolving door algorithm according to the compression deviation and the starting acquisition point; the compression deviation is determined according to the value of the previous point, the value of the current feature point, and the set compression ratio; the opening size of the upper and lower slopes of the ray is determined according to the compression deviation;

[0052] If the current acquisition point is outside the opening enclosed by the rays, it is determined that the previous point needs to be retained;

[0053] If the current acquisition point is within the opening enclosed by the rays, it is determined that the previous point does not need to be retained.

[0054] In a possible implementation, after determining that the previous point does not need to be retained, the method further includes: updating the upper and lower slopes of the ray in the revolving door algorithm according to the current acquisition point.

[0055] In a possible implementation, after determining that the previous point needs to be retained, the method further includes: updating the current acquisition point to a starting acquisition point of the next compression space.

[0056] In one possible implementation, the maximum time interval is determined based on the inverse of the sensor acquisition frequency and a set compression ratio; the set compression ratio is determined based on the data storage space occupied by the data to be compressed in one day, the target storage data duration, and the disk space.

[0057] As can be seen, the above method, by quantifying the compression ratio, allows data of a specified duration to be stored within limited disk space, reducing the need for users to expand disk capacity once the disk is full. It also adaptively calculates the compression deviation and records key data to align with data trends, thus reducing the user's difficulty in setting the compression deviation.

[0058] Figure 2 The complete flow chart of the data compression method provided by the embodiment of the present application is shown below. Figure 2 The data compression method provided in the embodiment of the present application is described.

[0059] First, estimate the data storage space (bytes) for one day of raw data and determine the time and disk space required to store the data.

[0060] The original data refers to the data before compression using compression software. In the embodiment of the present application, it is the data collected and recorded at intervals of one second. For example, the original temperature data for 2 seconds is {time=20220-12-25 16:30:01, quality=ok, value=36℃}, {time=20220-12-2516:30:02, quality=ok, value=36℃}.

[0061] The daily data storage space for each data entry is the number of bytes it occupies within a 24-hour period. The data collection frequency is fixed. For example, if a record is collected every one second, 60 records are collected per minute, 60*60 = 3600 records per hour, and 3600*24 = 86,400 records per day. A data entry {time = 20220-12-25 16:30:01, quality = ok, value = 36°C} occupies 8 + 4 + 8 = 20 bytes of storage space on a 64-bit operating system ({longlong, int, double}), regardless of byte alignment.

[0062] The raw data storage space for one day is the disk space occupied by the compressed data collected in one day. For example, if there are 100 collection points, each collecting data at a rate of one second, and each data point occupies 20 bytes of disk space, then the raw data is 100 * 20 * 86400 = 172,800,000 bytes.

[0063] Secondly, set the compression ratio (k) = original data storage space for one day (bytes) * 365 (days) * N (years) / disk space (G) / 1024 (M) / 1024 (K) / 1024 (Byte).

[0064] Thirdly, the longest time interval Δt=(t_cur2-t_cur1)*k is calculated.

[0065] Among them, t_cur2-t_cur1 is the inverse of the acquisition frequency of the tag point at the current moment, t_cur is the timestamp of the tag point at the current moment, and k is the set compression ratio.

[0066] Fourthly, the compression deviation ΔE=abs(v_pre-v_save)*k is calculated; wherein v_pre is the value of the label point at the previous moment, v_save is the value judged as the feature point, and k is the set compression ratio.

[0067] The compression deviation in the traditional revolving door algorithm is a fixed value, while the compression deviation calculation method provided in the embodiment of the present application is adjusted based on the value of the label point and the value of the feature point at the current moment and the compression ratio. When the difference between the current value and the feature point changes, the compression deviation will also change accordingly.

[0068] In the fifth aspect, three points are defined: the feature point {v_save, t_save}, the previous point {v_pre, t_pre}, and the current point {v_cur, t_cur}. v_* is the value, and t_* is the timestamp of the corresponding moment.

[0069] The feature point {v_save, t_save} is the point where the current point exceeds the longest time interval △t, or is determined to be a point that needs to be stored according to the revolving door algorithm.

[0070] The previous point {v_pre, t_pre} is the point one moment before the current point.

[0071] The current point {v_cur, t_cur} is the point with the largest timestamp in the current system.

[0072] In a sixth aspect, compressing the data to be compressed specifically includes the following steps:

[0073] Step 1: Get the collection point and determine whether it is the first point in the system.

[0074] Data is collected by monitoring instruments. For example, a temperature sensor can have a model like {time = 20220-12-25 16:30:02, quality = ok, value = 36°C}, where time represents the collection time, quality indicates the quality of the current data (good, bad, offline, etc.), and value represents the collected value. These three attributes represent a temperature sensor. This is a collection point, and all sensors can be represented using these three attributes.

[0075] In the collector, configure the corresponding acquisition protocol, acquisition properties, device number, data type, register address, etc. in advance according to the needs.

[0076] Step 2: If it is the first point in the system, identify the collected point as a feature point. Store the feature point {v_save, t_save}. Update the value of the feature point to the value of the previous point: v_save = v_pre, t_save = t_pre. {v_save, t_save} = {v_pre, t_pre}.

[0077] Step 3: If it is not the first point in the system, determine whether the time difference between the current point and the feature point exceeds the longest time interval △t. If so, update the feature point and the previous point as the current point, {v_save, t_save} = {v_cur, t_cur} and {v_pre, t_pre} = {v_cur, t_cur}; then determine whether the previous point needs to be retained according to the revolving door algorithm; if so, store the previous point, and update the feature point and the previous point as the current point; if not, assign the current point to the previous point {v_pre, t_pre} = {v_cur, t_cur}, and update the upper and lower slopes.

[0078] According to the revolving door algorithm, the current point value either falls within the angle between the two lines, in which case the upper or lower slope is updated to reduce the angle range. Alternatively, the current point value falls outside the angle between the two lines, exceeding the deviation range. The previous point needs to be stored, and the current point is considered a feature point.

[0079] Through the above compression process, the collection points in the time series are compressed by this method, and only a small number of data points need to be retained, thereby reducing the disk space required for storage.

[0080] Figure 3 The diagram shows a revolving door algorithm provided by an embodiment of the present application, with E as the compression deviation, a, b, c, d, e, f, and g as the data to be compressed, and I as the starting point, which is also the value to be compressed.

[0081] Save the starting point i. Then, using i as the base, calculate ±E to get two points, i+E and iE. Then, using the incoming point a as the vertex and i+E and iE as the edges, draw two rays, with an upward slope and a downward slope, respectively. The upward slope should increase, while the downward slope should decrease. You can determine whether the previous point is scalable by comparing the angle between them, which is essentially a comparison of the upper and lower slopes.

[0082] exist Figure 4 In the example, b is within the range of the upper and lower slopes of a, so a needs to be discarded and the upper and lower slopes are updated according to b, making the upper slope larger and the lower slope smaller.

[0083] exist Figure 5 In the example, the newly acquired data point c is outside the updated upper and lower slopes, so point b is determined to be a point that needs to be stored. c is updated as a feature point, and the same operation as starting point i is repeated. During the reconstruction process, the revolving door algorithm replaces the original data with a straight line defined by the start and end points of the compressed interval.

[0084] As can be seen, the data compression method provided in the embodiment of the present application can estimate the storage space m1 occupied by data stored without compression for several years and the available disk space m2. m2 / m1 is the compression ratio, and the compression deviation is calculated based on the compression ratio. This eliminates the need for users to set the compression deviation. When the disk space is full, the space occupied by files from the past few years can be overwritten, eliminating the need for capacity expansion. This allows data of a specified time period to be stored in limited disk space, reducing disk capacity expansion operations during operation and maintenance.

[0085] By quantifying the compression ratio, users can estimate the space occupied by the original data after compression and set an appropriate compression ratio based on the disk space. The compression deviation is calculated based on the compression ratio, and the size of the compression deviation is adaptively calculated to reduce the possibility of users setting an inappropriate compression deviation. This allows key data to be recorded and adapted to data trend changes.

[0086] Key data are the peaks and troughs of the trend curve within each time interval. After recording the key data, the curve formed by these key data represents the fitted trend. This is equivalent to taking the broken line drawn from the original 10 points and subtracting 5 points. The remaining 5 key points are connected to form a new broken line.

[0087] In summary, the embodiments of the present application provide a data compression method, which obtains data to be compressed and processes the collection points in the data to be compressed one by one according to the time sequence; each collection point carries the value and timestamp of the collection point; if the feature point does not yet exist in the data compression system, the current collection point is assigned to the feature point and the previous point; the previous point is the point at the moment before the current collection point; if the feature point already exists in the data compression system, it is determined whether the time difference between the current collection point and the current feature point is greater than the longest time interval. If so, the current collection point is assigned to the feature point and the previous point; if not, it is determined whether the previous point needs to be retained according to the revolving door algorithm. If so, the previous point is stored and the current collection point is assigned to the feature point and the previous point. If not, the current collection point is assigned to the previous point. Lossy compression is performed on data according to the compression target, and the user does not need to set the compression deviation. This method is efficient and convenient.

[0088] Based on the same technical concept, the embodiment of the present application also provides a data compression system, such as Figure 6 As shown, the system includes:

[0089] The data acquisition module 601 is used to acquire the data to be compressed and process the collection points in the data to be compressed one by one according to the time sequence; each collection point carries the value of the collection point and a timestamp;

[0090] The initial feature point determination module 602 is configured to assign the current acquisition point to the feature point and the previous point if the feature point does not yet exist in the data compression system; the previous point is the point immediately before the current acquisition point;

[0091] The data compression module 603 is configured to, if a feature point already exists in the data compression system, determine whether the time difference between the current acquisition point and the current feature point is greater than the maximum time interval; if so, assign the current acquisition point to the feature point and the previous point; if not, determine whether the previous point needs to be retained based on a revolving door algorithm; if so, store the previous point and assign the current acquisition point to the feature point and the previous point; if not, assign the current acquisition point to the previous point.

[0092] In a possible implementation, the data compression module 603 is specifically configured to:

[0093] A ray is drawn in the coordinate axis of the revolving door algorithm based on the compression deviation and the starting acquisition point; the compression deviation is determined based on the value of the previous point, the value of the feature point, and the set compression ratio; the opening size of the upper and lower slopes of the ray is determined based on the compression deviation; if the current acquisition point is outside the opening enclosed by the ray, it is determined that the previous point needs to be retained; if the current acquisition point is within the opening enclosed by the ray, it is determined that the previous point does not need to be retained.

[0094] In one possible implementation, the maximum time interval is determined based on the inverse of the sensor acquisition frequency and a set compression ratio; the set compression ratio is determined based on the data storage space occupied by the data to be compressed in one day, the target storage data duration, and the disk space.

[0095] The present application also provides an electronic device corresponding to the method provided in the above embodiment. Figure 7 , which shows a schematic diagram of an electronic device provided in some embodiments of the present application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected via the bus 202. The memory 201 stores a computer program executable on the processor 200. When the processor 200 executes the computer program, it executes the method provided in any of the aforementioned embodiments of the present application.

[0096] The memory 201 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element and at least one other network element are connected via at least one physical port 203 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0097] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs. The processor 200 executes the programs upon receiving execution instructions. The methods disclosed in any of the aforementioned embodiments of the present application may be applied to or implemented by the processor 200.

[0098] The processor 200 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 200 or by software instructions. The above processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 201 , and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

[0099] The electronic device provided in the embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented by them.

[0100] The present application also provides a computer-readable storage medium corresponding to the method provided in the above embodiment. Figure 8The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided by any of the aforementioned embodiments is executed.

[0101] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0102] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0103] It should be noted that:

[0104] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other device. Various general-purpose devices may also be used in conjunction with the teachings herein. Based on the above description, it is apparent that the structure required for constructing such devices is suitable. In addition, the present application is not directed to any specific programming language. It should be understood that various programming languages ​​may be utilized to implement the present application described herein, and the above description of specific languages ​​is provided for the purpose of disclosing the best mode of implementation of the present application.

[0105] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0106] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in fewer than all the features of the individual embodiments disclosed above. Accordingly, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim itself serving as a separate embodiment of the present application.

[0107] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0108] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0109] The various component embodiments of the present application can be implemented in hardware, or implemented in a software module running on one or more processors, or implemented in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the creation device of the virtual machine according to an embodiment of the present application. The application can also be implemented as a part or all of the equipment or device program (for example, computer program and computer program product) for performing the method described herein. Such a program realizing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0110] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0111] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data compression method, characterized in that: The method comprises: Obtaining data to be compressed, and processing the collection points in the data to be compressed one by one according to the time sequence; each collection point carries the value of the collection point and a timestamp; If the feature point does not exist in the data compression system, the current acquisition point is assigned to the feature point and the previous point; the previous point is the point at the previous moment of the current acquisition point; If a feature point already exists in the data compression system, determine whether the time difference between the current acquisition point and the current feature point is greater than the longest time interval. If so, assign the current acquisition point to the feature point and the previous point. If not, determine whether the previous point needs to be retained based on a revolving door algorithm. If so, store the previous point and assign the current acquisition point to the feature point and the previous point. If not, assign the current acquisition point to the previous point. Wherein, judging whether the previous point needs to be retained according to the revolving door algorithm includes: drawing a ray in the coordinate axis of the revolving door algorithm according to the compression deviation and the starting acquisition point; the compression deviation is determined according to the value of the previous point, the value of the current feature point and the set compression ratio; the opening size of the upper and lower slopes of the ray is determined according to the compression deviation; if the current acquisition point is outside the opening enclosed by the ray, judging that the previous point needs to be retained; if the current acquisition point is within the opening enclosed by the ray, judging that the previous point does not need to be retained.

2. The method according to claim 1, wherein After determining that the previous point does not need to be retained, the method further includes: The up and down slopes of the ray in the revolving door algorithm are updated according to the current acquisition point.

3. The method according to claim 1, wherein After determining that the previous point needs to be retained, the method further includes: The current acquisition point is updated as the starting acquisition point of the next compression space.

4. The method according to claim 1, wherein The maximum time interval is determined based on the inverse of the sensor acquisition frequency and the set compression ratio; The set compression ratio is determined based on the data storage space occupied by the to-be-compressed data in one day, the target storage duration of the data, and the disk space.

5. A data compression system, characterized in that: The system comprises: A data acquisition module is used to acquire the data to be compressed and process the collection points in the data to be compressed one by one according to the time sequence; each collection point carries the value of the collection point and a timestamp; An initial feature point determination module is used to assign the current acquisition point to the feature point and the previous point if the feature point does not exist in the data compression system; the previous point is the point at the previous moment of the current acquisition point; The data compression module is configured to, if a feature point already exists in the data compression system, determine whether a time difference between a current acquisition point and a current feature point is greater than a maximum time interval; if so, assign the current acquisition point to the feature point and the previous point; if not, determine whether the previous point needs to be retained according to a revolving door algorithm; if so, store the previous point and assign the current acquisition point to the feature point and the previous point; if not, assign the current acquisition point to the previous point; the data compression module is specifically configured to: draw a ray on a coordinate axis of the revolving door algorithm based on a compression deviation and a starting acquisition point; the compression deviation is determined based on a value of the previous point, a value of the feature point, and a set compression ratio; the opening size of the upper and lower slopes of the ray is determined based on the compression deviation; if the current acquisition point is outside the opening enclosed by the ray, determine that the previous point needs to be retained; and if the current acquisition point is within the opening enclosed by the ray, determine that the previous point does not need to be retained.

6. The system according to claim 5, wherein: The maximum time interval is determined based on the inverse of the sensor acquisition frequency and the set compression ratio; The set compression ratio is determined based on the data storage space occupied by the to-be-compressed data in one day, the target storage duration of the data, and the disk space.

7. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1 to 4.

Images