Lossless compression and decompression method based on time sequence data and related equipment

Abstract

The invention discloses a lossless compression and decompression method based on time series data, which is characterized in that a sampled numerical value, a timestamp and a quality code are written into a cache when one piece of data is sampled, and direct write-in operation on a time series database can be reduced through a cache mechanism, so that the pressure of the database is reduced, the response speed of a system is improved, and the cache is subjected to local processing; the preprocessing step may comprise operations such as data cleaning, deduplication and sorting, the operations are beneficial to reducing redundant data, improving compression efficiency and ensuring the quality of compressed data, the preprocessed data are compressed according to the sequence of ID, starting time, ending time, quantity, timestamp, quality code and numerical value, the data are compressed according to a specific sequence, and the compression efficiency is improved. And the decompression process is a process of recovering the compressed data to an original state, so that the problems of high read-write pressure and high storage pressure of an upper computer of a traditional wind power control system are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data compression and decompression, and specifically relates to a lossless compression and decompression method based on time series data and related equipment. Background Art

[0002] The host computer of the wind power control system, as the core of wind farm data collection and management, plays a vital role in its operation. Whenever the system starts, the host computer begins to perform a complex and frequent task - periodically sampling various monitoring points in the lower computer. These monitoring points are spread over every corner of the wind farm, from the blade speed of the wind turbine to the vibration state of the tower, from the output power of the generator to the ambient temperature and wind speed, etc., every detail is accurately captured and converted into data.

[0003] Due to the large scale of wind farms, the number of monitoring points often reaches an astonishing tens of thousands or even hundreds of thousands. Each monitoring point carries valuable information about the operating status of the wind farm, and their stable operation is crucial to the overall efficiency and safety of the wind farm. However, it is precisely such high-density monitoring points that generate an inestimable amount of data at a sampling frequency of milliseconds.

[0004] These data are pouring into the host computer at an alarming speed, like a surging tide. If they are not processed and directly stored, the host computer will face unprecedented challenges. First, the read and write pressure of the hard disk will increase sharply, because the system needs to continuously write data to the disk and also be ready to read new data for processing. This high-intensity read and write operation will not only accelerate the wear of the hard disk, but also may cause the system response speed to slow down, affecting the sending of control instructions with high real-time requirements.

[0005] Secondly, disk space can quickly become scarce. Even with high-capacity hard disks, it is difficult to sustain such a large data storage demand for a long time. Over time, the disk space will gradually fill up and eventually become unable to store new data. This will not only interrupt the continuity of data, but may also affect the normal operation of the wind farm and subsequent data analysis.

[0006] In addition, if a large amount of raw data is directly stored without being processed, it will also bring great difficulties to the subsequent data management work. When it is necessary to find specific information or conduct data analysis, the system will have to spend a lot of time and resources to filter and extract useful information in the face of such a large data set. This not only reduces work efficiency, but may also affect the accuracy and timeliness of decision-making.

[0007] In summary, if the massive amount of data generated by the wind power control system host computer during the periodic sampling process is not effectively processed and managed, it will bring serious storage pressure, performance bottlenecks and data management problems to the system. These problems are not only related to the daily operation and efficiency improvement of wind farms, but also directly related to the sustainable development and energy security of the wind power industry. Summary of the invention

[0008] The present invention provides a lossless compression and decompression method based on time series data and related equipment, which solves the problems of high reading and writing pressure and high storage pressure of the host computer of the traditional wind power control system.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] A lossless compression and decompression method based on time series data, comprising:

[0011] Each time a data is sampled, the sampled value, timestamp, and quality code are written into the cache;

[0012] When the cache is full, preprocess the cache;

[0013] The preprocessed data is compressed in the order of ID, start time, end time, quantity, timestamp, quality code and value. After compression, the data is stored in the time series database.

[0014] When decompressing, the compressed data is read from the time series database and the original data is decompressed;

[0015] Fill the decompressed timestamp, quality code and value into the array in order.

[0016] Preferably, the steps of preprocessing the cache are specifically as follows:

[0017] Divide the data into 3 arrays for storage;

[0018] Sort timestamps, values ​​and quality codes in chronological order;

[0019] The timestamp, value and quality code are preprocessed in sequence.

[0020] Preferably, the steps for preprocessing the timestamp are as follows: starting from the second data, the timestamp is compared with the first data. If the timestamp difference is equal to the sampling period, the next data is compared until the timestamp difference is different. The data is saved as a timestamp, and the above steps are continued from the next inconsistent difference until all timestamp comparisons are completed.

[0021] Preferably, the step of preprocessing the quality code is specifically, starting from the second data, comparing the quality code with the first data, if consistent, comparing with the next one, until the quality codes are inconsistent, and performing the above steps starting from the next inconsistent quality code until all quality code comparisons are completed.

[0022] Preferably, the step of preprocessing the numerical value is specifically, starting from the second value, comparing the numerical value with the previous value, and if they are consistent, comparing with the next one, until the numerical values ​​are unequal, and performing the above steps starting from the next unequal numerical value until all numerical comparisons are completed.

[0023] Preferably, the LZO algorithm is used for both compression and decompression.

[0024] Preferably, the step of decompressing the original data is specifically to first determine the ID, start time and end time, and then restore the timestamp, quality code and numerical information in sequence.

[0025] A lossless compression and decompression system based on time series data, comprising:

[0026] Writing module: used to write the sampled value, timestamp, and quality code into the cache every time a data is sampled;

[0027] Preprocessing module: used to preprocess the cache when it is full;

[0028] Compression module: used to compress the preprocessed data in the order of ID, start time, end time, quantity, timestamp, quality code and value, and store the data in the time series database after compression;

[0029] Decompression module: used for decompression, reading compressed data from the time series database and decompressing the original data;

[0030] Filling module: used to fill the decompressed timestamp, quality code and value into the array in sequence.

[0031] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements the steps of a lossless compression and decompression method based on time series data.

[0032] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a lossless compression and decompression method based on time series data.

[0033] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a lossless compression and decompression method based on time series data. Each time a data is sampled, the sampled value, timestamp, and quality code are written into the cache. Through the cache mechanism, the direct write operation to the time series database can be reduced, thereby reducing the pressure on the database and improving the response speed of the system. The cache is processed locally. The preprocessing steps may include data cleaning, deduplication, sorting and other operations. These operations help to reduce redundant data, improve compression efficiency, and ensure the quality of compressed data. The preprocessed data is sorted according to ID, start time, end time, quantity, time, etc. The data is compressed in the order of stamp, quality code and numerical value. Compressing data in a specific order can more effectively utilize the characteristics of the compression algorithm, reduce data redundancy, and achieve a higher compression ratio. Storing the compressed data in a time series database can make full use of the storage structure and query performance optimized for time series data in the time series database to improve data retrieval efficiency and storage efficiency. The decompression process is the process of restoring the compressed data to its original state. Lossless compression ensures that the decompressed data is completely consistent with the original data, ensuring the integrity and accuracy of the data, and solving the problems of high read and write pressure and high storage pressure on the host computer of the traditional wind power control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a lossless compression and decompression method based on time series data of the present invention;

[0035] Figure 2 This is a flowchart of timestamp preprocessing according to an embodiment of the present invention;

[0036] Figure 3 This is a flow chart of quality code preprocessing according to an embodiment of the present invention;

[0037] Figure 4 This is a flowchart of the first arrival preprocessing according to an embodiment of the present invention;

[0038] Figure 5 The present invention is a block diagram of a lossless compression and decompression system based on time series data. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0040] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0042] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the present invention provides a lossless compression and decompression method based on time series data, including S101 writing the sampled value, timestamp, and quality code into a cache each time a data is sampled;

[0044] The cache mechanism can reduce direct write operations to the time series database, thereby alleviating the pressure on the database and improving the system's response speed. At the same time, the cache can also be used as a temporary storage area before data preprocessing to facilitate subsequent processing.

[0045] S102: When the cache is full, pre-process the cache;

[0046] The preprocessing steps may include data cleaning, deduplication, sorting, and other operations, which help reduce redundant data, improve compression efficiency, and ensure the quality of compressed data.

[0047] S103 compresses the preprocessed data in the order of ID, start time, end time, quantity, timestamp, quality code and value, and stores the data in a time series database after compression;

[0048] Compressing data in a specific order (such as ID, start time, end time, etc.) can more effectively utilize the characteristics of the compression algorithm, reduce data redundancy, and achieve a higher compression ratio. At the same time, this orderly compression method also facilitates subsequent decompression and data processing.

[0049] Storing compressed data in a time series database can fully utilize the storage structure and query performance optimized for time series data in the time series database, thereby improving data retrieval efficiency and storage efficiency.

[0050] During decompression at S104, the compressed data is read from the time series database and the original data is decompressed;

[0051] The decompression process is the process of restoring the compressed data to its original state. Lossless compression ensures that the decompressed data is completely consistent with the original data, ensuring the integrity and accuracy of the data.

[0052] S105 fills the decompressed timestamp, quality code and value into the array in order.

[0053] The decompressed data (including timestamp, quality code and value) is filled into the array in order to facilitate subsequent data processing and analysis. As an efficient data structure, the array can quickly access and modify data elements, improving the speed and flexibility of data processing.

[0054] The specific steps for preprocessing the cache are:

[0055] Divide the data into 3 arrays for storage;

[0056] Sort timestamps, values ​​and quality codes in chronological order;

[0057] The timestamp, value and quality code are preprocessed in sequence.

[0058] Dividing the data into three arrays (timestamp, value, and quality code) for storage helps to quickly locate and access relevant data during subsequent processing, thereby improving processing efficiency.

[0059] Sorting timestamps, values, and quality codes in chronological order can make consecutive identical or similar data more concentrated, which is beneficial for subsequent compression algorithms to identify and reduce redundancy and improve compression rate.

[0060] Through the preprocessing step, some obvious redundant data can be identified and removed in advance, further reducing storage requirements.

[0061] The specific steps for preprocessing the timestamp are as follows: starting from the second data, compare the timestamp with the first data. If the timestamp difference is equal to the sampling period, compare the next data until the timestamp difference is different. Save the data as a timestamp, and continue the above steps from the next inconsistent difference until all timestamp comparisons are completed.

[0062] By comparing the difference between the timestamp and the sampling period, data segments with consecutive identical timestamps can be identified, and only the starting timestamp and duration (or number of data points) are stored, thereby significantly reducing the storage requirements for timestamps.

[0063] Improve data continuity identification: This step helps to quickly restore the time continuity of data during decompression, facilitating subsequent data analysis and processing.

[0064] The specific steps for preprocessing the quality code are: starting from the second data, compare the quality code with the first data, if they are consistent, compare the next one, until the quality codes are inconsistent, and then perform the above steps from the next inconsistent quality code until all quality code comparisons are completed.

[0065] Reduce quality code storage: Continuous identical quality codes only need to be stored once, reducing the amount of quality code storage while retaining the quality information of the data.

[0066] Simplified data processing: In subsequent data processing, the corresponding quality processing logic can be directly applied according to the pre-processed quality code segment, which simplifies the processing flow.

[0067] The steps of preprocessing the values ​​are as follows: starting from the second value, the value is compared with the previous value, if they are consistent, then the next value is compared, until the values ​​are unequal, and the above steps are repeated from the next unequal value until all the value comparisons are completed.

[0068] Improve compression efficiency: By identifying consecutive identical values, the amount of storage for the values ​​can be reduced, allowing the compression algorithm to compress the data more effectively.

[0069] Maintain data accuracy: Lossless compression ensures that the original value can be fully restored after decompression, ensuring the precision and accuracy of the data.

[0070] The lzo algorithm is used for both compression and decompression.

[0071] High compression rate: The LZO algorithm is known for its high compression rate and fast compression / decompression speed, and is suitable for scenarios that require fast processing of large amounts of data.

[0072] Low resource usage: The LZO algorithm occupies relatively few CPU and memory resources during compression and decompression, which helps reduce system load.

[0073] The specific steps for decompressing the original data are to first determine the ID, start time and end time, and then restore the timestamp, quality code and numerical information in sequence.

[0074] Quickly restore data: By first determining the ID, start time and end time, and then restoring the timestamp, quality code and numerical information in sequence, the original data can be efficiently restored to facilitate subsequent data analysis and application.

[0075] Maintain data integrity: Lossless decompression ensures that the recovered data is completely consistent with the original data, ensuring data integrity and reliability.

[0076] The embodiment of the present invention provides a lossless compression and decompression method based on time series data, and the specific steps include:

[0077] 1. Apply for a memory that can store 2048 moments of data for each point. Each time a value is sampled, the sampled value, timestamp, and quality code are written into the cache.

[0078] 2 When the cache of 2048 moments is full, the cache is preprocessed as follows:

[0079] a) Divide the data of 2048 moments into three arrays for storage, store the timestamp in time

[2048] , the value in value

[2048] , and the quality code in quality

[2048] .

[0080] b) Sort the timestamp, data, and quality code according to the timestamp, from small to large.

[0081] c) Preprocess the timestamp, such as Figure 2 As shown. Starting from the second data, compare the timestamp with the first data. If the timestamp difference is equal to the sampling period (the sampling period of the host computer, usually 1s or 500ms), compare the next data until the timestamp difference is different and save the data as timestamp (number, the difference is equal to the number of timestamps). Continue the above steps from the next inconsistent difference until all timestamp comparisons are completed. The final data format is as follows: 2023-11-03-1-1-1-500(100)2023-11-03-1-2-1-000(1948)

[0082] d) Preprocess the quality code such as Figure 3 As shown. Starting from the second data, compare the quality code with the first data. If they are consistent, compare the next one until the quality codes are inconsistent. Repeat the above steps from the next inconsistent quality code until all quality codes are compared. The final data format is as follows: 0(2000)1(1)0(47).

[0083] e) Preprocess the values, such as Figure 4 As shown, starting from the second value, compare the value with the previous value. If they are consistent, compare with the next one until the values ​​are unequal. Repeat the above steps from the next unequal value until all value comparisons are completed. The final data format is as follows: 1.1(100)1.2(2)1.3(1900)1.4(46).

[0084] f) The preprocessed data is compressed using the lzo algorithm in the order of ID, start time, end time, quantity, timestamp, quality code, and value, and the data is stored in a time series database after compression.

[0085] Unzip:

[0086] a) Read the compressed data from the time series database and use lzo to decompress the original data.

[0087] b) After decompression, bytes 0-3 are ID, bytes 4-11 are start time, bytes 12-19 are end time, and bytes 20-21 are quantity.

[0088] c) Restore the timestamp information, read data from the 22nd byte, and stop when encountering (). For example, 2023-11-03-1-1-1-500(100), restore the first 100 timestamps to 2023-11-03-1-1-1-500 as the start time, and the scanning cycle is the arithmetic progression of the step length (2023-11-03-1-1-1-500 2023-11-03-1-1-2-000). The number in brackets is the number. When the number does not reach the total number, repeat the above steps.

[0089] d) Restore the quality code information. After the timestamp restoration is completed, continue to read the quality code information. Start reading data from the last ) of the timestamp and stop when encountering (). For example, 0(100) means that 100 consecutive quality codes are 0. Repeat the above steps until the total number of quality codes read reaches the total number.

[0090] e) Restore the value. After the quality code is restored, continue to restore the value. Start reading data from the last ) of the quality code and stop when you encounter (). For example, 1(100) means that 100 consecutive values ​​are 1. Repeat the above steps until the total number of values ​​read reaches the total number.

[0091] f) Fill the above parsed timestamp, quality code, and value into the array in order, which is the final result.

[0092] like Figure 5 As shown, the present invention provides a lossless compression and decompression system based on time series data, comprising:

[0093] Writing module: used to write the sampled value, timestamp, and quality code into the cache every time a data is sampled;

[0094] Preprocessing module: used to preprocess the cache when it is full;

[0095] Compression module: used to compress the preprocessed data in the order of ID, start time, end time, quantity, timestamp, quality code and value, and store the data in the time series database after compression;

[0096] Decompression module: used for decompression, reading compressed data from the time series database and decompressing the original data;

[0097] Filling module: used to fill the decompressed timestamp, quality code and value into the array in sequence.

[0098] The specific steps for preprocessing the cache are:

[0099] Divide the data into 3 arrays for storage;

[0100] Sort timestamps, values ​​and quality codes in chronological order;

[0101] The timestamp, value and quality code are preprocessed in sequence.

[0102] The specific steps for preprocessing the timestamp are as follows: starting from the second data, compare the timestamp with the first data. If the timestamp difference is equal to the sampling period, compare the next data until the timestamp difference is different. Save the data as a timestamp, and continue the above steps from the next inconsistent difference until all timestamp comparisons are completed.

[0103] The specific steps for preprocessing the quality code are: starting from the second data, compare the quality code with the first data, if they are consistent, compare the next one, until the quality codes are inconsistent, and then perform the above steps from the next inconsistent quality code until all quality code comparisons are completed.

[0104] The steps of preprocessing the values ​​are as follows: starting from the second value, the value is compared with the previous value, if they are consistent, then the next value is compared, until the values ​​are unequal, and the above steps are repeated from the next unequal value until all the value comparisons are completed.

[0105] The lzo algorithm is used for both compression and decompression.

[0106] The specific steps for decompressing the original data are to first determine the ID, start time and end time, and then restore the timestamp, quality code and numerical information in sequence.

[0107] A terminal device is provided in one embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0108] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0109] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0110] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0111] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0112] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0113] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields, and the above-mentioned specific embodiments are only illustrative and instructive, rather than restrictive. Under the guidance of the specification, a person skilled in the art can also make many forms without departing from the scope of protection of the claims of the present invention, all of which belong to the scope of protection of the present invention.

Claims

1. A lossless compression and decompression method based on time series data, characterized in that: include Each time a data is sampled, the sampled value, timestamp, and quality code are written into the cache; When the cache is full, preprocess the cache; The preprocessed data is compressed in the order of ID, start time, end time, quantity, timestamp, quality code and value. After compression, the data is stored in the time series database. When decompressing, the compressed data is read from the time series database and the original data is decompressed; Fill the decompressed timestamp, quality code and value into the array in order.

2. According to claim 1, a lossless compression and decompression method based on time series data is characterized in that: The specific steps for preprocessing the cache are: Divide the data into 3 arrays for storage; Sort timestamps, values ​​and quality codes in chronological order; The timestamp, value and quality code are preprocessed in sequence.

3. The method for lossless compression and decompression based on time series data according to claim 1, characterized in that: The specific steps for preprocessing the timestamp are as follows: starting from the second data, compare the timestamp with the first data. If the timestamp difference is equal to the sampling period, compare the next data until the timestamp difference is different. Save the data as a timestamp, and execute the above steps from the next timestamp with inconsistent difference until all timestamp comparisons are completed.

4. The method for lossless compression and decompression based on time series data according to claim 1, characterized in that: The specific steps for preprocessing the quality code are: starting from the second data, compare the quality code with the first data, if they are consistent, compare the next one, until the quality codes are inconsistent, and then perform the above steps from the next inconsistent quality code until all quality code comparisons are completed.

5. The method for lossless compression and decompression based on time series data according to claim 1, characterized in that: The steps of preprocessing the values ​​are as follows: starting from the second value, the value is compared with the previous value, if they are consistent, then the next value is compared, until the values ​​are unequal, and the above steps are repeated from the next unequal value until all the value comparisons are completed.

6. The method for lossless compression and decompression based on time series data according to claim 1, characterized in that: The lzo algorithm is used for both compression and decompression.

7. The method for lossless compression and decompression based on time series data according to claim 1, characterized in that: The specific steps for decompressing the original data are to first determine the ID, start time and end time, and then restore the timestamp, quality code and numerical information in sequence.

8. A lossless compression and decompression system based on time series data, characterized in that: include: Writing module: used to write the sampled value, timestamp, and quality code into the cache every time a data is sampled; Preprocessing module: used to preprocess the cache when it is full; Compression module: used to compress the preprocessed data in the order of ID, start time, end time, quantity, timestamp, quality code and value, and store the data in the time series database after compression; Decompression module: used for decompression, reading compressed data from the time series database and decompressing the original data; Filling module: used to fill the decompressed timestamp, quality code and value into the array in sequence.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a lossless compression and decompression method based on time series data as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a lossless compression and decompression method based on time series data as described in any one of claims 1 to 7 are implemented.

Images