A data storage system and compression method for time tags of industrial real-time data

Abstract

The application discloses a kind of data storage system and time label compression method of industrial real-time data, wherein the first data of any data block in data storage system includes first time label, second data includes second time compression value, the pth data includes the pth time compression value, the pth time compression value is the conversion value of the difference of the pth time label difference value and the (p-1)th time label difference value, the pth time label difference value is the difference of the pth time label and the (p-1)th time label, further include the time precision mark of corresponding data, the compression method of time label includes, first time label is stored completely, the corresponding n time label of the n data and the (n-1)th time label corresponding to the (n-1)th data obtain time deviation, and determine the time precision mark bit corresponding to the n data, can automatically apply different precision time label, provide better time label compression algorithm, greatly reduce the occupation of disk space.

Description

Technical Field

[0001] This invention relates to the field of industrial data processing, specifically to a data storage system for real-time industrial data and a method for compressing time stamps. Background Technology

[0002] In process industries, historical data such as process data and operating status of various production units within an enterprise are used to create trend charts, various statistical reports, and for various online process analyses. Production continues continuously, and historical data is constantly generated. In DCS applications, tag data is often collected at the second level, while time stamp storage supports the microsecond level.

[0003] Because each data point contains a timestamp, storing a complete timestamp for each data point would require a huge amount of space.

[0004] In industrial data storage, block storage is often used. This involves storing a continuous segment of data with the same identifier in a contiguous block of disk space. This continuous data segment stores only a reference time, and each subsequent data segment stores only the deviation from the reference time or the time deviation from the previous data segment. The storage of this deviation value can be either direct storage of the deviation or categorized storage of the deviation, with the deviation value being divided into seconds and milliseconds.

[0005] On storage systems that support microsecond time stamps, if the collected data is only in the second or millisecond range, directly storing the aforementioned deviation value often wastes space. For example, if the deviation value is 1 second, storing it on the disk would be 1,000,000, requiring 3 bytes. Storing the deviation in seconds and milliseconds separately also wastes more space compared to storing them directly at the same time. For example, if the deviation value is 1800 milliseconds, storing it uniformly only requires 2 bytes, while storing it separately requires 3 bytes.

[0006] Therefore, neither categorized storage nor direct storage can adequately support various scenarios, and a new time-stamp compression method is urgently needed to further reduce disk space usage based on the storage deviation value. Summary of the Invention

[0007] To overcome the shortcomings of the above technologies, this invention provides a data storage system for industrial real-time data and a time stamp compression method, which can automatically adapt to time stamps of different precision, provide a better time stamp compression algorithm, and greatly reduce disk space usage.

[0008] The technical solution adopted by this invention to overcome its technical problems is:

[0009] The first aspect of this application proposes a data storage system for industrial real-time data, comprising several data blocks for storing industrial real-time data. The first data in any data block of the tag data storage includes at least a first time tag, and the second data includes at least a second time compression value, wherein the second time compression value is a converted value of the difference between the second time tag and the first time tag, and the m-th time tag is the acquisition time value of the m-th industrial real-time data, where m ≥ 1. The p-th data corresponding to any data block includes at least a p-th time compression value, where p ≥ 3, and the p-th time compression value is a converted value of the difference between the p-th time tag difference and the (p-1)-th time tag difference, where the p-th time tag difference is the difference between the p-th time tag and the (p-1)-th time tag. The m-th data in any data block of the tag data storage further includes a time precision flag corresponding to the m-th time compression value.

[0010] Furthermore, the time precision flag is 2 bits.

[0011] The second aspect of this application proposes a time stamp compression method for a data storage system based on industrial real-time data, comprising: storing the first time stamp corresponding to the first data to be stored in the same data block completely in a cache; obtaining the time deviation based on the nth time stamp corresponding to the nth data to be stored in the same data block and the (n-1)th time stamp corresponding to the (n-1)th data, and determining the time precision flag bit corresponding to the nth data based on the time deviation, where n≥2; converting the time deviation into the nth time compression value based on the time precision flag bit corresponding to the nth data; storing the time precision flag bit and time compression value corresponding to the n data of the same data block respectively in a cache array, and determining whether the calculation of the time deviation of the industrial real-time data to be stored is completed; if the calculation of the time deviation of the industrial real-time data to be stored is not completed, then the calculation of the time deviation of the industrial real-time data continues; otherwise, the data in the cache is compressed and stored.

[0012] Furthermore, the step of obtaining the time deviation based on the nth time tag corresponding to the nth data to be stored in the same data block and the (n-1)th time tag corresponding to the (n-1)th data specifically includes: if n=2, then the time deviation of the second data to be stored in the same data block is the difference between the second time tag and the first time tag; if n≥3, then the time deviation of the nth data to be stored in the same data block is the time deviation between the difference of the nth time tag and the (n-1)th data, wherein the difference of the nth time tag is the difference between the nth time tag and the (n-1)th time tag.

[0013] Furthermore, the determination of the time precision flag corresponding to the nth data based on the time deviation specifically includes: if the time deviation is 0, the time precision flag is the first flag; if the time deviation is a microsecond time interval, the time precision flag is the second flag; if the time deviation is a millisecond time interval, the time precision flag is the third flag; and if the time deviation is a second time interval, the time precision flag is the fourth flag.

[0014] Furthermore, the step of converting the time deviation into the nth time compression value based on the time precision flag corresponding to the nth data specifically includes: if the time deviation is the third flag, then the time deviation is converted into milliseconds as the nth time compression value; if the time deviation is the fourth flag, then the time deviation is converted into seconds as the nth time compression value.

[0015] Based on the time flag, it adaptively determines whether time deviation conversion is needed.

[0016] Furthermore, the step of storing the time precision flag and time compression value corresponding to the n data of the same data block into a cache array, and determining whether the calculation of the time deviation of the industrial real-time data to be stored is completed, specifically includes: storing the time precision flag and time compression value into a first array and a second array respectively, and determining whether the calculation of the time deviation of the industrial real-time data to be stored is completed based on whether n+1 is greater than the number of data of the industrial real-time data to be stored.

[0017] Furthermore, the step of compressing and storing the data in the cache specifically includes encoding each piece of data in the second array based on ZigZag.

[0018] The encoded data will not contain negative numbers.

[0019] Furthermore, the step of compressing and storing the data in the cache also includes compressing the encoded second array and the first array based on Simple8B.

[0020] The beneficial effects of this invention are:

[0021] 1. By storing the difference between time stamp values ​​and setting a time precision flag, the disk space occupied by the industrial data storage system with microsecond-level time stamps is greatly reduced.

[0022] 2. It is compatible with the acquisition frequency and accuracy of industrial data in industrial control systems, and automatically adapts to time stamps of different accuracies. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the time-stamped compression format for industrial real-time data in the data storage system of this invention.

[0024] Figure 2 This is a flowchart of a method for compressing industrial real-time data time stamps according to an embodiment of the present invention;

[0025] Figure 3 This is a flowchart of a time stamp compression method according to an embodiment of the present invention. Detailed Implementation

[0026] First, some abbreviations and key terms mentioned in this invention will be explained.

[0027] Tag number: Industrial control field I / O point.

[0028] Tag data: The data for a tag at a specific moment, generally including a timestamp, a value, and a quality code. The timestamp indicates when the data was generated, the quality code indicates the data status and whether the data is reliable, and the value indicates the instrument reading.

[0029] To facilitate a better understanding of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following are merely exemplary and do not limit the scope of protection of the present invention.

[0030] In the embodiments of the present invention, the storage format and compression method of time stamps are mainly described.

[0031] The industrial real-time data storage system described in this application supports the storage of microsecond time stamps, and the industrial real-time data is stored in block storage format. For example, data with the same identifier for a continuous period of time is stored in a contiguous block of disk space. If a data block allocated to it is full, the data is allocated to the next data block to continue continuous storage. The timestamp of an industrial real-time data point is the time value at which the data was collected. Because the data storage system supports the storage of microsecond time stamps, the collected time value is in microseconds.

[0032] Furthermore, the unique characteristics of industrial real-time data must be considered when compressing it. Only a small portion of the tag data in industrial real-time data changes frequently, such as rapid oscillations caused by faults; in these cases, microsecond-level event tags are required for storage. However, much of the tag data exhibits slow-changing characteristics. The tag data that needs to be stored may only be saved when its operating state changes, such as during the opening and closing process of a tag. Data changes and time changes share common characteristics; for this type of data, millisecond-level and second-level time tags are sufficient for the application.

[0033] like Figure 1As shown in the diagram, this embodiment illustrates the time-stamped compression format of industrial real-time data in the data storage system. Each data point represents the industrial real-time data that will be stored in the historical data storage system. The target industrial real-time data storage system divides the storage of tag data into several data blocks.

[0034] The first data in any data block of the tag number data storage includes a first time tag; the second data corresponding to any data block includes at least a second time compression value, wherein the second time compression value is a converted value of the difference between the second time tag and the first time tag, and the m-th time tag is the acquisition time value of the m-th industrial real-time data, m≥1; the p-th data corresponding to any data block includes at least a p-th time compression value, wherein p≥3, the p-th time compression value is a converted value of the difference between the p-th time tag difference and the p-1-th time tag difference, and the p-th time tag difference is the difference between the p-th time tag and the p-1-th time tag; the m-th data in any data block of the tag number data storage also includes a time precision flag bit corresponding to the m-th time compression value.

[0035] In one embodiment of the present invention, the time precision flag is a 2-bit flag used to represent the time precision flag of the m-th time compression value.

[0036] Based on the data block format of the aforementioned data storage system, this invention also proposes a time-stamped compression method for industrial real-time data, such as... Figure 2 As shown, the first time tag corresponding to the first data to be stored in the same data block is completely stored in the cache; the time deviation is obtained based on the nth time tag corresponding to the nth data to be stored in the same data block and the (n-1)th time tag corresponding to the (n-1)th data, and the time precision flag bit corresponding to the nth data is determined based on the time deviation, where n≥2; based on the time precision flag bit corresponding to the nth data, the time deviation is converted into the nth time compression value; the time precision flag bits and time compression values ​​corresponding to the n data of the same data block are stored in the cache array respectively, and it is determined whether the calculation of the time deviation of the industrial real-time data to be stored is completed; if the calculation of the time deviation of the industrial real-time data to be stored is not completed, the calculation of the time deviation of the industrial real-time data continues; otherwise, the data in the cache is compressed and stored.

[0037] It should be noted that the actual data compression starts from the second data point, and the specific implementation steps are shown in Figure 3.

[0038] Step S101: Initialize parameters.

[0039] Let the initial variable deltaT = 0 for the time deviation, and the intermediate variable i = 2, where i is used as an intermediate variable to calculate the number of data to be processed.

[0040] Step S102: Calculate the time deviation.

[0041] Assume the time label for the first data point D1 is T1, the time label for the second data point D2 is T2, and the time label for the nth data point Dn is Tn.

[0042] Taking n=2 as an example, the intermediate variable for calculating the time deviation is T2-T1, and the time deviation delta = intermediate variable for calculating the time deviation - deltaT = T2-T1-0 = T2-T1; and the initial variable of the time deviation is deltaT = T2-T1.

[0043] S103, performs compression processing on the time deviation delta.

[0044] To determine whether the time deviation delta includes milliseconds or seconds, if delta = 0, the time precision flag is 00. If delta includes microseconds, the flag is 01, indicating a microsecond time interval. When delta includes milliseconds but not microseconds, the flag is 10, indicating a millisecond time interval. Then, delta is set to delta / 1000, converting microseconds to milliseconds.

[0045] When delta contains only second-level data, the time precision flag is identified by 11, which indicates a second-level time interval. Then, delta = delta / 1000 / 1000, which converts microseconds to seconds.

[0046] It should be noted that for the first data, the time precision flag is given a default value, which is usually 00.

[0047] S104. The calculated time deviation delta and time precision flag are placed into the time deviation array and the time precision flag array, respectively.

[0048] In step S105, after processing each piece of data, the intermediate variable i is incremented. For example, after processing the time deviation of the second piece of data, i is incremented by 1, becoming 3. If i is greater than the number of industrial real-time data points that need to be stored, further data compression processing is performed, and the process jumps to step S2. If i is less than or equal to the number of industrial real-time data points that need to be stored, the process returns to step S102 to calculate the time deviation of the next piece of data.

[0049] If i = 3, and the number of industrial real-time data points to be stored is 10, then return to S102 to calculate the time deviation of the third data point. The intermediate variable for the time deviation calculation is T3 - T2.

[0050] Time deviation delta = intermediate variable in time deviation calculation - deltaT = T3 - T2 - T2 - T1; and the initial variable of time deviation deltaT = T3 - T2. Therefore, time deviation is the deviation value of the time tag deviation.

[0051] S2 encodes each data point in the time deviation array using ZigZag to prevent negative time deviations.

[0052] In some implementations, the difference is due to the calculated time deviation. For example, the data tags to be saved are 3:00.100, 3:01.150, and 3:02.100, which represent 3 minutes 0 seconds 100 milliseconds, 3 minutes 01 seconds 150 milliseconds, and 3 minutes 02 seconds 100 milliseconds, respectively. The difference between the second and first time tags is 1050 milliseconds, and the difference between the third and second time tags is 950 milliseconds. Therefore, the third time compression value is 950 milliseconds minus 1050 milliseconds, which is -100 milliseconds. Since this is a negative number, ZigZag encoding is required.

[0053] S3 uses Simple8B to compress the time offset array and the time precision flag array.

[0054] In practical distributed real-time database platforms, in scenarios involving second-level (or periodic) data acquisition and time stamp precision in seconds, the time deviation delta is 0, and the flag is also 0. Two int64 bytes can store the time stamps of more than 200 data points.

[0055] For example, the saved data is tagged with times of 3:00, 3:01, and 3:05. Since the data changes on a second-by-second basis, the third time compression value is 3 seconds, which includes the second-by-second data. Therefore, the delta is 0 and the flag is also 0.

[0056] When the time precision of the data acquisition is in milliseconds, the delta value is basically within hundreds of milliseconds, and an int64 can store at least 7 time tags.

[0057] The time stamp compression method proposed in this invention stores time stamps in units of seconds, milliseconds, or microseconds, depending on the characteristics of the time interval, and also stores a flag indicating the time interval. Based on the characteristics of real-time industrial data, it can automatically adapt time stamps of different precision, providing a better time stamp compression algorithm, significantly reducing disk space usage, and achieving a higher compression ratio.

[0058] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0059] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

Claims

1. A time-stamped compression method for industrial real-time data, characterized in that, include: Store the first time tag corresponding to the first data to be stored in the same data block completely into the cache; The time deviation is obtained based on the nth time tag corresponding to the nth data to be stored in the same data block and the (n-1)th time tag corresponding to the (n-1)th data, and the time precision flag bit corresponding to the nth data is determined based on the time deviation, where n≥2; Based on the time precision flag corresponding to the nth data, the time deviation is converted into the corresponding time compression value; Store the time precision flag and time compression value corresponding to the n data of the same data block into the cache array, and determine whether the time deviation calculation of the industrial real-time data to be stored has been completed. If the time deviation of the industrial real-time data to be stored has not been calculated, then continue to calculate the time deviation of the industrial real-time data; otherwise, compress and store the data in the cache. The time deviation is obtained based on the nth time tag corresponding to the nth data to be stored in the same data block and the (n-1)th time tag corresponding to the (n-1)th data, specifically including: If n=2, then the time deviation of the second data to be stored in the same data block is the difference between the second time tag and the first time tag; If n≥3, then the time deviation of the nth data to be stored in the same data block is the difference between the nth time tag difference and the (n-1)th time tag difference, where the nth time tag difference is the difference between the nth time tag and the (n-1)th time tag. The determination of the time precision flag corresponding to the nth data point based on the time deviation specifically includes: Check sequentially whether the time deviation is 0, whether it includes microseconds, whether it includes milliseconds, and whether it includes seconds. If the time deviation is 0, then the time precision flag is the first flag. If the time deviation includes microseconds, then the time precision flag is the second flag. If the time deviation includes milliseconds, then the time precision flag is the third flag. If the time deviation includes seconds, then the time precision flag is the fourth flag.

2. The time-stamping compression method for industrial real-time data according to claim 1, characterized in that, The step of converting the time deviation into the nth time compression value based on the time precision flag corresponding to the nth data point specifically includes: if the time deviation is the third flag, then the time deviation is converted into milliseconds as the nth time compression value; if the time deviation is the fourth flag, then the time deviation is converted into seconds as the nth time compression value.

3. The time-stamping compression method for industrial real-time data according to claim 1, characterized in that, The step of storing the time precision flag and time compression value corresponding to n data points of the same data block into a cache array, and determining whether the time deviation calculation of the industrial real-time data to be stored has been completed, specifically includes: Store the n time precision flags and the time compression value into the first array and the second array respectively. The system determines whether the calculation of the time deviation of the industrial real-time data to be stored has been completed based on whether n+1 is greater than the number of data points to be stored.

4. The time-stamping compression method for industrial real-time data according to claim 1, characterized in that, The process of compressing and storing data in the cache specifically includes: Each piece of data in the second array is encoded based on ZigZag.

5. The time-stamping compression method for industrial real-time data according to claim 1, characterized in that, The step of compressing and storing data in the cache also includes compressing the encoded second array and the first array based on Simple8B.

6. A data storage system for industrial real-time data, used to run the time-stamp compression method for industrial real-time data as described in any one of claims 1-5, comprising at least a plurality of data blocks for storing industrial real-time data, characterized in that, The first data of any data block in the tag data storage includes at least a first time tag, and the second data includes at least a second time compression value, wherein the second time compression value is the converted value of the difference between the second time tag and the first time tag, and the m-th time tag is the acquisition time value of the m-th industrial real-time data, where m≥1; The p-th data corresponding to any data block includes at least the p-th time compression value, where p ≥ 3. The p-th time compression value is the converted value of the difference between the p-th time tag difference and the (p-1)-th time tag difference, where the p-th time tag difference is the difference between the p-th time tag and the (p-1)-th time tag. The m-th data in any data block of the tag data storage also includes a time precision flag bit corresponding to the m-th time compression value.

7. A data storage system for industrial real-time data according to claim 6, characterized in that, The time precision flag is 2 bits.

Images