Data compression method and device, computer equipment and storage medium

Abstract

The invention relates to a data compression method and device, computer equipment and a storage medium. The method comprises the following steps: receiving a time sequence type data frame; the data type and the timestamp of the received time sequence type data frame and at least one data item in the received time sequence type data frame are determined, and the number of the timestamp of the same time sequence type data frame is one; compressing the timestamp, and recording an obtained timestamp compression value into a timestamp compression sequence corresponding to the data type; and establishing a mapping relationship between the timestamp compression sequence and the data item numerical value compression sequence, compressing the at least one data item to obtain at least one data item numerical value compression value, and recording the at least one data item numerical value compression value into the data item numerical value compression sequence according to the mapping relationship. By adopting the method, the compression calculation amount and the data storage amount of the time sequence type data frame can be reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data compression method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the rapid development of information technology, the amount of data generated and used in daily life has increased dramatically. This data often occupies a large amount of storage space and consumes a significant amount of network resources during transmission. Therefore, data compression technology is particularly important. In certain scenarios, such as mobile devices and embedded devices, storage space remains a bottleneck. Data compression can effectively reduce the storage space occupied by data, allowing more data to be stored within limited storage space. In network communications, data transmission efficiency is directly related to the user experience. Uncompressed data transmission not only consumes a large amount of network bandwidth but also takes a longer time. Therefore, data compression can significantly reduce the size of data transmission, thereby improving network transmission efficiency.

[0003] Time series data frames are particularly massive and require high real-time performance. Currently, compression of time series data frames typically utilizes one or more of the following: static compression, differential coding, block coding, time series model compression, feature extraction and dimensionality reduction, and data stream processing and real-time compression. However, these compression methods suffer from data loss, high computational and storage resource overhead, or low timeliness. Summary of the Invention

[0004] Based on this, it is necessary to provide a data compression method, device, computer equipment and storage medium that can reduce the computational complexity of time-series data frames while taking into account timeliness and data accuracy to address at least one of the above technical issues.

[0005] A data compression method, comprising:

[0006] Receive time-series data frames;

[0007] Determining a data type and a timestamp of a received time series data frame and at least one data item in the received time series data frame, wherein the number of timestamps of the same time series data frame is one;

[0008] Compress the timestamp and record the obtained timestamp compression value into the timestamp compression sequence corresponding to the data type;

[0009] Establishing a mapping relationship between a timestamp compression sequence and a data item value compression sequence;

[0010] compressing at least one data item to obtain a numerical compression value of at least one data item;

[0011] At least one data item numerical compression value is recorded into the data item numerical compression sequence according to the mapping relationship.

[0012] In one embodiment, compressing the timestamp and recording the obtained timestamp compressed value into a timestamp compressed sequence corresponding to the data type includes:

[0013] Determine whether the timestamp is the starting timestamp of the time series data frame of the data type;

[0014] If it is a start timestamp, obtain the first control code corresponding to the start timestamp;

[0015] Concatenate the timestamp and the first control code to obtain a first timestamp compression value;

[0016] Recording the first timestamp compression value into the timestamp compression sequence;

[0017] If it is not the starting timestamp, the difference between the current timestamp and the previous timestamp of the time series data frame of the data type is calculated;

[0018] Calculate the target timestamp difference based on the current timestamp difference and the previous timestamp difference, where the previous timestamp difference is the difference between the previous timestamp and the timestamp of the previous data frame of the time series data frame of the data type;

[0019] Obtain a second control code corresponding to the target timestamp difference;

[0020] Concatenate the target timestamp difference and the second control code to obtain a second timestamp compression value;

[0021] The second timestamp compressed value is recorded in the timestamp compressed sequence.

[0022] In one embodiment, the method further includes:

[0023] Determine the first number of bits required for the target timestamp difference;

[0024] determining a second number of bits occupied by the second control code;

[0025] Calculate the total number of bits corresponding to the second timestamp compression value according to the first number of bits and the second number of bits;

[0026] Record the total number of bits into the timestamp compressed sequence.

[0027] In one embodiment, the method further includes:

[0028] Segment-compressing the timestamp compression sequence to obtain multiple timestamp segment-compressed data;

[0029] According to the mapping relationship, obtain the data item numerical segmented data corresponding to each timestamp segmented data;

[0030] The data item numerical segmented data is compressed to obtain the data item numerical segmented compressed data.

[0031] In one embodiment, the timestamp compression sequence is segmented and compressed to obtain a plurality of timestamp segmented compressed data, including:

[0032] Get the preset segment timestamp difference;

[0033] The current timestamp difference is calculated based on the timestamp and the forward timestamp. The forward timestamp is the previous timestamp corresponding to the timestamp in the timestamp compression sequence.

[0034] If the current timestamp difference is greater than the preset segment timestamp difference, the timestamp is divided into new segments to obtain the current timestamp segment;

[0035] Set the segment splicing code corresponding to the current timestamp segment;

[0036] Get the total number of bits of the fragment corresponding to the current timestamp fragment, which includes the number of bits occupied by the segment splicing code and the number of bits occupied by the timestamp compression value;

[0037] Comparing whether the total number of bits of the segment is greater than the preset number of bits of the segment;

[0038] If it is greater, the current timestamp segment is segment compressed to obtain timestamp segment compressed data.

[0039] In one embodiment, segment compression is performed on the current timestamp segment to obtain timestamp segment compressed data, including:

[0040] Compress the current timestamp segment by bytes to obtain the number of bytes in the timestamp segment;

[0041] Get the segment compression marker code corresponding to the current timestamp segment;

[0042] Determine the length of the segment corresponding to the current stamp segment according to the number of bytes in the timestamp segment;

[0043] The timestamp segmented compressed data corresponding to the current timestamp segment is obtained by splicing the segment compression mark code, segment length and number of bytes of the timestamp segment.

[0044] In one embodiment, the method further comprises:

[0045] When the total number of bits in the segment is less than or equal to the preset number of segment bits, there is no need to perform segment compression on the current timestamp segment;

[0046] Get the fragment marker code corresponding to the current timestamp fragment;

[0047] The timestamp segment data corresponding to the current timestamp segment is obtained by splicing the timestamp compression value and the segment marker code.

[0048] In one embodiment, the method further includes:

[0049] Obtain the segment index information corresponding to each timestamp segment data, the segment index information includes the start timestamp, start position, segment data length and validity;

[0050] Generate a segment index table corresponding to the timestamp compression sequence according to the segment index information.

[0051] In one embodiment, compressing the data item numerical segment data to obtain the data item numerical segment compressed data includes:

[0052] Get the data item value type corresponding to the data item value segmentation data;

[0053] Get the data item numerical pattern corresponding to the data item numerical segmentation data;

[0054] Determine a data item compression algorithm for matching data item value segment data based on the data item value type and data item value regularity;

[0055] The data item numerical segmented data is compressed according to a data item compression algorithm to obtain the data item numerical segmented compressed data.

[0056] A data compression device, comprising:

[0057] A receiving module, used for receiving time-series data frames;

[0058] a processing module, configured to determine a data type and a timestamp of a received time series data frame and at least one data item in the received time series data frame, wherein the number of timestamps of the same time series data frame is one;

[0059] A first compression module is used to compress the timestamp and record the obtained timestamp compression value into the timestamp compression sequence corresponding to the data type;

[0060] The second compression module is used to establish a mapping relationship between the timestamp compression sequence and the data item numerical compression sequence, compress at least one data item, obtain at least one data item numerical compression value, and record the at least one data item numerical compression value into the data item numerical compression sequence according to the mapping relationship.

[0061] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0062] Receive time-series data frames;

[0063] Determining a data type and a timestamp of a received time series data frame and at least one data item in the received time series data frame, wherein the number of timestamps of the same time series data frame is one;

[0064] Compress the timestamp and record the obtained timestamp compression value into the timestamp compression sequence corresponding to the data type;

[0065] A mapping relationship between a timestamp compression sequence and a data item numerical compression sequence is established, at least one data item is compressed to obtain at least one data item numerical compression value, and at least one data item numerical compression value is recorded in the data item numerical compression sequence according to the mapping relationship.

[0066] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0067] Receive time-series data frames;

[0068] Determining a data type and a timestamp of a received time series data frame and at least one data item in the received time series data frame, wherein the number of timestamps of the same time series data frame is one;

[0069] Compress the timestamp and record the obtained timestamp compression value into the timestamp compression sequence corresponding to the data type;

[0070] A mapping relationship between a timestamp compression sequence and a data item numerical compression sequence is established, at least one data item is compressed to obtain at least one data item numerical compression value, and at least one data item numerical compression value is recorded in the data item numerical compression sequence according to the mapping relationship.

[0071] The above-mentioned data compression method, apparatus, computer equipment and storage medium receive a time-series data frame, determine the data type, timestamp and at least one data item in the received time-series data frame, wherein the number of timestamps of the same time-series data frame is one, compress the timestamp, record the obtained timestamp compression value in the timestamp compression sequence corresponding to the data type, establish a mapping relationship between the timestamp compression sequence and the data item numerical compression sequence, compress at least one data item, obtain at least one data item numerical compression value, and record the at least one data item numerical compression value in the data item numerical compression sequence according to the mapping relationship. By separating the timestamps and data items of time-series data frames of the same data type and compressing them, multiple data items under the same time-series data frame can share the same timestamp compression sequence, which can reduce unnecessary computing and storage resource overhead. Moreover, the data items are compressed after the data items of the time-series data frame are determined. In this way, redundant information in the data, such as repeated data items or fixed data patterns, can be identified and removed. During the compression process, the characteristics of the data can be more effectively utilized to achieve higher compression efficiency to ensure timeliness without causing loss to the data item pairs, and the data accuracy can also be guaranteed not to be affected. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 1 is a flow chart of a data compression method according to an embodiment;

[0073] Figure 2 1 is a flow chart of a timestamp compression step in one embodiment;

[0074] Figure 3 1 is a flow chart of a timestamp compression step in one embodiment;

[0075] Figure 4 1 is a flow chart of a data compression method according to an embodiment;

[0076] Figure 5 1. A schematic flow chart of a time stamp compression sequence segment compression step in one embodiment;

[0077] Figure 6 1 is a flow chart of a current timestamp segment compression step in one embodiment;

[0078] Figure 7 1 is a flow chart of a current timestamp segment compression step in one embodiment;

[0079] Figure 8 1. A schematic flow chart of a time stamp compression sequence segment compression step in one embodiment;

[0080] Figure 91 is a flow chart of a data item numerical value segmentation data compression step in one embodiment;

[0081] Figure 10 is a structural block diagram of a data compression device in one embodiment;

[0082] Figure 11 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0084] In some exemplary embodiments, a data compression method is provided, which can be applied to a computer device, which can be various types of terminals or servers, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device (such as a smart speaker), smart home appliance (such as a smart TV), smart watch, vehicle terminal, wearable device, virtual reality (VR) device, etc., but is not limited thereto.

[0085] like Figure 1 As shown, the data compression method includes the following steps:

[0086] Step 102: Receive a time-series data frame.

[0087] Among them, the time-series data frame is a data frame identified by time reference. The data frame is the basic unit in network communication. It contains the address information of the sender and receiver, control information and data to be transmitted. The size of the data frame is usually determined by the network equipment to ensure that no errors will occur during the data transmission process.

[0088] Step 104 : Determine the data type and timestamp of the received time series data frame and at least one data item in the received time series data frame, wherein the number of timestamps of the same time series data frame is one.

[0089] A time series data frame records the corresponding timestamp and includes at least one data item and a data frame identifier. The data frame identifier is used to distinguish the data items contained in the time series data frame and is also used to serve as or determine the data type of the time series data frame. The data item is used to describe data-related information and may include data item type, data length, data content, etc. The data type corresponding to the time series data frame can also be determined by the data item type. Time series data frames of the same type have the same data item type.

[0090] For example, time series data frame A includes two data items, namely speed and torque, and time series data frame B includes two data items, namely speed and window amplitude. Since the types of data items included in time series data frame A and time series data frame B are different, time series data frame A and time series data frame B belong to time series data frames of different data types.

[0091] Among them, at least one data item under the same frame is connected to the data compression device at the same time as a frame of data, the timestamps of at least one data item under the same frame are the same, the number of timestamps of the same time-series data frame is one, and the number of data items of the same time-series data frame is at least one.

[0092] Specifically, the received time series data frame is parsed, and specifically, the data items of the time series data frame are separated to obtain at least one independent data item.

[0093] Furthermore, after obtaining at least one data item, if the time series data frame itself contains a timestamp, the corresponding timestamp of the time series data frame is obtained. A timestamp is a specific data format that is typically used to indicate the precise time of an event or data at a specific point in time. It typically contains date and time information, and the accuracy of the timestamp is crucial for ensuring data correctness and consistency. The timestamp of the time series data frame can be the sending time or the receiving time, which can be determined based on actual circumstances.

[0094] Among them, if the timestamp of the time series data frame does not pass in a valid data frame timestamp, such as 0 or a negative value, the current time is taken as the timestamp corresponding to the time series data frame, that is, the current time is taken as the timestamp corresponding to each data item, where the current time can be the current time of the server.

[0095] Step 106: compress the timestamp, and record the obtained timestamp compression value into the timestamp compression sequence corresponding to the data type.

[0096] Compression is a mechanism that uses a specific compression algorithm to reduce computational complexity. After obtaining the timestamp corresponding to a time-series data frame, the timestamp is compressed to produce a compressed timestamp sequence. Timestamp compression utilizes a second-order differential coding algorithm based on the periodic nature of time-series data frames. Unlike previous techniques, which directly store the timestamp in its original format, timestamps are compressed to produce a compressed timestamp value, which forms a compressed timestamp sequence. This compression reduces the number of bits in the compressed timestamp value compared to the original timestamp format, requiring less computation and storage resources.

[0097] Specifically, after receiving a time series data frame, the timestamp corresponding to the received time series data frame is determined, and the timestamp of the time series data frame is compressed using a timestamp compression algorithm to obtain a timestamp compression value corresponding to the timestamp. Furthermore, a timestamp compression sequence of the same data type as the received time series data frame is obtained, and the timestamp compression value is recorded in the timestamp compression sequence.

[0098] Among them, a timestamp compression sequence can be created in advance for each data type of time series data frame based on actual business conditions and actual application scenarios. The timestamp compression sequence is used to compress and record the timestamps of the time series data frames. A timestamp compression sequence is created for each data type of time series data frame, and the timestamps of the same type of time series data frames are compressed into the same timestamp compression sequence.

[0099] Step 108: Establish a mapping relationship between the timestamp compression sequence and the data item numerical compression sequence, compress at least one data item, obtain at least one data item numerical compression value, and record the at least one data item numerical compression value into the data item numerical compression sequence according to the mapping relationship.

[0100] At least one data item is independently compressed simultaneously with, before, or after the timestamp is compressed. That is, each independent data item is independently compressed, and different compression algorithms may be selected based on the data item to perform numerical compression of the data item, obtaining a numerical compression value corresponding to each data item, and recording each numerical compression value of the data item in a numerical compression sequence of data items that is mapped to the timestamp compression sequence. Real-time compression is performed immediately after the received data frame is parsed into data items.

[0101] Among them, a timestamp compression sequence and a data item value compression sequence can be created in advance for each data type of time series data frame. The data item value compression sequence is used to compress and record the values ​​of the data items in the time series data frame. For the time series data frame of the same data type, one or more data item value compression sequences are created, and the specific number is the same as the number of data items in the time series data frame of the data type.

[0102] Furthermore, a mapping relationship is established between each data item's numerical compression sequence and its timestamp compression sequence. Through this mapping relationship, each data item's numerical compression sequence in a time-series data frame of the same data type is mapped to a unique timestamp compression sequence in that data type's time-series data frame. This mapping relationship allows multiple data item numerical compression sequences to share the same timestamp compression sequence, thereby reducing unnecessary timestamp compression and storage overhead.

[0103] In the above-mentioned data compression method, a time series data frame is received, the data type, timestamp and at least one data item in the received time series data frame are determined, wherein the number of timestamps in the same time series data frame is one, the timestamp is compressed, and the obtained timestamp compression value is recorded in a timestamp compression sequence corresponding to the data type, and at least one data item is compressed, and the obtained at least one data item numerical compression value is recorded in a data item numerical compression sequence that has a mapping relationship with the timestamp compression sequence. By separating the timestamps and data items of the time series data frame of the same data type and compressing them separately, multiple data items under the same time series data frame can share the same timestamp compression sequence, which can reduce unnecessary computing and storage resource overhead. Moreover, the data items are compressed after the data items of the time series data frame are determined. In this way, redundant information in the data, such as repeated data items or fixed data patterns, can be identified and removed. In the compression process, the characteristics of the data can be more effectively utilized to achieve higher compression efficiency to ensure timeliness, without causing loss to the data item pairs, and also ensuring that the data accuracy is not affected.

[0104] In some exemplary embodiments, Figure 2 As shown, the timestamp is compressed, and the obtained timestamp compression value is recorded in the timestamp compression sequence corresponding to the data type, including:

[0105] Step 202: Determine whether the timestamp is the start timestamp of a time series data frame of the data type.

[0106] Step 204: If it is a start timestamp, obtain a first control code corresponding to the start timestamp.

[0107] Step 206: Concatenate the timestamp and the first control code to obtain a first timestamp compression value.

[0108] Step 208: Record the first timestamp compression value into the timestamp compression sequence.

[0109] Among them, the starting timestamp is the first timestamp of the time-series data frame of the same data type, which can be understood as the first timestamp in the timestamp compression sequence of the data type. Therefore, after obtaining the timestamp, it is necessary to determine whether the timestamp is the first timestamp of the time-series data frame of the same data type. If it is the first timestamp, there is no need to perform any compression processing on the timestamp, keep the original format of the timestamp, and obtain the first control code corresponding to the starting timestamp. The first control code here is used to mark that the timestamp is the first timestamp of the same type of time-series timestamp. The first control code can be used to determine that the timestamp is the starting timestamp. Furthermore, the first control code is spliced ​​with the timestamp to obtain the timestamp compression value corresponding to the timestamp.

[0110] Step 210: If it is not the starting timestamp, the difference between the current timestamp and the previous timestamp of the time series data frame of the data type is calculated.

[0111] Step 212 , calculate a target timestamp difference based on the current timestamp difference and the previous timestamp difference, where the previous timestamp difference is the difference between the previous timestamp and the timestamp of the previous data frame of the time series data frame of the data type.

[0112] Step 214: Obtain a second control code corresponding to the target timestamp difference.

[0113] Step 216: Concatenate the target timestamp difference and the second control code to obtain a second timestamp compression value.

[0114] Step 218: Record the second timestamp compression value into the timestamp compression sequence.

[0115] Among them, after obtaining the timestamp, it is necessary to determine whether the timestamp is the first timestamp of the time series data frame of the same data type. If not, obtain the timestamp and the previous timestamp of the time series data frame of the same data type, calculate the difference between the timestamp and the previous timestamp, and obtain the current timestamp difference.

[0116] Furthermore, the difference between the previous timestamp and the previous timestamp of the same type of time series data frame is obtained to obtain the previous timestamp difference, and the difference between the current timestamp difference and the previous timestamp difference is calculated to obtain the target timestamp difference. It can be understood that the target timestamp difference is the difference between the current timestamp difference and the previous timestamp difference.

[0117] For example, the previous timestamp difference in the timestamp compression sequence is pre_tm_stamp_delta, the previous timestamp is pre_tm_stamp, the incoming timestamp is cur_tm_stamp, the difference with the current timestamp pre_tm_stamp is calculated as stamp_delta = cur_tm_stamp - pre_tm_stamp, and the target timestamp difference between the current timestamp difference and the previous timestamp difference is calculated as tm_stamp_delta_delta = stamp_delta - pre_tm_stamp_delta.

[0118] Among them, the second control code here is a marking code used to mark the range of the target timestamp difference. The second control code can be determined according to actual business needs, actual product needs or actual application scenarios. For example, 0, 2 (binary 10), 6 (binary 110), 14 (binary 1110), 62 (binary 111110) can be set, and the second control code 30 (binary 11110) is set to mark the first element of the timestamp compression sequence.

[0119] Furthermore, after obtaining the target timestamp difference and the corresponding second control code, the target timestamp difference and the second control code are concatenated to obtain the timestamp compressed value corresponding to the timestamp, which is recorded in the timestamp compression sequence. As can be seen, the number of bits in the compressed timestamp value after compression is significantly smaller than the number of bits in the uncompressed timestamp, thereby reducing the computational complexity of the timestamp. For example, if the target timestamp difference is 1 and the second control code is 10, the concatenated compressed timestamp value is 101.

[0120] In some exemplary embodiments, Figure 3 As shown, the method further includes:

[0121] Step 302: Determine a first number of bits required for the target timestamp difference.

[0122] Step 304: Determine the second number of bits occupied by the second control code.

[0123] Step 306: Calculate the total number of bits corresponding to the compressed timestamp value based on the first number of bits and the second number of bits.

[0124] Step 308: Record the total number of bits into the timestamp compression sequence.

[0125] Among them, the first bit number is the number of bits of the target timestamp difference, and the second bit number is the number of bits of the first control code. After obtaining the timestamp compression value, the total number of bits of the timestamp compression value needs to be recorded in the timestamp compression sequence.

[0126] Specifically, the number of bits of the target timestamp difference is counted to obtain a first number of bits, and the number of bits of the first control code is counted to obtain a second number of bits. Furthermore, since the timestamp compression value is obtained by concatenating the target timestamp difference and the second control code, the first number of bits and the second number of bits are counted to calculate the total number of bits corresponding to the timestamp compression value, and finally the counted total number of bits is recorded in the timestamp compression sequence. For example, if the second control code is 10, the second number of bits is 2, the target timestamp difference is 1, and the first number of bits is 1, then the total number of bits of the timestamp compression value 101 is 3 bits.

[0127] In some exemplary embodiments, Figure 4 As shown, the above method also includes:

[0128] Step 402: compress the timestamp compression sequence in sections to obtain a plurality of timestamp segment compression data.

[0129] Step 404: Obtain the data item value segmented data corresponding to each timestamp segmented data according to the mapping relationship.

[0130] Step 406: compress the data item numerical segment data to obtain data item numerical segment compressed data.

[0131] Among them, because through the mapping relationship, each data item numerical compression sequence in the data frame of the same data type is mapped to a unique timestamp compression sequence of the data frame of this data type, the effect of multiple data item numerical compression sequences sharing the same timestamp compression sequence is achieved. Therefore, when the timestamp compression sequence is segmented, the data item data compression sequence can be segmented at the same time.

[0132] Specifically, the timestamp compression sequence can be segmented and compressed according to the segment size to obtain multiple timestamp segmented compressed data. Based on the mapping relationship, the data item numerical segmented data corresponding to each timestamp segmented compressed data is obtained, thereby achieving segmentation of the data item numerical compressed sequence. Further, each data item numerical segmented data is compressed. The specific compression algorithm can be determined based on the data content in the data item numerical segmented data. Finally, the data item numerical segmented compressed data corresponding to each compressed data item numerical segmented data is obtained.

[0133] In some exemplary embodiments, Figure 5 As shown, the timestamp compression sequence is segmented and compressed to obtain multiple timestamp segmented compressed data, including:

[0134] Step 502: Obtain a preset segment timestamp difference.

[0135] Step 504 : Calculate the current timestamp difference based on the timestamp and the forward timestamp, where the forward timestamp is the previous timestamp corresponding to the timestamp in the timestamp compression sequence.

[0136] Step 506: If the current timestamp difference is greater than the preset segment timestamp difference, the timestamp is divided into new segments to obtain the current timestamp segment.

[0137] Step 508: Set the segment splicing code corresponding to the current timestamp segment.

[0138] The preset segment timestamp difference is the segment size of the timestamp compression sequence, which can be a time unit, such as 3000 milliseconds, or can be determined based on actual business needs, product requirements, or application scenarios. The previous timestamp is the previous timestamp in the timestamp compression sequence. The difference between the incoming timestamp and the previous timestamp is calculated to obtain the current timestamp difference.

[0139] Furthermore, it is determined whether the current timestamp difference is greater than the preset segment timestamp difference. If so, it indicates that the incoming timestamp can be divided into a new time period, obtaining the current timestamp segment. To mark the current timestamp segment, different timestamp segments correspond to different segment splicing codes. Therefore, the segment splicing code corresponding to the current timestamp segment can be set based on actual business needs, actual product needs, or actual application scenarios. For example, the segment marking code can be set to 126, and the corresponding binary code is 1111110.

[0140] It should be noted that each timestamp segment starts with an independent byte, that is, the content of the same byte is not stored in different timestamp segments, so that data in the timestamp segment can be queried and copied.

[0141] Step 510: Obtain the total number of bits of the segment corresponding to the current timestamp segment. The total number of bits of the segment includes the number of bits occupied by the segment splicing code and the number of bits occupied by the timestamp compression value.

[0142] Step 512: Compare whether the total number of bits of the segment is greater than the preset number of bits of the segment.

[0143] Step 514: If it is greater than, perform segment compression on the current timestamp segment to obtain timestamp segment compressed data.

[0144] Among them, the total number of fragment bits here is the number of bits corresponding to the current timestamp fragment. Since the current timestamp fragment includes the segmented splicing code and the timestamp compression value, the number of bits of the segmented splicing code and the number of bits of the timestamp compression value are counted respectively to obtain the total number of fragment bits corresponding to the current timestamp fragment.

[0145] If the length of the current timestamp segment exceeds a preset threshold, segment compression is performed on the current timestamp segment. Specifically, if the total number of bits in the current timestamp segment exceeds a preset threshold, segment compression is performed again. Specifically, a preset segment bit number is obtained, which can be determined based on actual business needs, product requirements, or application scenarios, such as 800 bits.

[0146] Furthermore, the total number of bits of the current timestamp segment is compared to see if it is greater than the preset number of segment bits. If so, it indicates that the current timestamp segment needs to be segmented and compressed, and the timestamp segment compressed data corresponding to the current timestamp segment is obtained. Conversely, if it is less than or equal to the preset number of segments, it indicates that the current timestamp segment does not need to be segmented and compressed, and no processing is required for the current timestamp segment, thus avoiding unnecessary resource consumption.

[0147] In some exemplary embodiments, Figure 6 As shown, the current timestamp segment is segment compressed to obtain timestamp segment compressed data, including:

[0148] Step 602: perform byte compression on the current timestamp segment to obtain the number of bytes of the timestamp segment.

[0149] Step 604: Obtain the segment compression mark code corresponding to the current timestamp segment.

[0150] Step 606: Determine the segment length corresponding to the current stamp segment according to the number of bytes in the timestamp segment.

[0151] Step 608: splice the timestamp segmented compressed data corresponding to the current timestamp segment according to the segment compression mark code, the segment length and the number of bytes of the timestamp segment to obtain the timestamp segmented compressed data.

[0152] Among them, the current timestamp fragment is segment compressed to obtain timestamp segment compressed data, and the timestamp segment compressed data includes a segment compression mark code, a segment length and the number of bytes of the timestamp fragment. Among them, the segment compression mark code is used to mark that the timestamp segment compressed data is obtained by re-segmenting the current timestamp fragment, and the segment length is the length of the current timestamp fragment, and the number of bytes of the timestamp fragment is the storage capacity and transmission capacity of the timestamp fragment.

[0153] Specifically, after obtaining the current timestamp fragment, the number of bits of the current timestamp fragment is byte compressed to obtain the number of bytes of the timestamp fragment, and at the same time, the fragment compression marker code corresponding to the current timestamp fragment is obtained. The fragment compression marker code here can be determined based on actual business needs, actual product needs or actual application scenarios. For example, the fragment compression marker code 254 is set, binary 11111110, to mark that the timestamp segmented compressed data is timestamp compressed sequence data that has been compressed again.

[0154] Furthermore, it is also necessary to determine the fragment length corresponding to the current timestamp fragment based on the number of timestamp fragment bytes. Specifically, the length of the timestamp fragment bytes can be calculated to obtain the fragment length. For example, if the number of timestamp fragment bytes is 8, then the fragment length is 8.

[0155] Finally, the segment compression marker code, segment length, and timestamp segment byte count are concatenated to obtain the timestamp segment compressed data corresponding to the current timestamp segment. For example, if the segment compression marker code is set to 126, the corresponding binary value is 1111110, the segment length is 3, and the timestamp segment byte count is 3, the timestamp segment compressed data obtained by concatenation is 111111033.

[0156] In some exemplary embodiments, Figure 7 As shown, the method further includes:

[0157] Step 702: When the total number of bits of the segment is less than or equal to the preset number of segment bits, there is no need to perform segment compression on the current timestamp segment.

[0158] Step 704: Obtain the segment marking code corresponding to the current timestamp segment.

[0159] Step 706: splice the timestamp segment data corresponding to the current timestamp segment according to the timestamp compression value and the segment marker code.

[0160] Among them, if the total number of bits of the current timestamp fragment is less than or equal to the preset segmentation bit number, it means that the current timestamp fragment does not meet the segmentation requirements. In order to avoid wasting unnecessary resources, there is no need to segment the current timestamp fragment, but it is necessary to mark the current timestamp fragment as a fragment that has not been re-segmented and is an ordinary segmented data, which is marked by a fragment marking code.

[0161] The fragment marker code is used to mark the current timestamp fragment as uncompressed fragment data. It is a normal segmented data. The specific value can be determined based on actual business needs, product requirements, or application scenarios. For example, 30 (binary 11110) can be set as the fragment marker code. Finally, the timestamp compression value and the fragment marker code are concatenated to obtain the timestamp segment data corresponding to the current timestamp fragment.

[0162] In some exemplary embodiments, Figure 8 As shown, the method further includes:

[0163] Step 802: Obtain segment index information corresponding to each timestamp segment data. The segment index information includes a start timestamp, a start position, a segment data length, and validity.

[0164] Step 804: Generate a segment index table corresponding to the timestamp compression sequence according to the segment index information.

[0165] In order to facilitate the management of each timestamp segmented data and quickly locate and query the data item value of a specified time period, the segment information of each timestamp segmented data can be recorded in a segment index table.

[0166] Among them, the segmentation index information describes the segmentation information of the timestamp segmented data, and the segmentation index information includes the starting timestamp, starting position, segmentation data length and validity of the timestamp segmented data, wherein the validity indicates whether the timestamp segmented data has received valid data, the starting timestamp is the first timestamp of the timestamp segmented data, and the starting position is the offset of the timestamp segmented data relative to the starting bit of the timestamp compression sequence data.

[0167] Specifically, the starting timestamp, starting position, segmented data and validity corresponding to the current timestamp segmented data are obtained to form segmented index information of the current timestamp segmented data, wherein the current timestamp segmented data is any timestamp segmented data among the timestamp segmented data.

[0168] Furthermore, the segment index information of each timestamp segment data is used to generate a segment index table corresponding to the timestamp compression sequence. The segment index table here can be a stack of arrays, and each element records a segment index information.

[0169] Among them, each time a new segment is added, the content of the corresponding element in the segment index table is updated, its validity is modified to valid, the starting timestamp is the newly added timestamp, and the starting position is the offset of the current new segment relative to the starting bit of the compressed sequence data of the timestamp. When a segment compression is completed, the content of the corresponding element in the segment index table is updated, mainly updating the data length content of this segment.

[0170] In some exemplary embodiments, Figure 9 As shown, the data item numerical segmented data is compressed to obtain the data item numerical segmented compressed data, including:

[0171] Step 902: Obtain the data item value type corresponding to the data item value segment data.

[0172] Step 904: Obtain the data item numerical pattern corresponding to the data item numerical segment data.

[0173] Step 906: Determine a data item compression algorithm for matching data item value segment data according to the data item value type and data item value regularity.

[0174] Step 908: compress the data item numerical segmented data according to the data item compression algorithm to obtain the data item numerical segmented compressed data.

[0175] Among them, after the timestamp compression sequence is segmented and compressed, multiple timestamp segmented compressed data are obtained. Through the mapping relationship, the data item numerical segmented data corresponding to each timestamp segmented compressed data is obtained to achieve segmentation of the numerical compression sequence of each data item.

[0176] Furthermore, after the segmentation, the numerical segmented data of each data item is compressed, and a suitable compression algorithm is selected to compress the matching numerical segmented data of the data item to obtain the numerical segmented compressed data of the data item.

[0177] Among them, the selection of compression algorithm can be specifically selected according to the numerical type of the data item and the numerical rule of the data item. Specifically, it can be to obtain the data item numerical type and the numerical rule of the data item corresponding to the data item numerical segmentation data, determine the matching data item compression algorithm according to the data item numerical type and the data item numerical rule, and finally, compress the data item numerical segmentation data through the data item compression algorithm to obtain the data item numerical segmentation compressed data.

[0178] For example, integer data items can choose Delta differential coding, Delta-of-Delta second-order differential coding, Zig-zag coding and other algorithms, and floating-point data can choose Gorilla algorithm and other algorithms.

[0179] The compression algorithm may be stored at the beginning of the data item numerical compression sequence (using 8 bits to enumerate and encode different compression types for storage), so that corresponding decompression may be performed according to different compression types during decompression.

[0180] Among them, a one-to-one correspondence is maintained between the record sequence of each data item in the same data frame and the timestamp compression record sequence of this data frame. For example, a data frame of type A contains 10 data items, which will be compressed and stored using a timestamp compression sequence and 10 numerical compression sequences. When a data frame of type A is connected to the compression system, a timestamp compression record will be generated and stored in the timestamp compression sequence. At the same time, numerical compression records of 10 data items will be generated and stored in the corresponding 10 numerical compression sequences. When another data frame of type A is connected, another timestamp compression record will be generated and stored in the timestamp compression sequence, and numerical compression records of 10 data items will be generated and stored in the corresponding 10 numerical compression sequences. Each data item numerical compression sequence sets a corresponding variable to record information such as the effective length of each data item numerical compression sequence.

[0181] In some exemplary embodiments, segmented compression may be performed on a compressed sequence of data item values. When a new segment (except the first segment) is generated in a data frame timestamp sequence, segmented compression may be performed on the data item data value sequence in the same data frame in a manner similar to timestamp segmented compression. Specifically, when a new segment (except the first segment) is generated in a data frame timestamp sequence, corresponding segmentation marking information may be added to the data item data value sequence in the same data frame in a manner similar to timestamp segmented compression and a manner similar to adding segmentation marking information to timestamps.

[0182] Among them, in order to facilitate the management of the segmented and compressed data of each data item, the segmentation index information corresponding to the data item numerical compression sequence can be set, and a segmentation index information table is set for each data item numerical compression sequence to record the data item numerical compression segmentation information. When the segmentation index table is assigned to the timestamp sequence, the segmentation index table of the same size is assigned to each data item numerical sequence under the same type of data frame. Specifically, the segmentation information in the data item numerical compression sequence can be recorded in this segmentation index information table in the same way as the timestamp compression sequence segmentation index table.

[0183] In some exemplary embodiments, a compressed data organization module may be used to cache compressed sequences and provide external data query and decompression interfaces, specifically including the following:

[0184] a) collecting timestamp segmentation information and data item value segmentation information during compression;

[0185] b) caching the segment information, timestamp compression sequence, and data item number compression sequence;

[0186] c) Provide external data query interface and data query function by time period and data item;

[0187] d) Provide data decompression function to external parties:

[0188] i. Decompress the timestamp compressed sequence using Delta-of-Delta second-order difference decoding;

[0189] ii. According to the compression algorithm type of the data item numerical compression sequence header, select the corresponding decompression algorithm to perform data item numerical decompression.

[0190] In some exemplary embodiments, the data compression method is described in detail by the following content, which specifically includes the following content:

[0191] 1. Write data frame structure definition

[0192] a) Describe the identifier of each data frame

[0193] b) Describe the data items that each data frame consists of. The data item information mainly includes:

[0194] i. Type

[0195] ii. Length

[0196] iii. Offset position relative to the frame header

[0197] iv. Potential change patterns of data items (in order to select an appropriate compression algorithm), such as range, mean, standard deviation, and mode of change, such as increasing or decreasing.

[0198] c) Describe the potential change patterns of data frames (in order to select an appropriate compression algorithm), such as: generation based on periodicity, generation based on uncertain events.

[0199] 2. Use the software interface provided by this compression algorithm to access the time series data frame

[0200] a) Agreed data frame access interface parameters, including data frame identifier (used as a basis for subsequent analysis of the data items contained in the frame), data frame data content, data frame data length, and data frame timestamp (optional. If not filled in, the timestamp at the time of access will be used as the value when the data is accessed by the compression algorithm).

[0201] b) Parse the data frame into a set of data items according to the data frame structure definition. If no valid data frame timestamp is passed in (such as 0 or a negative value), use the current time to set the timestamp information for each data item. Otherwise, use the passed data frame timestamp as the timestamp of each data item.

[0202] 3. For each type of data frame, create a timestamp compressed sequence object and a data item value compressed sequence object

[0203] i. Timestamp compression object, used to perform data frame timestamp compression and record auxiliary information, such as the start time of the compressed sequence, data length, segmentation information (the concept of segmentation will be introduced later), etc.

[0204] ii. A type of data frame, creating a timestamp compressed sequence object

[0205] iii. Data item numerical compression object, used to perform numerical compression of data frame data items and record auxiliary information, such as compressed sequence data length, segmentation information (the concept of segmentation will be introduced later), etc.

[0206] iv. A type of data frame creates one or more data item value compressed sequence objects, the specific number of which is the same as the number of data items under the data frame

[0207] v. Set whether to generate segmentation mark new_segment_flag for each type of data frame, and set the default value to

[0208] false, if a new segment is generated when timestamp compression is performed for the data frame later, this flag will be set to true. When performing value compression for each data item in the data frame, this flag will be referenced to decide whether to add a segment marker to the value compression sequence

[0209] 4. Establish the mapping relationship between data item value compression sequence object and timestamp compression sequence object

[0210] i. Multiple data items in the same frame are connected to the compression system at the same time as a frame of data.

[0211] ii. A data frame can be decomposed into multiple data items, but only one timestamp needs to be recorded

[0212] iii. Create a mapping table, where each data item value compression sequence object of the same type of data frame is mapped to a unique timestamp compression object of this type of data frame

[0213] iv. Through this mapping relationship, multiple data item numerical compression sequences can share the same timestamp compression object, thereby reducing unnecessary timestamp compression and storage overhead.

[0214] 5. Use the Delta-of-Delta second-order differential coding (hereinafter referred to as differential compression) algorithm to compress the timestamp sequence of the data frame

[0215] a) Pass the timestamp of each type of data frame into the timestamp compression module

[0216] b) Record the starting timestamp start_tm_stamp of each type of data frame compression sequence. If the timestamp added to a type of data frame compression sequence is the first timestamp of the current type of data frame, assign this timestamp to start_tm_stamp

[0217] c) Record the segment start timestamp segment_start_tm_stamp of each type of data frame compression sequence. If the timestamp added to a type of data frame compression sequence is the first timestamp of the current type of data frame, assign this timestamp to segment_start_tm_stamp

[0218] d) Update the segment flag new_segment_flag to false

[0219] e) Record the segment starting valid bit number segment_start_pos of each type of data frame compression sequence. If the timestamp added to a type of data frame compression sequence is the first timestamp of the current type of data frame, then assign segment_start_pos to 0

[0220] f) Calculate the timestamp difference value tm_stamp_delta_delta of the incoming timestamp in the timestamp compression module:

[0221] i. Use variables to record the difference between the two adjacent timestamps in the current segment (see the segmentation description in the subsequent steps) pre_tm_stamp_delta and the previous timestamp value pre_tm_stamp

[0222] ii. Initially set the timestamp difference pre_tm_stamp_delta to 0 and the last timestamp value

[0223] pre_tm_stamp is 0

[0224] iii. When a new timestamp cur_tm_stamp is passed in, calculate the difference between it and pre_tm_stamp

[0225] stamp_delta=cur_tm_stamp-pre_tm_stamp

[0226] iv. Calculate the difference between two adjacent timestamps in the current segment tm_stamp_delta_delta =

[0227] stamp_delta-pre_tm_stamp_delta

[0228] v. Update pre_tm_stamp_delta and pre_tm_stamp, pre_tm_stamp_delta =

[0229] stamp_delta, pre_tm_stamp = cur_tm_stampg) stores the timestamp difference value in the timestamp compression sequence and records the number of valid bits in the compression sequence i. According to the maximum time difference difference allowed by the system and the common time difference difference in the system, set several value segments and record the number of bits required for the corresponding value segments. For example, this system divides 0, 0 bits, [1,63], 7 bits, [64,255], 9 bits, [256,2047], 12 bits, [2048,65535], 17 bits

[0230] ii. Set the control code, which is used to mark the range of the difference value. In this system, the values ​​are set to 0, 2 (binary 10), 6 (binary 110), 14 (binary 1110), and 62 (binary 111110). The control code 30 (binary 11110) is also set to mark the first element of the sequence (this control code is the common segmentation mark code mentioned later).

[0231] iii. If the current timestamp compression sequence is empty or a new segment is enabled, a byte is stored in the timestamp compression, and the upper 5 bits of the byte are set to the control code 30 (binary 11110). Then, the current timestamp content is appended to the timestamp compression sequence with a 48-bit width value, thus completing the recording of the first element.

[0232] iv. When a timestamp has already been stored in the timestamp compression sequence, if tm_stamp_delta_delta == 0,

[0233] Then add a bit with the value of 0 to the compressed sequence (in fact, if the number of valid bits in the compressed sequence content is an integer multiple of 8 bits, add a byte to the sequence and set the highest bit of the byte to the value of 0; otherwise, set the value of the next lowest bit of the valid bit of the last byte of the compressed sequence to 0), and increase the number of valid bits of the compressed sequence by 1 (similar practices are used for adding control code bit values ​​and difference bit values ​​subsequently)

[0234] v. When a timestamp has been stored in the timestamp compression sequence, if tm_stamp_delta_delta < 0, calculate its absolute value to get abs_delta_delta, check which value segment abs_delta_delta falls in, append the control code of this value segment to the timestamp compression sequence, and append the difference value tm_stamp_delta_delta to the timestamp compression sequence according to the bit length corresponding to the value segment.

[0235] vi. When a timestamp has already been stored in the timestamp compression sequence, if tm_stamp_delta_delta>0,

[0236] Set tm_stamp_delta_delta to tm_stamp_delta_delta-1, check

[0237] In which numerical segment of the time difference does tm_stamp_delta_delta fall? The control code of this numerical segment is appended to the timestamp compression sequence, and the difference value tm_stamp_delta_delta is appended to the timestamp compression sequence according to the bit length corresponding to the numerical segment.

[0238] h) Record the number of valid bits of the compressed timestamp sequence tm_stamp_bit_length. Each time a new timestamp is added, the length of the control code and the length of the valid value are added to tm_stamp_bit_length. 6. Use the segment compression module to perform segment compression on the timestamp compression sequence.

[0239] a) Set the segment size segment_size, where each segment is a time unit (the default setting in this algorithm is 3000 milliseconds). When the time of the data added to the sequence minus segment_start_tm_stamp is greater than

[0240] When segment_size is set, the following steps are performed to achieve segment compression

[0241] b) Set the segment splicing code and record the segment splicing control data thereafter

[0242] i. Each segmented data content starts with an independent byte, that is, there is no situation where the same byte content stores multiple segmented data, so as to facilitate the query and copy of segmented data

[0243] ii. The segment splicing control data consists of four parts: segment splicing marker code, padding code length, padding code content, and valid segment data. The marker code is set to 126 (binary 1111110); the padding code length is 3, which can represent 0 to 7 padding bits, used to store the number of padding bits, indicating the offset of the subsequent splicing data relative to the byte 0 bit; the padding code content is 0, and the length is the value of the padding code length part; the valid segment data is the timestamp compressed sequence data of this segment after re-compression (segment compression)

[0244] Or the content of the timestamp compressed sequence data of this section that has not been compressed again, whether to compress the timestamp compressed sequence of this section again depends on the data length of the compressed sequence of this section.

[0245] c) Set the segment compression threshold (e.g. 800 bits) segment_compress_threshold. When the bit length of the timestamp compressed sequence data in this segment is greater than this threshold, perform segment recompression (segment compression for short).

[0246] i. Compress only when the threshold is exceeded to avoid unnecessary resource consumption

[0247] ii. For segment compression, you can choose a general compression algorithm, such as zstd

[0248] iii. Set the control code to distinguish whether the segmented data is compressed. After the segmented data is compressed, the control code needs to be added to the segmented data first.

[0249] d) Set the segment mark code 30 (binary 11110) to mark the data content as a time stamp compressed sequence data that has not been compressed again, that is, ordinary segment data

[0250] e) Set the segment compression mark code 254 (binary 11111110) to mark the data content as a compressed sequence data of the timestamp of this segment after recompression, that is, compressed segment data

[0251] f) Reorganize the segmented timestamp compressed sequence data

[0252] i. Replace the original data with the segment splicing control data starting with the segment splicing code at the segment starting position segment_start_pos

[0253] ii. If the segment data does not need to be compressed again, append the segment data content (including the complete content of the byte where the segment mark code 30 is located) to the segment splicing control data.

[0254] iii. If the segmented data is compressed again, the following contents are appended after the segmented splicing control data: segmented compression mark code 254 (binary 11111110), segmented compressed data byte length (expressed as a 16-bit unsigned number), segmented data bit length before compression (expressed as a 16-bit unsigned number), segmented compressed data content (length is the segmented compressed data byte length)

[0255] g) Update the effective bit length tm_stamp_bit_length of the current timestamp sequence to

[0256] tm_stamp_bit_length minus the difference in segment data length before and after segment compression, plus the segment splicing control data length

[0257] h) Update segment_start_pos to the next bit after this segment data (the same as the effective bit length tm_stamp_bit_length of the current timestamp sequence)

[0258] i) Update the segment flag new_segment_flag to true

[0259] 7. Set up a segment index information table for each timestamp compression sequence and record the segment information in the timestamp compression sequence into this segment index information table

[0260] a) Set up a segment index table. The index table is a one-dimensional array, and each element records a segment index information

[0261] b) In order to quickly locate and query data in a specified time period, the segment index table size is determined by the segment size segment_size and the maximum time max_tm_duration allowed for recording of the timestamp sequence.

[0262] segment_table_size=(max_tm_duration+segment_size-1) / segment_size

[0263] c) Each element in the segment table represents a segment index, even if no valid data has been received in that segment

[0264] d) Each segment index information includes: validity (indicates whether valid data is received in the current segment), start timestamp, start position, data length

[0265] e) When adding a new segment, update the content of the corresponding element in the segment index table, modify its validity to valid, the starting timestamp to the newly added timestamp, and the starting position to the offset of the starting bit of the compressed sequence data of the current new segment relative to the timestamp.

[0266] f) When a segment compression is completed, the content of the corresponding element in the segment index table is updated, mainly to update the data length content of this segment

[0267] 8. Compress the numerical sequence of each data item in the data frame

[0268] a) Select the compression algorithm. The specific compression algorithm can be selected according to the numerical type and rules of the data item. For example, for integer data items, Delta differential encoding, Delta-of-Delta second-order differential encoding,

[0269] Zig-zag encoding and other algorithms, floating-point data can use Gorilla algorithm, etc.

[0270] b) Store the compression algorithm at the beginning of the data item numerical compression sequence (using 8 bits to enumerate and encode different compression types) so that the corresponding decompression can be performed according to different compression types during decompression.

[0271] c) Maintain a one-to-one correspondence between the record sequence of each data item in the same data frame and the timestamp compression record sequence of this data frame. For example, a data frame of type A contains 10 data items, which will be compressed and stored using a timestamp compression sequence and 10 numerical compression sequences. When a data frame of type A is connected to the compression system, a timestamp compression record will be generated and stored in the timestamp compression sequence. At the same time, 10 numerical compression records of the data items will be generated and stored in the corresponding 10 numerical compression sequences. When another data frame of type A is connected, another timestamp compression record will be generated and stored in the timestamp compression sequence, and 10 numerical compression records of the data items will be generated and stored in the corresponding 10 numerical compression sequences.

[0272] d) Set the corresponding variable for each data item numerical compression sequence to record the effective length of each data item numerical compression sequence and other information

[0273] 9. Perform segmented compression on the data item numerical compression sequence

[0274] a) When a new segment (except the first segment) is generated in the data frame timestamp sequence, that is,

[0275] When new_segment_flag is true, segment compression is performed on the data item value sequence in the same data frame in a similar way to timestamp segment compression.

[0276] b) When a new segment is generated in the data frame timestamp sequence (except the first segment), similar to the timestamp segment compression, the corresponding segmentation mark information is added to the data item data value sequence under the same data frame in a similar way. 10. For each data item value compression sequence, a segment index information table is set to record the data item value compression segment information.

[0277] a) When allocating a segment index table for a timestamp sequence, allocate a segment index table of the same size for each data item value sequence under the same type of data frame

[0278] b) Record the segment information in the data item value compression sequence into this segment index information table in the same way as the timestamp compression sequence segment index table

[0279] 11. Use the compressed data organization module to cache the compressed sequence and provide data query and decompression interfaces to the outside world

[0280] a) collecting timestamp segmentation information and data item value segmentation information during compression;

[0281] b) caching the segment information, timestamp compression sequence, and data item number compression sequence;

[0282] c) Provide external data query interface and data query function by time period and data item;

[0283] d) Provide data decompression function to external parties:

[0284] i. Decompress the timestamp compressed sequence using Delta-of-Delta second-order differential decoding

[0285] ii. According to the compression algorithm type of the data item numerical compression sequence header, select the corresponding decompression algorithm to perform data item numerical decompression

[0286] It should be understood that, although the various steps in the above flow chart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0287] In some exemplary embodiments, Figure 10 As shown, a data compression device 1000 is provided, comprising: a receiving module 1002, a processing module 1004, a first compression module 1006 and a second compression module 1008, wherein:

[0288] The receiving module 1002 is configured to receive time-series data frames.

[0289] The processing module 1004 is configured to determine a data type and a timestamp of a received time series data frame and at least one data item in the received time series data frame, wherein the number of timestamps of the same time series data frame is one.

[0290] The first compression module 1006 is configured to compress the timestamp and record the obtained timestamp compression value into a timestamp compression sequence corresponding to the data type.

[0291] The second compression module 1008 is used to establish a mapping relationship between the timestamp compression sequence and the data item numerical compression sequence, compress at least one data item, obtain at least one data item numerical compression value, and record the at least one data item numerical compression value into the data item numerical compression sequence according to the mapping relationship.

[0292] In some exemplary embodiments, the first compression module 1006 determines whether the timestamp is the starting timestamp of the time series data frame of the data type. If it is the starting timestamp, the first control code corresponding to the starting timestamp is obtained, the timestamp and the first control code are spliced ​​to obtain a first timestamp compression value, and the first timestamp compression value is recorded in the timestamp compression sequence. If it is not the starting timestamp, the current timestamp difference is calculated based on the timestamp and the previous timestamp of the time series data frame of the data type, and the target timestamp difference is calculated based on the current timestamp difference and the previous timestamp difference. The previous timestamp difference is the difference between the previous timestamp and the previous data frame timestamp of the time series data frame of the data type. The second control code corresponding to the target timestamp difference is obtained, the target timestamp difference and the second control code are spliced ​​to obtain a second timestamp compression value, and the second timestamp compression value is recorded in the timestamp compression sequence.

[0293] In some exemplary embodiments, the first compression module 1006 determines the first number of bits required for the target timestamp difference, determines the second number of bits occupied by the second control code, calculates the total number of bits corresponding to the timestamp compression value based on the first number of bits and the second number of bits, and records the total number of bits in the timestamp compression sequence.

[0294] In some exemplary embodiments, the data compression device 1000 performs segmented compression on the timestamp compression sequence to obtain multiple timestamp segmented compressed data, obtains the data item numerical segmented data corresponding to each timestamp segmented data according to the mapping relationship, compresses the data item numerical segmented data, and obtains the data item numerical segmented compressed data.

[0295] In some exemplary embodiments, the data compression device 1000 obtains a preset segmented timestamp difference, and calculates a current timestamp difference based on the timestamp and the forward timestamp, where the forward timestamp is the previous timestamp corresponding to the timestamp in the timestamp compression sequence. If the current timestamp difference is greater than the preset segmented timestamp difference, the timestamp is divided into new segments to obtain the current timestamp segment, and the segmented splicing code corresponding to the current timestamp segment is set. The total number of segments bits corresponding to the current timestamp segment is obtained, and the total number of segments bits includes the number of bits occupied by the segmented splicing code and the number of bits occupied by the timestamp compression value. The total number of segments bits is compared to see whether it is greater than the preset segmented bit number. If so, the current timestamp segment is segment compressed to obtain timestamp segment compressed data.

[0296] In some exemplary embodiments, the data compression device 1000 performs byte compression on the current timestamp segment to obtain the number of bytes of the timestamp segment, obtains the segment compression marker code corresponding to the current timestamp segment, determines the segment length corresponding to the current timestamp segment based on the number of bytes of the timestamp segment, and splices the segment compression marker code, segment length and number of bytes of the timestamp segment to obtain the timestamp segmented compressed data corresponding to the current timestamp segment.

[0297] In some exemplary embodiments, when the total number of bits in the fragment is less than or equal to the preset number of segment bits, the data compression device 1000 does not need to perform segment compression on the current timestamp fragment, obtains the fragment marker code corresponding to the current timestamp fragment, and obtains the timestamp segment data corresponding to the current timestamp fragment based on the timestamp compression value and the fragment marker code.

[0298] In some exemplary embodiments, the data compression device 1000 obtains segment index information corresponding to each timestamp segment data, the segment index information includes the starting timestamp, starting position, segment data length and validity, and generates a segment index table corresponding to the timestamp compression sequence based on each segment index information.

[0299] In some exemplary embodiments, the data compression device 1000 obtains the data item numerical type corresponding to the data item numerical segmented data, obtains the data item numerical rule corresponding to the data item numerical segmented data, determines a data item compression algorithm that matches the data item numerical segmented data according to the data item numerical type and the data item numerical rule, compresses the data item numerical segmented data according to the data item compression algorithm, and obtains the data item numerical segmented compressed data.

[0300] The specific definition of the data compression device can be found in the definition of the data compression method above and will not be repeated here. Each module in the above-mentioned data compression device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0301] In some exemplary embodiments, a computer device is provided, the internal structure of which can be as follows: Figure 11As shown. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When executed by the processor, the computer program implements a data compression method.

[0302] Those skilled in the art will understand that Figure 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0303] In some exemplary embodiments, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: receiving a time-series data frame; determining the data type, timestamp, and at least one data item in the received time-series data frame, wherein the number of timestamps for the same time-series data frame is one; compressing the timestamp and recording the obtained timestamp compression value into a timestamp compression sequence corresponding to the data type; establishing a mapping relationship between the timestamp compression sequence and the data item numerical compression sequence, compressing at least one data item to obtain at least one data item numerical compression value, and recording the at least one data item numerical compression value into the data item numerical compression sequence according to the mapping relationship.

[0304] In some exemplary embodiments, when the processor executes the computer program, it also implements the following steps: determining whether the timestamp is the starting timestamp of the time-series data frame of the data type; if it is the starting timestamp, obtaining the first control code corresponding to the starting timestamp; splicing the timestamp and the first control code to obtain a first timestamp compression value; recording the first timestamp compression value in a timestamp compression sequence; if it is not the starting timestamp, calculating the current timestamp difference based on the timestamp and the previous timestamp of the time-series data frame of the data type; calculating the target timestamp difference based on the current timestamp difference and the previous timestamp difference, the previous timestamp difference being the difference between the previous timestamp and the previous data frame timestamp of the time-series data frame of the data type; obtaining the second control code corresponding to the target timestamp difference; splicing the target timestamp difference and the second control code to obtain a second timestamp compression value; recording the second timestamp compression value in a timestamp compression sequence.

[0305] In some exemplary embodiments, when the processor executes the computer program, it further implements the following steps: determining a first number of bits required for the target timestamp difference; determining a second number of bits occupied by the second control code; calculating a total number of bits corresponding to the second timestamp compression value based on the first number of bits and the second number of bits; and recording the total number of bits in a timestamp compression sequence.

[0306] In some exemplary embodiments, when the processor executes the computer program, it also implements the following steps: compressing the timestamp compression sequence into segments to obtain a plurality of timestamp segmented compressed data; obtaining the data item numerical segmented data corresponding to each timestamp segmented data according to the mapping relationship; and compressing the data item numerical segmented data to obtain the data item numerical segmented compressed data.

[0307] In some exemplary embodiments, when the processor executes the computer program, it also implements the following steps: obtaining a preset segmented timestamp difference; calculating a current timestamp difference based on the timestamp and the forward timestamp, where the forward timestamp is the previous timestamp corresponding to the timestamp in the timestamp compression sequence; if the current timestamp difference is greater than the preset segmented timestamp difference, dividing the timestamp into new segments to obtain a current timestamp segment; setting a segmented splicing code corresponding to the current timestamp segment; obtaining the total number of segments bits corresponding to the current timestamp segment, where the total number of segments bits includes the number of bits occupied by the segmented splicing code and the number of bits occupied by the timestamp compression value; comparing whether the total number of segments bits is greater than the preset number of segments bits; if so, performing segment compression on the current timestamp segment to obtain timestamp segmented compressed data.

[0308] In some exemplary embodiments, when the processor executes the computer program, it also implements the following steps: performing byte compression on the current timestamp segment to obtain the number of bytes of the timestamp segment; obtaining the segment compression mark code corresponding to the current timestamp segment; determining the segment length corresponding to the current timestamp segment based on the number of bytes of the timestamp segment; and splicing the segment compression mark code, the segment length and the number of bytes of the timestamp segment to obtain the timestamp segmented compressed data corresponding to the current timestamp segment.

[0309] In some exemplary embodiments, the processor further implements the following steps when executing the computer program: when the total number of bits of the fragment is less than or equal to the preset number of segment bits, there is no need to perform segment compression on the current timestamp fragment; obtain the fragment marker code corresponding to the current timestamp fragment; and obtain the timestamp segment data corresponding to the current timestamp fragment by splicing the timestamp compression value and the fragment marker code.

[0310] In some exemplary embodiments, when the processor executes the computer program, it also implements the following steps: obtaining segment index information corresponding to each timestamp segment data, the segment index information including the starting timestamp, starting position, segment data length and validity; generating a segment index table corresponding to the timestamp compression sequence based on each segment index information.

[0311] In some exemplary embodiments, when the processor executes the computer program, it also implements the following steps: obtaining the data item numerical type corresponding to the data item numerical segmented data; obtaining the data item numerical rule corresponding to the data item numerical segmented data; determining a data item compression algorithm that matches the data item numerical segmented data based on the data item numerical type and the data item numerical rule; compressing the data item numerical segmented data based on the data item compression algorithm to obtain the data item numerical segmented compressed data.

[0312] In some exemplary embodiments, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: receiving a time-series data frame; determining the data type, timestamp and at least one data item in the received time-series data frame, wherein the number of timestamps of the same time-series data frame is one; compressing the timestamp and recording the obtained timestamp compression value in a timestamp compression sequence corresponding to the data type; establishing a mapping relationship between the timestamp compression sequence and the data item numerical compression sequence, compressing at least one data item to obtain at least one data item numerical compression value, and recording the at least one data item numerical compression value in the data item numerical compression sequence according to the mapping relationship.

[0313] In some exemplary embodiments, when the computer program is executed by the processor, the following steps are implemented: determining whether the timestamp is the starting timestamp of the time-series data frame of the data type; if it is the starting timestamp, obtaining the first control code corresponding to the starting timestamp; splicing the timestamp and the first control code to obtain a first timestamp compression value; recording the first timestamp compression value in a timestamp compression sequence; if it is not the starting timestamp, calculating the current timestamp difference based on the timestamp and the previous timestamp of the time-series data frame of the data type; calculating the target timestamp difference based on the current timestamp difference and the previous timestamp difference, the previous timestamp difference being the difference between the previous timestamp and the timestamp of the previous data frame of the time-series data frame of the data type; obtaining the second control code corresponding to the target timestamp difference; splicing the target timestamp difference and the second control code to obtain a second timestamp compression value; recording the second timestamp compression value in the timestamp compression sequence.

[0314] In some exemplary embodiments, when the computer program is executed by a processor, the following steps are implemented: determining a first number of bits required for the target timestamp difference value; determining a second number of bits occupied by the second control code; calculating a total number of bits corresponding to the second timestamp compression value based on the first number of bits and the second number of bits;

[0315] Record the total number of bits into the timestamp compressed sequence.

[0316] In some exemplary embodiments, when the computer program is executed by a processor, the following steps are implemented: segmentally compressing the timestamp compression sequence to obtain a plurality of timestamp segmented compressed data; obtaining the data item numerical segmented data corresponding to each timestamp segmented data according to a mapping relationship; and compressing the data item numerical segmented data to obtain the data item numerical segmented compressed data.

[0317] In some exemplary embodiments, when the computer program is executed by a processor, the following steps are implemented: obtaining a preset segmented timestamp difference; calculating a current timestamp difference based on the timestamp and the forward timestamp, where the forward timestamp is the previous timestamp corresponding to the timestamp in the timestamp compression sequence; if the current timestamp difference is greater than the preset segmented timestamp difference, dividing the timestamp into new segments to obtain a current timestamp segment; setting a segmented splicing code corresponding to the current timestamp segment; obtaining the total number of segments bits corresponding to the current timestamp segment, where the total number of segments bits includes the number of bits occupied by the segmented splicing code and the number of bits occupied by the timestamp compression value; comparing whether the total number of segments bits is greater than the preset number of segments bits; if so, performing segment compression on the current timestamp segment to obtain timestamp segmented compressed data.

[0318] In some exemplary embodiments, when the computer program is executed by a processor, the following steps are implemented: byte compression is performed on the current timestamp segment to obtain the number of bytes of the timestamp segment; a segment compression marker code corresponding to the current timestamp segment is obtained; the segment length corresponding to the current timestamp segment is determined based on the number of bytes of the timestamp segment; and timestamp segment compression data corresponding to the current timestamp segment is obtained by splicing the segment compression marker code, the segment length and the number of bytes of the timestamp segment.

[0319] In some exemplary embodiments, when the computer program is executed by a processor, the following steps are implemented: when the total number of bits of the segment is less than or equal to the preset number of segment bits, there is no need to perform segment compression on the current timestamp segment; obtain the segment marker code corresponding to the current timestamp segment; and obtain the timestamp segment data corresponding to the current timestamp segment based on the timestamp compression value and the segment marker code.

[0320] In some exemplary embodiments, when the computer program is executed by a processor, the following steps are implemented: obtaining segment index information corresponding to each timestamp segment data, the segment index information including the starting timestamp, the starting position, the segment data length and the validity; generating a segment index table corresponding to the timestamp compression sequence according to each segment index information.

[0321] In some exemplary embodiments, when the computer program is executed by a processor, the following steps are implemented: obtaining the data item numerical type corresponding to the data item numerical segmented data; obtaining the data item numerical rule corresponding to the data item numerical segmented data; determining a data item compression algorithm that matches the data item numerical segmented data based on the data item numerical type and the data item numerical rule; compressing the data item numerical segmented data based on the data item compression algorithm to obtain the data item numerical segmented compressed data.

[0322] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0323] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0324] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A data compression method, comprising: Receive time-series data frames; Determining a data type and a timestamp of a received time series data frame and obtaining at least one data item in the received time series data frame, wherein the number of timestamps of the same time series data frame is one; compressing the timestamp, and recording the obtained timestamp compression value into a timestamp compression sequence corresponding to the data type; Establishing a mapping relationship between the timestamp compression sequence and the data item value compression sequence; compressing the at least one data item to obtain at least one data item numerical compression value; The at least one data item numerical compression value is recorded into the data item numerical compression sequence according to the mapping relationship.

2. The method according to claim 1, characterized in that The compressing the timestamp and recording the obtained timestamp compression value into the timestamp compression sequence corresponding to the data type includes: Determine whether the timestamp is the starting timestamp of a time series data frame of the data type; If it is a start timestamp, obtaining a first control code corresponding to the start timestamp; Concatenate the timestamp and the first control code to obtain a first timestamp compression value; Recording the first timestamp compression value into the timestamp compression sequence; If it is not the starting timestamp, calculating the difference between the current timestamp and the previous timestamp of the time series data frame of the data type; Calculating a target timestamp difference value based on a current timestamp difference value and a previous timestamp difference value, wherein the previous timestamp difference value is a difference value between the previous timestamp value and a timestamp value of a previous data frame of a time series data frame of the data type; Obtaining a second control code corresponding to the target timestamp difference; concatenating the target timestamp difference and the second control code to obtain a second timestamp compression value; The second timestamp compressed value is recorded in the timestamp compressed sequence.

3. The method according to claim 2, characterized in that The method further comprises: determining a first number of bits required for the target timestamp difference; determining a second number of bits occupied by the second control code; Calculate the total number of bits corresponding to the timestamp compression value according to the first number of bits and the second number of bits; The total number of bits is recorded into the timestamp compressed sequence.

4. The method according to claim 1, wherein The method further comprises: Performing segmented compression on the timestamp compression sequence to obtain a plurality of timestamp segmented compressed data; According to the mapping relationship, segment information is set for the data item value corresponding to each of the timestamp segmented compressed data; The data item numerical value compression sequence is segmented and compressed according to the segmentation information to obtain data item numerical value segmented compressed data.

5. The method according to claim 4, characterized in that The step of performing segmented compression on the timestamp compression sequence to obtain a plurality of timestamp segmented compressed data includes: Get the preset segment time length; Calculating a current timestamp difference based on the timestamp and a forward timestamp, where the forward timestamp is a previous segment start timestamp corresponding to the timestamp in the timestamp compression sequence; If the current timestamp difference is greater than the preset segmentation time length, the timestamp is divided into new segments to obtain a current timestamp segment; Setting the segment splicing code corresponding to the current timestamp segment; Obtaining a total number of bits of a segment corresponding to the current timestamp segment, where the total number of bits of the segment includes the number of bits occupied by the segmented splicing code and the number of bits occupied by the timestamp compression value; Comparing whether the total number of bits of the segment is greater than a preset number of segment bits; If it is greater, the current timestamp segment is segment compressed to obtain timestamp segment compressed data.

6. The method according to claim 5, characterized in that The performing segment compression on the current timestamp segment to obtain timestamp segment compressed data includes: Performing byte compression on the current timestamp segment to obtain the number of bytes of the timestamp segment; Obtaining a segment compression marker code corresponding to the current timestamp segment; Determine the length of the segment corresponding to the current stamp segment according to the number of bytes of the timestamp segment; The timestamp segmented compressed data corresponding to the current timestamp segment is obtained by splicing the segment compression mark code, the segment length and the number of bytes of the timestamp segment.

7. The method according to claim 5, characterized in that The method further comprises: When the total number of bits of the segment is less than or equal to the preset number of segment bits, there is no need to perform segment compression on the current timestamp segment; Obtaining a segment marker code corresponding to the current timestamp segment; The timestamp segment data corresponding to the current timestamp segment is obtained by splicing the timestamp compression value and the segment marker code.

8. The method according to any one of claims 4 to 7, characterized in that: The method further comprises: Obtaining segment index information corresponding to each of the timestamp segment data, wherein the segment index information includes a start timestamp, a start position, a segment data length, and validity; A segment index table corresponding to the timestamp compression sequence is generated according to each segment index information.

9. The method according to claim 4, characterized in that The step of performing segmented compression on the data item numerical compression sequence according to the segmentation information to obtain segmented compressed data of the data item numerical values ​​includes: Segmenting the data item numerical compression sequence according to the segmentation information to obtain a plurality of data item numerical segmentation data; Obtain the data item value type corresponding to the data item value segment data; Obtaining data item numerical patterns corresponding to the data item numerical segmentation data; Determine a data item compression algorithm for matching the data item value segment data according to the data item value type and the data item value regularity; The data item numerical segmented data is compressed according to the data item compression algorithm to obtain data item numerical segmented compressed data.

10. A data compression device, characterized in that: The device comprises: A receiving module, used for receiving time-series data frames; a processing module, configured to determine a data type and a timestamp of a received time series data frame and at least one data item in the received time series data frame, wherein the number of timestamps of the same time series data frame is one; a first compression module, configured to compress the timestamp and record the obtained timestamp compression value into a timestamp compression sequence corresponding to the data type; The second compression module is used to establish a mapping relationship between the timestamp compression sequence and the data item numerical compression sequence, compress the at least one data item, obtain at least one data item numerical compression value, and record the at least one data item numerical compression value into the data item numerical compression sequence according to the mapping relationship.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images