Data processing method and device

Abstract

The embodiments of this specification provide a data processing method and device, wherein the data processing method includes obtaining time series data to be processed, and dividing the time series data into first time series data and second time series data; encoding the first time series data based on a dictionary coding algorithm to generate a first coding value; encoding the second time series data based on a column compression algorithm to generate a second coding value; and generating processed time series data according to the first coding value and the second coding value. The data processing method divides the time series data into two parts according to the data type, and compresses and encodes each part of the time series data using a compression coding method corresponding to the time series data, thereby greatly reducing the size of the time series data, reducing the storage space occupied by the time series data in the database, and increasing the amount of time series data written into the database.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of data processing technology, and more particularly to a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, and a computer-readable storage medium. Background Art

[0002] Time series data (full name: Time Series Data) is defined as a string of data indexed by the time dimension. It can also be understood as detection data based on a series of indicators generated continuously at a stable frequency. For example, when monitoring the air quality of a city, a series of data is generated by collecting a sulfur dioxide concentration value every second. In various applications, such time series data needs to be persistently stored in a database for query. The existing method of storing time series data usually stores the collected raw time series data in the database. This storage method will cause the time series data to occupy a large amount of storage space. When persistently storing massive time series data, directly storing the raw time series data will also result in a waste of storage space.

[0003] Therefore, there is an urgent need to provide a data processing method that can save storage space for time series data. Summary of the Invention

[0004] In view of this, the present specification provides a data processing method. One or more embodiments of the present specification also relate to a data processing apparatus, a computing device, and a computer-readable storage medium to address technical deficiencies in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided a data processing method, including:

[0006] Acquire time series data to be processed, and divide the time series data into first time series data and second time series data;

[0007] Encoding the first time series data based on a dictionary encoding algorithm to generate a first encoding value;

[0008] Encoding the second time series data based on a column compression algorithm to generate a second encoded value;

[0009] Processed time series data is generated according to the first encoding value and the second encoding value.

[0010] Optionally, encoding the first time series data based on a dictionary encoding algorithm to generate a first encoding value includes:

[0011] Create a local dictionary table based on the preset table structure;

[0012] Adding the first time series data to the local dictionary table and setting a corresponding digital identifier for the first time series data;

[0013] The first time series data is encoded based on the local dictionary table to generate a first encoding value, wherein the local dictionary table stores the first time series data and a digital identifier corresponding to the first time series data.

[0014] Optionally, the adding the first time series data to the local dictionary table and setting a corresponding digital identifier for the first time series data includes:

[0015] Get the i-th first time series data, where i∈[1,n] and n is a positive integer;

[0016] Determine whether the i-th first time series data is in the local dictionary table,

[0017] If so, obtain the digital identifier corresponding to the i-th first time series data in the local dictionary table,

[0018] If not, adding the i-th first time series data to the local dictionary table and setting a corresponding digital identifier for the i-th first time series data;

[0019] Determine whether i is greater than or equal to n;

[0020] If so, then end.

[0021] If not, then i is incremented by 1 and the i-th first time series data is obtained.

[0022] Optionally, encoding the first time series data based on the local dictionary table to generate a first encoded value includes:

[0023] The digital identifier corresponding to the first time series data in the local dictionary table is used to replace the first time series data to generate a first encoding value of the first time series data.

[0024] Optionally, the adding the i-th first time series data to the local dictionary table and setting a corresponding digital identifier for the i-th first time series data includes:

[0025] Add the i-th first time series data to the local dictionary table, and obtain the digital identifier corresponding to the i-1-th first time series data;

[0026] The digital identifier corresponding to the (i-1)th first time series data is incremented at a preset interval to generate a digital identifier corresponding to the (i)th first time series data.

[0027] Optionally, encoding the second time series data based on a column compression algorithm to generate a second encoded value includes:

[0028] Acquire timestamp data and non-timestamp data in the second time series data;

[0029] The timestamp data in the same column are encoded based on a delta-of-delta algorithm, and the non-timestamp data in the same column are encoded based on a preset algorithm to generate a second encoded value.

[0030] Optionally, before encoding the non-timestamp data in the same column based on a preset algorithm, the method further includes:

[0031] When the types of the non-timestamp data in the same column are inconsistent, the types of the non-timestamp data are unified in format.

[0032] Optionally, before obtaining the time series data to be processed, the process further includes:

[0033] receiving a write request for the time series data, and writing the time series data into a time series data log based on the write request;

[0034] Accordingly, the step of obtaining the time series data to be processed includes:

[0035] The time series data to be processed is obtained from the time series data log.

[0036] Optionally, the first time series data includes key time series data, and the second time series data includes value time series data;

[0037] Accordingly, dividing the time series data into first time series data and second time series data includes:

[0038] The time series data is divided into key time series data and value time series data according to data type.

[0039] Optionally, the processed time series data includes a first encoding value, a second encoding value, and a local dictionary table.

[0040] According to a second aspect of the embodiments of this specification, there is provided a data processing device, including:

[0041] a data acquisition module, configured to acquire time series data to be processed and divide the time series data into first time series data and second time series data;

[0042] a first encoding module, configured to encode the first time series data based on a dictionary encoding algorithm to generate a first encoding value;

[0043] a second encoding module, configured to encode the second time series data based on a column compression algorithm to generate a second encoding value;

[0044] The data processing module is configured to generate processed time series data according to the first coding value and the second coding value.

[0045] Optionally, the first encoding module is further configured to:

[0046] Create a local dictionary table based on the preset table structure;

[0047] Adding the first time series data to the local dictionary table and setting a corresponding digital identifier for the first time series data;

[0048] The first time series data is encoded based on the local dictionary table to generate a first encoding value, wherein the local dictionary table stores the first time series data and a digital identifier corresponding to the first time series data.

[0049] Optionally, the first encoding module is further configured to:

[0050] Get the i-th first time series data, where i∈[1,n] and n is a positive integer;

[0051] Determine whether the i-th first time series data is in the local dictionary table,

[0052] If so, obtain the digital identifier corresponding to the i-th first time series data in the local dictionary table,

[0053] If not, adding the i-th first time series data to the local dictionary table and setting a corresponding digital identifier for the i-th first time series data;

[0054] Determine whether i is greater than or equal to n;

[0055] If so, then end.

[0056] If not, then i is incremented by 1 and the i-th first time series data is obtained.

[0057] Optionally, the first encoding module is further configured to:

[0058] The digital identifier corresponding to the first time series data in the local dictionary table is used to replace the first time series data to generate a first encoding value of the first time series data.

[0059] Optionally, the first encoding module is further configured to:

[0060] Add the i-th first time series data to the local dictionary table, and obtain the digital identifier corresponding to the i-1-th first time series data;

[0061] The digital identifier corresponding to the (i-1)th first time series data is incremented at a preset interval to generate a digital identifier corresponding to the (i)th first time series data.

[0062] Optionally, the second encoding module is further configured to:

[0063] Acquire timestamp data and non-timestamp data in the second time series data;

[0064] The timestamp data in the same column are encoded based on a delta-of-delta algorithm, and the non-timestamp data in the same column are encoded based on a preset algorithm to generate a second encoded value.

[0065] Optionally, the device further includes:

[0066] The format conversion module is configured to unify the formats of the non-timestamp data when the types of the non-timestamp data in the same column are inconsistent.

[0067] Optionally, the device further includes:

[0068] a data writing module, configured to receive a write request for the time series data, and write the time series data into a time series data log based on the write request;

[0069] Accordingly, the data acquisition module is further configured to:

[0070] The time series data to be processed is obtained from the time series data log.

[0071] Optionally, the first time series data includes key time series data, and the second time series data includes value time series data;

[0072] Accordingly, the data acquisition module is further configured to:

[0073] The time series data is divided into key time series data and value time series data according to data type.

[0074] Optionally, the processed time series data includes a first encoding value, a second encoding value, and a local dictionary table.

[0075] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0076] memory and processor;

[0077] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the instructions are executed by the processor, the steps of the data processing method are implemented.

[0078] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the data processing method are implemented.

[0079] One embodiment of this specification implements a data processing method and device, wherein the data processing method includes obtaining time series data to be processed and dividing the time series data into first time series data and second time series data; encoding the first time series data based on a dictionary coding algorithm to generate a first coding value; encoding the second time series data based on a column compression algorithm to generate a second coding value; and generating processed time series data based on the first coding value and the second coding value. The data processing method divides the time series data into two parts according to the data type and compresses and encodes the data using the compression coding method corresponding to each part of the time series data, thereby greatly reducing the size of the time series data, reducing the storage space occupied by the time series data in the database, and increasing the amount of time series data written to the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a schematic diagram of a log-structured merge tree storage engine provided by an embodiment of this specification that applies the data processing method provided by an embodiment of this specification;

[0081] Figure 2 is a flow chart of a data processing method provided by one embodiment of this specification;

[0082] Figure 3 is a schematic diagram of time series data in a data processing method provided in one embodiment of this specification;

[0083] Figure 4 This is a schematic diagram of time series data divided according to data types in a data processing method provided in one embodiment of this specification;

[0084] Figure 5 This is a schematic diagram of encoding and compressing first time series data based on a local dictionary table in a data processing method provided by one embodiment of this specification;

[0085] Figure 6 is a schematic diagram of second time series data in a data processing method provided by an embodiment of this specification;

[0086] Figure 7This is a log structure diagram of time series data after encoding and compression in a data processing method provided by one embodiment of this specification;

[0087] Figure 8 This is a schematic diagram of the structure of a data processing device provided by one embodiment of this specification;

[0088] Figure 9 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0089] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0090] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0091] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0092] First, the terms involved in one or more embodiments of this specification are explained.

[0093] Time Series Database (TSDB): A data management system that provides efficient access to time series data and statistical analysis capabilities.

[0094] Time Series Data: By definition, it is a series of data indexed by the time dimension. It can also be understood as a series of measurement data based on a series of indicators that are continuously generated at a stable frequency. For example, when monitoring the air quality of a city, a series of data is generated by collecting a sulfur dioxide concentration value every second.

[0095] Metric: An indicator of monitored data, such as wind speed and temperature.

[0096] Tag: It is used to indicate the specific object that the indicator item is monitoring, and belongs to the data subcategory under the specified measurement. A tag consists of a tag key (TagKey) and a corresponding tag value (TagValue). For example, "City (TagKey) = Hangzhou (TagValue)" is a tag (Tag), or "Computer room (TagKey) = A, IP = 172.220.110.1 (TagValue)" is also a tag (Tag). Specifically, when monitoring data, if the specified measurement is "temperature" and the tag is "city = Hangzhou", it is determined that the monitored data represents the temperature in Hangzhou.

[0097] Tag key (TagKey, Tagk): Specifies the object type for the metric (a corresponding tag value will be used to locate the specific object under that object type), such as country, province, city, data center, IP address, etc.

[0098] Tag Value (TagValue, Tagv): The value corresponding to the tag key (TagKey). For example, if the tag key (TagKey) is "country", the tag value (TagValue) can be specified as "China".

[0099] Value: The value corresponding to the measurement, such as 15 (wind force) and 20℃ (temperature).

[0100] Timestamp: The time when the data is generated.

[0101] Field: A subcategory of data under a given metric. A metric can support multiple fields. For example, if the metric is wind, it can have two fields: direction and speed.

[0102] Value: The numerical value corresponding to the measurement, such as 56°C, 1000 r / s, etc. (in practice, without units). If there are multiple fields, each field has a corresponding value.

[0103] LSM: English full name: Log Structured Merge Tree, Chinese full name: Log Structured Merge Tree. The working principle of the LSM engine is to keep the modified increments of data in memory. After reaching the specified size limit, these modification operations are written to disk in batches.

[0104] In this specification, a data processing method is provided. This specification also relates to a data processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0105] The writing characteristics of time series data are continuous high concurrency and high throughput writing, with more write operations and fewer read operations. Therefore, it is more suitable to use the LSM engine to write time series data, that is, the data processing method provided in the embodiment of this specification can be applied to the LSM engine. In actual applications, the data processing method provided in the embodiment of this specification can be applied not only to the LSM engine, but also to any data storage engine that needs to write time series data, such as the InfluxDB TSM storage engine, etc. This specification does not impose any restrictions on this. For ease of understanding, in the embodiments of this specification, the data processing method provided in the embodiments of this specification is introduced in detail by taking the writing of time series data through the LSM engine as an example.

[0106] See also Figure 1 , Figure 1 A schematic diagram of a log-structured merge tree storage engine using the data processing method provided in an embodiment of this specification is shown.

[0107] Figure 1 It is the main component of an LSM engine, including four parts: a collection of time series log files, memory, a collection of time series data files written to disk, and data collation. Specifically, the working principle of the LSM engine is as follows:

[0108] Step 1: Write logs, that is, write time series data into log files in the time series log file set.

[0109] Specifically, the time series data is written into a log file in the time series log file set, and the time series data in the log file is compressed and encoded based on the data processing method of the embodiment of this specification.

[0110] Step 2: After writing the log, write the time series data in the log to the memtable. When the time series data in the memtable reaches the specified size limit, convert the valid time series data in the memtable into a memtable snapshot.

[0111] Step 3: Write the time series data in the memory table snapshot in batches to the corresponding disk files (i.e., database).

[0112] Specifically, the time series data in the memory table snapshot is written in batches into corresponding files in the second time series data file set.

[0113] Step 4: De-duplicate the time series data written to the corresponding disk file or further compress the time series data using a block compression algorithm based on the configuration to achieve time series data compression.

[0114] In actual applications, when the time series data written into one or more corresponding time series data files are sorted (for example, sorting, deduplication, or compression of the time series data), the sorted time series data will be written to a new disk file, and the original one or more old time series data files will be deleted to save disk space.

[0115] It can be seen that the log is the first step in the LSM engine to write time series data. The write throughput of the log largely determines the write throughput of the LSM engine. Therefore, the embodiments of this specification provide a data processing method that aims to compress and encode the time series data of the log to reduce the size of the log, reduce the data IO (Input / Output) overhead of writing the log to disk, and improve the write throughput of the log.

[0116] The following is a detailed introduction to the data processing method provided in one or more embodiments of this specification, with the purpose of implementing compression encoding of time series data in the log by the data processing method.

[0117] See also Figure 2 , Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0118] Step 202: Acquire time series data to be processed, and divide the time series data into first time series data and second time series data.

[0119] Among them, time series data is detection data of a series of indicators that are continuously generated at a stable frequency. For example, when detecting the air quality of a city, a series of data is generated by collecting a sulfur dioxide concentration value every second; or when detecting the wind speed and temperature of a city, a series of data is generated by collecting a wind speed or temperature value every second, etc.

[0120] In the embodiments of this specification, when "processing" can be understood as encoding and compression, obtaining the time series data to be processed can be understood as obtaining the time series data to be encoded and compressed and containing a timestamp.

[0121] The data types include but are not limited to character types and numeric types, wherein the numeric types include but are not limited to integer types, floating point types, and Boolean types.

[0122] Specifically, when the data type includes a character type and a numeric type, the first time series data may be key time series data, and the second time series data may be value time series data, then dividing the time series data into the first time series data and the second time series data includes:

[0123] The time series data is divided into key time series data and value time series data according to data type.

[0124] In an embodiment of the present specification, after obtaining the time series data to be encoded and compressed, the time series data can be divided into two parts: key time series data and value time series data according to the data type. Subsequently, the time series data of these two data types can be encoded and compressed separately. The compression of the time series data is achieved through two encoding and compression links, which can greatly improve the compression efficiency of encoding and compressing the time series data.

[0125] In another embodiment of the present specification, before obtaining the time series data to be processed, the method further includes:

[0126] receiving a write request for the time series data, and writing the time series data into a time series data log based on the write request;

[0127] Accordingly, the step of obtaining the time series data to be processed includes:

[0128] The time series data to be processed is obtained from the time series data log.

[0129] Still taking the above as an example, the data processing method provided in the embodiment of this specification is applied to the LSM engine. In order to improve the write throughput of the log in the LSM engine, the time series data in the time series data log can be obtained and then compressed and encoded. Therefore, in the embodiment of this specification, the LSM engine is used as the execution body. After receiving a write request for time series data, the time series data is first written to the time series data log. Then, the time series data to be processed is obtained from the time series data log and compressed and encoded to reduce the size of the log, reduce the data IO overhead of writing the log to the disk, and improve the write throughput of the log.

[0130] Step 204: Encode the first time series data based on a dictionary encoding algorithm to generate a first encoding value.

[0131] The first time series data is key time series data, and the first encoding value is an encoding value obtained by compressing the key time series data through a dictionary encoding algorithm.

[0132] In a specific implementation, encoding the first time series data based on a dictionary encoding algorithm to generate a first encoding value includes:

[0133] Create a local dictionary table based on the preset table structure;

[0134] Adding the first time series data to the local dictionary table and setting a corresponding digital identifier for the first time series data;

[0135] The first time series data is encoded based on the local dictionary table to generate a first encoding value, wherein the local dictionary table stores the first time series data and a digital identifier corresponding to the first time series data.

[0136] The local dictionary can be understood as a dictionary created by encoding and compressing a portion of the key time series data of the time series data. The encoding compression of the key time series data can be achieved through the local dictionary table.

[0137] The default table structure of the local dictionary table is a map of nodes<string,int> Structure, where string corresponds to a string and int corresponds to an integer digital identifier. In the process of creating a local dictionary table, each string in the key time series data is first traversed, and the map (mapping table, that is, the local dictionary table) is checked to see if there is this string. If so, the id corresponding to the string in the map is used as the id of the string. If not, a non-repeating id is generated for the string (for example, the maximum value of the id + 1), and then the string and the corresponding id are inserted into the local dictionary table to realize the creation of the local dictionary table for the key time series data.

[0138] In the embodiment of this specification, the key timing data is encoded and compressed through a local dictionary table, and the encoding compression of the timing data is achieved by replacing the key timing data with a digital identifier that occupies fewer bits. This can reduce the size of the key timing data and save data storage space.

[0139] In a specific implementation, encoding the first time series data based on the local dictionary table to generate a first encoding value includes:

[0140] The digital identifier corresponding to the first time series data in the local dictionary table is used to replace the first time series data to generate a first encoding value of the first time series data.

[0141] In the embodiment of the present specification, encoding compression of time series data is achieved by replacing key time series data with digital identifiers that occupy fewer bits, which can reduce the size of the key time series data and save data storage space.

[0142] In another embodiment of the present specification, the adding the first time series data to the local dictionary table and setting a corresponding digital identifier for the first time series data includes:

[0143] Get the i-th first time series data, where i∈[1,n] and n is a positive integer;

[0144] Determine whether the i-th first time series data is in the local dictionary table,

[0145] If so, obtain the digital identifier corresponding to the i-th first time series data in the local dictionary table,

[0146] If not, adding the i-th first time series data to the local dictionary table and setting a corresponding digital identifier for the i-th first time series data;

[0147] Determine whether i is greater than or equal to n;

[0148] If so, then end.

[0149] If not, then i is incremented by 1 and the i-th first time series data is obtained.

[0150] In practical applications, the value of i generally starts at 1. First, a check is made to determine whether the first key sequence data item is in the local dictionary table. If so, the digital identifier corresponding to the key sequence data item in the local dictionary table is used as the digital identifier of the first key sequence data item. If not, a unique digital identifier is mapped to the key sequence data item, and the key sequence data item and the corresponding digital identifier are then inserted into the local dictionary table. Next, a check is made to determine whether i is greater than or equal to n, indicating whether the key sequence data item has been traversed. If so, the local dictionary table creation process ends. If not, i is incremented by 1 to obtain the second key sequence data item. The above process is repeated until all key sequence data items have been traversed, completing the creation of the local dictionary table.

[0151] See also Figure 3 and Figure 4 , Figure 3 is a schematic diagram of time series data in a data processing method provided in one embodiment of this specification. Figure 4 This is a schematic diagram of time series data divided according to data types in a data processing method provided in one embodiment of this specification.

[0152] Figure 3It includes multiple time series data, and each time series data is multi-valued time series data. In actual applications, there will also be single-valued time series data. Since single-valued time series data is only a special case of multi-valued time series data, the encoding and compression method for multi-valued time series data is also applicable to single-valued time series data. This manual will no longer elaborate on the encoding and compression method for single-valued time series data, where single value represents one value and multi-value represents two or more values.

[0153] Specifically, Figure 3 The metric in the table represents the indicator of the monitoring data, and tags represents the label. Each label consists of a label key (TagKey) and a corresponding label value (TagValue). For example, Figure 3 In the example, "ems_power_building="A"" is a tag, "ems_power_room="B2-1.NA62"" is a tag, and "ems_power_cabinet="C14"" is a tag. timestamp indicates a timestamp, and F1 and F2 indicate values, respectively.

[0154] Figure 4 is Figure 3 Based on the data type Figure 3 Schematic diagram of key time series data and value time series data after the time series data in is divided.

[0155] Figure 4 In the data type, Figure 3 After the time series data in is divided, Figure 3 The time series data in is divided into two parts, see Figure 4 The first part is the key time series data, that is, the metric and tags of each row, also known as the time series data key SeriesKey. The second part is the value time series data, that is, the timestamp, F1, and F2 of each row. Among them, timestamp is the timestamp data, F1 and F2 are the domain values. The first part can use the local dictionary table to encode and compress the metrics and tags. For the second part, a compression algorithm such as delta-of-delta can be used to perform column compression on the timestamp, F1, F2 and other domain values.

[0156] First, based on Figure 4 The process of encoding key time series data based on a dictionary encoding algorithm to generate a first encoding value is introduced in detail.

[0157] First, create a local dictionary table based on the preset table structure, such as map<String,int> Create a local dictionary table and then start traversing Figure 4The key time series data in the first step is to determine Figure 4 Is the first key time series data "ems_power_pose" in the local dictionary table? In the embodiment of this specification, since the local dictionary table is a newly created local dictionary table, the local dictionary table is empty and does not contain any time series data. Then, a digital identifier id of 1 is mapped to the key time series data "ems_power_pose", and the key time series data "ems_power_pose" and the corresponding id "1" are inserted into the local dictionary table. Then, the key time series data "ems_power_pose" in the key time series data table is replaced with the identifier "1" to achieve coding compression; then the second key time series data "ems_power_building" is traversed. ; Continue with the above judgment, increment a digital identifier id of 2 for the key time series data "ems_power_building" based on the previous digital identifier, insert the key time series data "ems_power_building" and the corresponding id "2" into the local dictionary table, and replace the key time series data "ems_power_building" in the key time series data table with the identifier "2" to achieve coding compression; repeat the above steps to encode and compress all key time series data, and when the traversed key time series data appears in the local dictionary table, the corresponding id can be used directly without reinserting the data in the local dictionary table. The above encoding method can be used to Figure 4 The metric and tags time series data in the final encoding and compression are completed. For the specific encoding and compression results, see Figure 5 , Figure 5 This is a schematic diagram of encoding and compressing first time series data based on a local dictionary table in a data processing method provided in an embodiment of this specification.

[0158] In an embodiment of the present specification, encoding and compression of key time series data is achieved through an encoding and compression method of a local dictionary table, which can greatly reduce the storage space of the first time series data in the log. Since both the local dictionary table and the time series data after encoding and compression need to be stored in the log file, in an embodiment of the present specification, only one unique digital identifier is used in the local dictionary table for repeated strings, avoiding repeated mapping relationships for the same string in the local dictionary table. In this way, the size of the local dictionary table can be reduced, further saving the storage space occupied when storing time series data.

[0159] In another embodiment of the present specification, the adding the i-th first time series data to the local dictionary table and setting a corresponding digital identifier for the i-th first time series data includes:

[0160] Add the i-th first time series data to the local dictionary table, and obtain the digital identifier corresponding to the i-1-th first time series data;

[0161] The digital identifier corresponding to the (i-1)th first time series data is incremented at a preset interval to generate a digital identifier corresponding to the (i)th first time series data.

[0162] In an embodiment of the present specification, when the i-th first time series data is added to the local dictionary table, the digital identifier corresponding to the i-1-th first time series data is first obtained, that is, what is the data identifier corresponding to the immediately preceding first time series data, and then based on the data identifier corresponding to the immediately preceding first time series data, the digital identifier is incremented according to a preset interval to generate a new unique digital identifier as the digital identifier corresponding to the i-th first time series data, wherein the preset interval can be set according to actual application, for example, set to 1, 2 or 3, etc.

[0163] For example, the digital identifier corresponding to the i-1th first time series data is 2, and the preset interval is 1, then the digital identifier corresponding to the i-th first time series data can be set to 3.

[0164] In the embodiment of the present specification, the creation of the local dictionary table can be achieved by simply increasing the number, without the need for a complicated calculation process, which greatly improves the processing efficiency of the data processing method of the embodiment of the present specification.

[0165] Step 206: Encode the second time series data based on a column compression algorithm to generate a second encoded value.

[0166] Column compression algorithms include, but are not limited to, the delta-of-delta algorithm, the XOR algorithm (exclusive OR), and the zigzag algorithm. The delta-of-delta algorithm can significantly reduce data storage costs and improve data writing and query performance. The XOR algorithm is based on the exclusive OR gate, meaning that identical values ​​are 0 and different values ​​are 1. The zigzag algorithm compresses data that becomes larger due to padding.

[0167] Specifically, encoding the second time series data based on a column compression algorithm to generate a second encoded value includes:

[0168] Acquire timestamp data and non-timestamp data in the second time series data;

[0169] The timestamp data in the same column are encoded based on a delta-of-delta algorithm, and the non-timestamp data in the same column are encoded based on a preset algorithm to generate a second encoded value.

[0170] In a specific implementation, the second time series data is value time series data, which includes timestamp data and non-timestamp data. Non-timestamp data includes, but is not limited to, character data, integer data, floating-point data, or Boolean data. In practical applications, non-timestamp data can be referred to as domain values, and domain values ​​can also be strings. Domain value strings and key value strings differ in their characteristics and therefore require different compression encoding methods. For example, domain value strings may vary significantly and are generally compressed using the aforementioned general column-based compression algorithm. However, key value strings, due to their high repetition rate, are generally compressed using the aforementioned local dictionary table method.

[0171] See also Figure 4 , Figure 4 The rightmost column shows an example of the result after encoding and compressing the timestamp data in the same column based on the delta-of-delta algorithm. Figure 4 As can be seen, encoding and compressing the timestamp data in the same column based on the delta-of-delta algorithm can compress the 10-bit integer code into 0 or 1. Since 0 can be represented by one bit in a computer, it can greatly save the storage space of time series data.

[0172] In practical applications, the time series data model used to generate time series data usually lacks a schema (schema is an abstract set of metadata, mainly declarations of elements and attributes, and definitions of complex and simple data types). This leads to the possibility of inconsistent data types within the same column and the problem of some rows having no values ​​in this column when performing column-based compression encoding of non-timestamp value-based time series data. This poses two technical challenges for column-based compression of non-timestamp value-based time series data: 1. How to perform column-based compression and decompression on columns of different data types; 2. How to encode when some rows do not have values ​​in this column; see Figure 6 , Figure 6 is a schematic diagram of second time series data in a data processing method provided in one embodiment of this specification. Figure 6 The example shows two problems: inconsistent data types in the same column and no values ​​in this column in some rows. That is, there is no value in column F1, and there is inconsistent data type and no value in column F2.

[0173] In the embodiments of this specification, in order to address the above two issues, when encoding and compressing non-timestamp data, the fields contained in each timeline, the corresponding value of each field and its data type are first recorded in the log, and all values ​​in the same column are iterated and encoded according to the data type. When decoding, the data type is first decoded, and then the actual value is decoded according to the data type.

[0174] See also Figure 7 , Figure 7 This is a log structure diagram of time series data after encoding and compression in a data processing method provided by an embodiment of this specification.

[0175] Specifically, the above Figure 7 The schema portion of the sequence key encoding result records the field values ​​for each row. For each row of data, the number of fields it contains, as well as the names and data types of these fields, are recorded. The schema information for each row is identified and recorded when the data of each column is compressed row by row. If a row has no value in a certain column, the field information for this column will not be included when recording the schema.

[0176] See also Figure 6 Assume that the entire batch of data has only two columns, F1 and F2, and there is no value in the F1 column of the third row. The schema information recorded in the third row only records the field information of F2, that is, the third row only contains the field of F2, but not the field of F1. Therefore, when decoding the third row, the field information contained in the third row will be decoded first, that is, there is only the F2 column, but not the F1 column. Then, there is no need to decode the value of F1 in the third row from the F1 column. Instead, you only need to decode the value corresponding to the F2 column in the third row from the decoder corresponding to the F2 column according to the data type corresponding to the third row.

[0177] When performing column compression on value values, the compression algorithms used for different data types are as follows:

[0178] Timestamp data can be compressed using the delta-of-delta algorithm; floating-point data in non-timestamp data can be compressed using the xor algorithm; integer data in non-timestamp data can be compressed using the zigzag algorithm; Boolean data in non-timestamp data can be compressed using 0 or 1.

[0179] In an embodiment of this specification, the second time series data is divided into timestamp data and non-timestamp data and compressed separately. Reasonable compression of timestamp data and non-timestamp data is achieved through different compression algorithms, and the compression of the second time series data is divided into two-step links for compression, which further improves the processing efficiency of the data processing method.

[0180] In the embodiment of this specification, before encoding the non-timestamp data in the same column based on a preset algorithm, the method further includes:

[0181] When the types of the non-timestamp data in the same column are inconsistent, the types of the non-timestamp data are unified in format.

[0182] During specific implementation, in order to solve the problem of inconsistent data types of the time series data in the same column of the above-mentioned non-timestamp data, and to avoid garbled characters or incompression during the encoding and compression of the non-timestamp data, the type of the non-timestamp data in the same column will be format converted to ensure the uniformity of the data type of the non-timestamp data in the same column.

[0183] Step 208: Generate processed time series data according to the first coding value and the second coding value.

[0184] Specifically, the first time series data is coded and compressed according to the local dictionary table to obtain the first coding value, and the second time series data is coded and compressed according to the column compression algorithm to obtain the second coding value. Then, the data block formed by the first coding value and the data block formed by the second coding value are copied into a large data block to form the final coding compression result of the time series data in a log, that is, the processed time series data.

[0185] The processed time series data at least includes a first coding value, a second coding value and a local dictionary table.

[0186] See also Figure 7 , Figure 7 This is a log structure diagram of time series data after encoding and compression in a data processing method provided by an embodiment of this specification.

[0187] Figure 7 The log structure diagram of time series data after encoding and compression mainly includes the following four parts:

[0188] Log block index: contains two integers, field offset and dict offset, pointing to the field values ​​in the log and the offset of the local dictionary respectively;

[0189] Dictionary encoding block: It is a local dictionary table that only stores the corresponding strings sorted in ascending order by ID.

[0190] Field value encoding blocks: multiple data blocks obtained by compressing multiple data types mixed by field column;

[0191] Sequence key encoding block: uses local dictionary encoded IDs for metrics and tags and a schema for each timeline containing fields and their data types.

[0192] in, Figure 7 The vint in the code represents a variable-length integer after encoding an unsigned integer using the zigzag compression algorithm. The dictionary encoding result of the dictionary encoding block only stores the dictionary string, and does not store the corresponding dictionary value identifier, because when encoding the local dictionary to the log, each pair in the dictionary is<String,id> Sort the ids in ascending order, and then only encode the strings in all the sorted pairs. Since the ids in the dictionary are generated using a fixed rule (such as auto-increment), the minimum id is fixed (for example, fixed to 1). If the auto-increment rule is used to generate the id, then when decoding the dictionary block, you only need to decode the corresponding strings in order, and then restore the corresponding ids in this order (the encoding is sorted by the size of the ids).

[0193] In addition, the data processing method of the embodiment of this specification compresses and encodes the time series data based on a batch of data. The time series data is stored in logs, and each log is regarded as a log block. Figure 3 Each row of data is a time series data record. For example, when compressing and encoding time series data, a batch of 500 or 1000 time series data stored in a log block can be encoded. When encoding and compressing these 500 or 1000 data, the local dictionary created is for this batch of data. The parameters used in the encoding process are all recorded in the data block after encoding these 500 or 1000 data. During decoding, the local dictionary table and parameters are directly restored from this data block, and the corresponding metrics, tags, and field domain values ​​can be directly restored. This process does not require obtaining external parameters or variables, so each log block can be considered the smallest autonomous coding unit.

[0194] In the embodiments of this specification, in response to a large number of repeated character strings in time series data, a local dictionary table is introduced to encode the character strings, which effectively reduces the repetition rate of the character strings in the time series data of the log. Each log is the smallest autonomous encoding and decoding unit, and there is no need to maintain external parameters and variables. In response to the feature that the values ​​of the same column in the time series data do not change much, compression algorithms such as delta-of-delta, zigzag, and xor are introduced to perform column-based compression on timestamps and domain values. By recording the schema of each time line, the problem of encoding and compressing time series data when different data types or no values ​​exist in the same column is effectively solved.

[0195] In an embodiment of this specification, the data processing method includes obtaining time series data to be processed and dividing the time series data into first time series data and second time series data; encoding the first time series data based on a dictionary encoding algorithm to generate a first encoding value; encoding the second time series data based on a column compression algorithm to generate a second encoding value; and generating processed time series data based on the first encoding value and the second encoding value. The data processing method divides the time series data into two parts according to data type and compresses and encodes each part of the time series data using a compression encoding method corresponding to the time series data. This greatly reduces the size of the time series data, reduces the storage space occupied by the time series data in the database, and increases the amount of time series data written to the database.

[0196] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 8 FIG1 shows a schematic diagram of the structure of a data processing device provided by an embodiment of this specification. Figure 8 As shown, the device includes:

[0197] The data acquisition module 802 is configured to acquire time series data to be processed and divide the time series data into first time series data and second time series data;

[0198] A first encoding module 804 is configured to encode the first time series data based on a dictionary encoding algorithm to generate a first encoding value;

[0199] A second encoding module 806 is configured to encode the second time series data based on a column compression algorithm to generate a second encoding value;

[0200] The data processing module 808 is configured to generate processed time series data according to the first coding value and the second coding value.

[0201] Optionally, the first encoding module 804 is further configured to:

[0202] Create a local dictionary table based on the preset table structure;

[0203] Adding the first time series data to the local dictionary table and setting a corresponding digital identifier for the first time series data;

[0204] The first time series data is encoded based on the local dictionary table to generate a first encoding value, wherein the local dictionary table stores the first time series data and a digital identifier corresponding to the first time series data.

[0205] Optionally, the first encoding module 804 is further configured to:

[0206] Get the i-th first time series data, where i∈[1,n] and n is a positive integer;

[0207] Determine whether the i-th first time series data is in the local dictionary table,

[0208] If so, obtain the digital identifier corresponding to the i-th first time series data in the local dictionary table,

[0209] If not, adding the i-th first time series data to the local dictionary table and setting a corresponding digital identifier for the i-th first time series data;

[0210] Determine whether i is greater than or equal to n;

[0211] If so, then end.

[0212] If not, then i is incremented by 1 and the i-th first time series data is obtained.

[0213] Optionally, the first encoding module 804 is further configured to:

[0214] The digital identifier corresponding to the first time series data in the local dictionary table is used to replace the first time series data to generate a first encoding value of the first time series data.

[0215] Optionally, the first encoding module 804 is further configured to:

[0216] Add the i-th first time series data to the local dictionary table, and obtain the digital identifier corresponding to the i-1-th first time series data;

[0217] The digital identifier corresponding to the (i-1)th first time series data is incremented at a preset interval to generate a digital identifier corresponding to the (i)th first time series data.

[0218] Optionally, the second encoding module 806 is further configured to:

[0219] Acquire timestamp data and non-timestamp data in the second time series data;

[0220] The timestamp data in the same column are encoded based on a delta-of-delta algorithm, and the non-timestamp data in the same column are encoded based on a preset algorithm to generate a second encoded value.

[0221] Optionally, the device further includes:

[0222] The format conversion module is configured to unify the formats of the non-timestamp data when the types of the non-timestamp data in the same column are inconsistent.

[0223] Optionally, the device further includes:

[0224] a data writing module, configured to receive a write request for the time series data, and write the time series data into a time series data log based on the write request;

[0225] Accordingly, the data acquisition module 802 is further configured to:

[0226] The time series data to be processed is obtained from the time series data log.

[0227] Optionally, the first time series data includes key time series data, and the second time series data includes value time series data;

[0228] Accordingly, the data acquisition module 802 is further configured to:

[0229] The time series data is divided into key time series data and value time series data according to data type.

[0230] Optionally, the processed time series data includes a first encoding value, a second encoding value, and a local dictionary table.

[0231] In an embodiment of the present specification, the data processing device includes obtaining time series data to be processed and dividing the time series data into first time series data and second time series data; encoding the first time series data based on a dictionary encoding algorithm to generate a first encoding value; encoding the second time series data based on a column compression algorithm to generate a second encoding value; and generating processed time series data based on the first encoding value and the second encoding value. The data processing method divides the time series data into two parts according to the data type and compresses and encodes the data using the compression encoding method corresponding to each part of the time series data, thereby greatly reducing the size of the time series data, reducing the storage space occupied by the time series data in the database, and increasing the amount of time series data written to the database.

[0232] The above is a schematic diagram of a data processing device according to this embodiment. It should be noted that the technical solution of the data processing device and the technical solution of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical solution of the data processing device, please refer to the description of the technical solution of the above-mentioned data processing method.

[0233] Figure 9 The following is a block diagram of a computing device 900 according to one embodiment of the present disclosure. Components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0234] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0235] In one embodiment of the present specification, the above components of the computing device 900 and Figure 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 9 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0236] The computing device 900 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 900 can also be a mobile or stationary server.

[0237] The processor 920 is configured to execute the following computer-executable instructions, which implement the steps of the data processing method when executed by the processor.

[0238] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned data processing method are of the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the above-mentioned data processing method.

[0239] An embodiment of the present specification further provides a computer-readable storage medium storing computer instructions, which implement the steps of the data processing method when executed by a processor.

[0240] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the above-mentioned data processing method.

[0241] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0242] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0243] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0244] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0245] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Acquire time series data to be processed, and divide the time series data into first time series data and second time series data, wherein the first time series data is key time series data and the second time series data is value time series data; Encoding the first time series data based on a dictionary coding algorithm to generate a first coding value, wherein the first coding value is obtained by encoding and compressing the first time series data by replacing the first time series data with a digital identifier that occupies fewer bits; Obtaining timestamp data and non-timestamp data in the second time series data, encoding the timestamp data in the same column based on a delta-of-delta algorithm, and encoding the non-timestamp data in the same column based on a preset algorithm to generate a second encoding value; Processed time series data is generated according to the first encoding value and the second encoding value.

2. According to the data processing method of claim 1, encoding the first time series data based on a dictionary encoding algorithm to generate a first encoding value comprises: Create a local dictionary table based on the preset table structure; Adding the first time series data to the local dictionary table, and setting a corresponding digital identifier for the first time series data; The first time series data is encoded based on the local dictionary table to generate a first encoding value, wherein the local dictionary table stores the first time series data and a digital identifier corresponding to the first time series data.

3. The data processing method according to claim 2, wherein the adding the first time series data to the local dictionary table and setting a corresponding digital identifier for the first time series data comprises: Get the i-th first time series data, where i∈[1,n], and n is a positive integer; Determine whether the i-th first time series data is in the local dictionary table, If so, obtain the digital identifier corresponding to the i-th first time series data in the local dictionary table, If not, adding the i-th first time series data to the local dictionary table, and setting a corresponding digital identifier for the i-th first time series data; Determine whether i is greater than or equal to n; If yes, then end. If not, i is incremented by 1 to continue acquiring the i-th first time series data.

4. The data processing method according to claim 2, wherein encoding the first time series data based on the local dictionary table to generate a first encoding value comprises: The digital identifier corresponding to the first time series data in the local dictionary table is used to replace the first time series data to generate a first encoding value of the first time series data.

5. The data processing method according to claim 3, wherein the adding the i-th first time series data to the local dictionary table and setting a corresponding digital identifier for the i-th first time series data comprises: Add the i-th first time series data to the local dictionary table, and obtain the digital identifier corresponding to the i-1-th first time series data; The digital identifier corresponding to the i-1th first time series data is incremented at a preset interval to generate a digital identifier corresponding to the i-th first time series data.

6. The data processing method according to claim 1, before encoding the non-timestamp data in the same column based on a preset algorithm, further comprising: When the types of the non-timestamp data in the same column are inconsistent, the types of the non-timestamp data are unified in format.

7. The data processing method according to claim 1, before obtaining the time series data to be processed, further comprising: receiving a write request for the time series data, and writing the time series data into a time series data log based on the write request; Accordingly, the step of obtaining the time series data to be processed includes: The time series data to be processed is obtained from the time series data log.

8. The data processing method according to any one of claims 1 to 7, wherein dividing the time series data into first time series data and second time series data comprises: The time series data is divided into key time series data and value time series data according to data type.

9. According to the data processing method of claim 2, the processed time series data includes a first encoding value, a second encoding value and a local dictionary table.

10. A data processing device, comprising: A data acquisition module is configured to acquire time series data to be processed, and divide the time series data into first time series data and second time series data, wherein the first time series data is key time series data and the second time series data is value time series data; A first encoding module is configured to encode the first time series data based on a dictionary encoding algorithm to generate a first encoding value, wherein the first encoding value is obtained by encoding and compressing the first time series data by replacing the first time series data with a digital identifier occupying fewer bits; a second encoding module, configured to obtain timestamp data and non-timestamp data in the second time series data, encode the timestamp data in the same column based on a delta-of-delta algorithm, and encode the non-timestamp data in the same column based on a preset algorithm to generate a second encoding value; The data processing module is configured to generate processed time series data according to the first coding value and the second coding value.

11. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the instructions are executed by the processor, the steps of the data processing method described in any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 9.

Images