A method and apparatus for storing variable-length type time-series data

Abstract

The application relates to the field of databases and particularly provides a variable-length type time series data storage processing method and device. A time series data storage engine stores time series data collected by a collection point, and the collected time series data is divided into different ZdataPoint objects. A ZdataPoint object contains a plurality of Zblocks, each ZBlock stores time series data within a period of time, the time periods of the ZBlocks do not cross each other, and each ZBlock has one or more 64K fixed-size memory spaces. When the first 64K memory is full, a new ZBlock object needs to be applied to store new time series data. The ZBlocks are sequentially sorted according to the time periods to form a ZBlock array. A background disk-falling thread falls the time series data in the ZBlock which has been full into a disk, and after the disk falling succeeds, information record of the time series data block is added to a file index part. Compared with the prior art, the application balances the utilization rate and the access efficiency of the memory, reduces the memory fragmentation as much as possible, and reduces the performance loss as much as possible.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of database, and particularly provides a storage processing method and device for time series data of indefinite length type. BACKGROUND

[0002] Most of the data stored in the database is of fixed length type, such as integer, float, long integer, bool type, and fixed length byte type. The fixed length byte type refers to binary, which is a type for storing fixed length byte arrays. Each value in a data column of the BINARY(M) type occupies M bytes, and if the length is less than M, the database will be supplemented with zero bytes on the right.

[0003] The maximum length of the fixed length byte type is limited, and is generally less than 1024 bytes. For byte arrays with a larger maximum length, the VARBINARY type is used, which is a more flexible data type than binary and is also used to represent byte arrays. However, VARBINARY can save byte arrays of variable length. Therefore, for data objects whose exact length is difficult to estimate, it is more reasonable to use the VARBINARY data type.

[0004] The maximum length attribute of VARBINARY is input when the user creates a table, such as the SQL statement: CREATE TABLE TS_VAR_LEN (TimeStamp int64, PicByte VARBINARY(128000)); This table creation statement creates an indefinite length column PicByte with a maximum length of 128000 bytes. The table creation statement also has a timestamp field, which is the index of the time series data.

[0005] Time series data refers to time series data, which is a data column recorded in chronological order. Each data in the same data column must be of the same caliber and must be comparable. Time series data can be time period data or time point data. Time series data may also contain indefinite length type values or super long byte array values, such as pictures, sounds, etc. For these indefinite length type values, the time series data storage engine needs to provide corresponding storage support.

[0006] The main problem in the prior art is that the index occupies a large space, and the length of the indefinite length type value varies greatly, so storing it with a fixed length can easily result in a large waste of memory. SUMMARY

[0007] The present application is aimed at the deficiencies of the prior art and provides a storage processing method for time series data of indefinite length type with strong practicality.

[0008] The further technical task of the present application is to provide a design reasonable, safe and applicable indefinite length type time sequence data storage processing device.

[0009] The technical scheme adopted by the present application to solve its technical problems is:

[0010] An indefinite length type time sequence data storage processing method, a time sequence data storage engine stores time sequence data collected by a collection point, and the collected time sequence data is divided into different ZdataPoint objects according to different collection point sources;

[0011] The ZdataPoint object contains a plurality of Zblocks, each ZBlock stores time sequence data within a period of time, and the time periods of the ZBlocks do not overlap, each ZBlock has one or more 64K fixed size memory spaces, wherein the first one is used to store fixed length type values, and the other 64K memory spaces are used to store indefinite length values;

[0012] When the first 64K memory is full, a new ZBlock object needs to be applied to store new time sequence data, and the ZBlocks are sequentially sorted according to the time period to form a ZBlock array;

[0013] There is a background disk writing thread that writes the time sequence data in the ZBlock that has been written full to the disk, and after the disk writing is successful, the information record of the time sequence data block is appended to the file index part.

[0014] Further, the ZBlock object contains a TSDataVector, which stores the tuple value of the record, and the length of all tuple groups in each ZdataPoint object is fixed, and the tuple group refers to the storage format formed by sequentially arranging and storing all column values of a record.

[0015] Further, the ZBlock object further contains a ZVarLenBufferMgr object, and the ZVarLenBufferMgr object will apply 0 to multiple 64K memory spaces as needed to store indefinite length type values.

[0016] Further, different storage strategies are formulated for the difference in the maximum length MAX_LEN of the indefinite length type VARBINARY, when MAX_LEN < 255, a complete 64K memory space is used by the TSDataVector to store, and the tuple group of the record is sequentially stored, and a span space is set at the end of the 64K memory space, and the span space indicates the memory fragmentation within the block.

[0017] Further, when the tuple of the record being inserted uses the span memory space, it indicates that the current ZBlock object has been stored full, and a new ZBlock object needs to be created to store the record being inserted.

[0018] Further, when the MAX_LEN >= 255 and the MAX_LEN < 2048, the TSDataVector uses a 64k memory space to store a part of the tuple of the record in sequence, and the remaining 64K space stores the column values of the variable-length column. The TSDataVector stores the tuple of the record in sequence from front to back, and the variable-length column value space is sequentially applied from back to front.

[0019] Further, the span memory fragment in the 64k memory space appears in the middle part of the memory space. When the span space is insufficient to store the tuple of the record being inserted and the variable-length column value, a new ZBlock object needs to be applied to store the record being inserted.

[0020] Further, when the MAX_LEN >= 2048, the TSDataVector uses a complete 64k memory space to store, and sequentially stores the tuple of the record. In the tuple, the variable-length type field occupies a fixed 16-byte space.

[0021] The span space at the end of the 64K memory space indicates the memory fragment in the block.

[0022] Further, the method for reading the value first parses the position bytes of the tuple, finds the ZVarLenBufferMgr object, and finds the starting reading position according to the offset.

[0023] Suppose that the array of the 64k memory space stored by the ZVarLenBufferMgr is block, and the formula for positioning the starting address for reading according to the offset is:

[0024]

[0025] % indicates the remainder; valuePtr is the starting address of the column value byte array;

[0026] The first 8 bytes of ValuePtr are taken, converted into a long type value to obtain the length of the column value byte array, and then the array length bytes are sequentially read. When the 64k memory space reads to the end, but the array length bytes have not been read, the next 64k memory space in the block array is continuously read until the array length bytes are read, and the reading operation of the column value is completed.

[0027] A time-series data storage and processing apparatus for variable-length data includes: at least one memory and at least one processor;

[0028] The at least one memory is used to store a machine-readable program;

[0029] The at least one processor is configured to call the machine-readable program to execute a variable-length time-series data storage processing method.

[0030] Compared with the prior art, the variable-length time-series data storage processing method and apparatus of the present invention have the following outstanding advantages:

[0031] This invention analyzes variable-length data types and formulates different access strategies to balance memory utilization and access efficiency, minimizing memory fragmentation and performance loss. It implements support for variable-length column value types in a time-series data storage engine and improves memory utilization through different access strategies. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Appendix Figure 1 This is an overall structural diagram of the time-series data storage engine in a variable-length time-series data storage processing method;

[0034] Appendix Figure 2 This is a schematic diagram of the ZBlock object data structure in a time-series data storage and processing method for variable-length data types.

[0035] Appendix Figure 3 This is a schematic diagram of a tuple structure in a time-series data storage and processing method for variable-length data types.

[0036] Appendix Figure 4 This is a diagram showing the organization of the 64K memory space of a ZBlock object when MAX_LEN < 255 in a variable-length time-series data storage processing method.

[0037] Appendix Figure 5 This is a diagram showing the organization of the 64K memory space of a ZBlock object when MAX_LEN>=255 and MAX_LEN<2048 in a variable-length time-series data storage processing method.

[0038] AppendixFigure 6 This is a diagram showing the organization of the 64K memory space of a ZBlock object when MAX_LEN>=2048 in a variable-length time-series data storage processing method. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The following is a preferred embodiment:

[0041] like Figure 1 As shown in this embodiment, a method for processing variable-length time-series data storage is described. The time-series data storage engine stores time-series data collected from numerous data points. The collected time-series data is divided into different ZdataPoint objects based on the different data point sources. Each ZdataPoint object contains many ZBlocks, and each ZBlock stores time-series data for a specific period of time. The time periods between ZBlocks do not overlap. Each ZBlock has one or more 64KB fixed-size memory spaces, where the first 64KB is used to store fixed-length values, and the other 64KB is used to store variable-length values. When the first 64KB of memory is full, a new ZBlock object needs to be allocated to store new time-series data. The ZBlocks are sorted sequentially according to the time period order to form a ZBlock array.

[0042] There is a background disk write thread that writes the time-series data in the already full ZBlock to the disk. After the disk write is successful, the information record of this time-series data block is appended to the file index.

[0043] like Figure 2 As shown, the ZBlock object contains a TSDataVector that stores the tuple values ​​of the records. The length of all tuples in each ZdataPoint object is fixed. A tuple refers to a storage format formed by storing all the column values ​​of a record sequentially.

[0044] tuple structure as follows Figure 3 As shown, because the number of columns collected for each ZdataPoint is fixed, and the data type used to store each column of data is also fixed, the tuple length of all records in each ZdataPoint object is consistent.

[0045] ZBlock object contains a ZVarLenBufferMgr object, which will apply for 0 to multiple 64k memory space as needed, used to store the value of the indefinite length type.

[0046] Different storage strategies are developed for the difference of the maximum length MAX_LEN of the indefinite length type VARBINARY. The following are all assumed that C3 column is VARBINARY indefinite length type:

[0047] (1) MAX_LEN < 255

[0048] TSDataVector uses a complete 64k memory space to store, and stores the tuple of the record in turn. Since 64K may not be able to divide the number of bytes occupied by the tuple of the record, there will be some span space at the end of the 64K memory space, which represents the memory fragmentation in the block.

[0049] As shown in Figure 4 , when the tuple of the record being inserted uses the span memory space, it means that the current ZBlock object has been full, and a new ZBlock object needs to be created to store the record being inserted.

[0050] In the tuple, the storage space of MAX_LEN is allocated to the value of C3 column, and the byte number of the C3 column value of the record is certainly less than or equal to MAX_LEN. In the continuous memory allocated to the C3 column, the first byte is used to store the actual byte number of the C3 column value in the record, Length area; then the actual byte array of the value is stored, Value area; the memory space of MAX_LEN may have a remainder, and the remaining part is filled with byte 0 to form the Span area.

[0051] (2) MAX_LEN >= 255 and MAX_LEN < 2048

[0052] TSDataVector uses a 64k memory space to store a part of the tuple of the record in turn. The remaining 64K space stores the column value of the indefinite length column. TSDataVector stores the tuple of the record from front to back in turn; the indefinite length column value space is applied from back to front.

[0053] As shown in Figure 5 , the span memory fragmentation in the 64k memory space appears in the middle part of the memory space, when the span space is not enough to store the tuple of the record being inserted and the value of the indefinite length column, a new ZBlock object needs to be applied to store the record being inserted.

[0054] The C3 column position of the tuple of each record occupies two bytes of the tuple, which is used to store the starting offset of the actual storage position of the C3 column value in the 64K memory space, such as Figure 5 The two fields in the C3 position of the tuple of the t2 record store the starting offset of Var_value2 in the 64K memory space. The actual storage format of the variable-length column value is that the first two bytes store the actual byte number of the column value, and then the byte array of the column value is stored. For example Figure 5 The storage format of Var_value2 is that the actual byte number Length of the column value is first stored using two bytes, and then the actual byte array of the column value is stored.

[0055] (3) When MAX_LEN >= 2048

[0056] The TSDataVector uses a complete 64K memory space for storage, and sequentially stores the tuple of the record. In the tuple, the variable-length type field occupies a fixed 16-byte space. Since 64K cannot necessarily be evenly divided by the byte number occupied by the record tuple, there will be some span space at the end of the 64K memory space, which represents the memory fragmentation within the block.

[0057] The 16 bytes occupied by the variable-length type field are used to store the actual storage position of the data, and the 16 bytes are divided into two parts: the first 8 bytes are used to store the pointer of the ZVarLenBufferMgr object in the ZBlock object, and the last 8 bytes are used to store the starting offset position of the column value stored in the ZVarLenBufferMgr object.

[0058] As shown in Figure 6 The variable-length column c3 in the tuple of the t2 record occupies 16 bytes of the tuple. In the 16 bytes, the first 8 bytes store the pointer Mgr_ptr of the ZVarLenBufferMgr object of the ZBlock object, and the last 8 bytes store the starting position offset of the c3 column value stored in the object, which corresponds to the starting address of the length1 value in the ZVarLenBufferMgr object. Length1 occupies 8 bytes, which stores the true length of the c3 column value, and then the byte array of the c3 column value is stored.

[0059] ZVarLenBufferMgr maintains 0 to multiple 64k memory spaces, and stores them in order according to the generation sequence in the form of an array. When a variable-length column value is stored, the first 64k memory space is applied, and then the variable-length column value byte array is stored. When the remaining space in the ZVarLenBufferMgr object is insufficient to store the variable-length column value being inserted, one or more 64k memory spaces are applied.

[0060] The method for reading the c3 column value is as follows:

[0061] First, the 16 bytes of the c3 column position of the Tuple tuple are parsed. According to the first 8 bytes, the ZVarLenBufferMgr object is found, and then according to the last 8 bytes offset, the starting reading position is found. Assuming that the array of 64k memory spaces stored by ZVarLenBufferMgr is block, the formula for locating the reading starting address according to the offset is as follows:

[0062]

[0063] % represents the remainder; valuePtr is the starting address of the stored c3 column value byte array.

[0064] The first 8 bytes of ValuePtr are taken, converted into a long type value, and the length of the c3 column value byte array is obtained, and then the array length bytes are sequentially read. When the 64k memory space is read to the end, but the array length bytes have not yet been read, the next 64k memory space in the block array is continuously read until the array length bytes are read, and the reading operation of the c3 column value is completed.

[0065] Based on the above method, the variable-length type time series data storage processing device in this embodiment includes at least one memory and at least one processor.

[0066] The at least one memory is configured to store a machine-readable program.

[0067] The at least one processor is configured to invoke the machine-readable program to execute a variable-length type time series data storage processing method.

[0068] The above specific embodiments are only specific cases of the present application, and the patent protection scope of the present application includes but is not limited to the above specific embodiments. Any appropriate changes or replacements made by any ordinary skilled person in the art to the variable-length type time series data storage processing method and device according to the claims of the present application shall fall within the patent protection scope of the present application.

[0069] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A method for processing time-series data of variable length type, characterized in that, The time-series data storage engine stores the time-series data collected from the collection points. The collected time-series data is divided into different ZdataPoint objects according to the different collection point sources. The ZdataPoint object contains several ZBlocks. Each ZBlock stores time-series data for a period of time. The time periods between ZBlocks do not overlap. Each ZBlock has one or more 64K fixed-size memory spaces, where the first one is used to store fixed-length values ​​and the other 64K memory spaces are used to store variable-length values. Once the first 64k memory is full, a new ZBlock object needs to be allocated to store the new timing data. The ZBlocks are sorted in order of time periods to form a ZBlock array. There is a background disk write thread that writes the time-series data in the already full ZBlock to the disk. After the disk write is successful, the information record of this time-series data block is appended to the file index. The ZBlock object contains TSDataVector, which stores the tuple values ​​of the records. The length of all tuples in each ZdataPoint object is fixed. A tuple refers to the storage format formed by storing all column values ​​of a record in sequence. The ZBlock object also contains a ZVarLenBufferMgr object, which will request 0 to multiple 64k memory spaces as needed to store values ​​of variable length. To address the differences in the maximum length MAX_LEN of the variable-length type VARBINARY, different storage strategies are formulated. When MAX_LEN < 255, TSDataVector uses a complete 64k memory space for storage, storing the tuples of the records sequentially, and setting a span space at the end of the 64K memory space, where the span space represents memory fragments within the block. When MAX_LEN >= 255 and MAX_LEN < 2048, the TSDataVector uses a 64k memory space to store a portion of the records in sequence, and stores the column values ​​of variable-length columns in the remaining 64k space. The TSDataVector stores the record tuples in sequence from front to back; the space for variable-length column values ​​is allocated sequentially from back to front.

2. The method for processing time-series data of variable length type according to claim 1, characterized in that, When the tuple of the record being inserted uses the span memory space, it means that the current ZBlock object is full and a new ZBlock object needs to be created to store the record being inserted.

3. The method for processing time-series data of variable length type according to claim 2, characterized in that, Span memory fragments in the 64k memory space appear in the middle part of the memory space. When the space of the span is insufficient to store the tuple and variable-length column values ​​of the record being inserted, a new ZBlock object needs to be allocated to store the record being inserted.

4. The method for processing time-series data of variable length type according to claim 3, characterized in that, When MAX_LEN >= 2048, the TSDataVector uses a full 64k memory space for storage, storing tuples of records sequentially. In the tuples, variable-length fields occupy a fixed 16 bytes of space. A span is set at the end of the 64K memory space, and the span represents memory fragments within the block.

5. The method for processing time-series data of variable length type according to claim 4, characterized in that, The method for reading numerical values ​​is to first parse the position bytes of the tuple, find the ZVarLenBufferMgr object, and then find the starting position for reading based on the offset. Assuming that the ZVarLenBufferMgr array storing 64k of memory space is a block, the formula for locating the starting address based on the offset is: ； % indicates modulo; valuePtr is the starting address of the array of bytes storing column values; Take the first 8 bytes of ValuePtr, convert them into long type values ​​to obtain the length of the column value byte array, and then read the bytes of the array length sequentially. When the end of the 64k memory space is reached, but the number of bytes of the array length has not yet been read, continue to read the next 64k memory space in the block array until the number of bytes of the array length is read, and complete the reading operation of the column value.

6. A time-series data storage and processing device for variable-length data, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to perform the method according to any one of claims 1 to 5.

Images