Hybrid processing method, device and computer equipment for unit and multi-unit time series data

Abstract

The application provides a kind of unit and multi-element time series data hybrid processing method, device and computer equipment, the method comprises: obtaining data to be stored;The timestamp column of multi-element time series data and the value column sequence of each component are serialized;The timestamp column after serialization is stored as time column block, and the value column of each component after serialization is stored as value column block;Update the metadata information of all data blocks of multi-element time series data;Generate the corresponding file time sequence metadata information of time column block, generate the file time sequence metadata information of each value column block;All metadata information is stored in the index area of multi-element storage file after serialization;Multi-element time series data is generated as the key of metadata tree of metadata tree of storage file.The application stores time series data in storage file by the way of generating data block and storing in data area and generating metadata tree and storing in index area, improves the storage and query efficiency of multi-element time series data.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a method, apparatus and computer device for hybrid processing of unit and multi-dimensional time-series data. Background Technology

[0002] Time-series data, also known as time-series data, specifically refers to data columns recorded in chronological order. There are two types of time-series data: single-dimensional time-series data and multi-dimensional time-series data. Single-dimensional time-series data contains only a single timestamp and a single value column; multi-dimensional time-series data contains a single timestamp and multiple components, with each component containing one or more value columns. With the popularization of the Internet of Things (IoT) and the continuous development of industrial technology, the demand for time-series data is becoming increasingly widespread, and the data volume is also growing significantly, creating a need for hybrid storage of single-dimensional and multi-dimensional time-series data.

[0003] Currently, in existing technologies, when time series data is stored in a mixed manner with unit time series data and multi-dimensional time series data, each component of the multi-dimensional time series data is generally stored as a unit time series data. For example, if the multi-dimensional time series data includes a column of timestamps and multiple components, each component is stored as a column of timestamps and a column of all values ​​of the component.

[0004] However, the inventors have discovered at least the following technical problems in the prior art: For multivariate time series data, assuming that the multivariate time series data contains one timestamp column and n components, and each component contains only one value column, for a total of n value columns, n timestamp columns and n value columns will be stored. When storing, (n-1) additional timestamp columns will be stored. When querying, according to the query conditions and timestamp columns, the query will read the value columns of all components in the stored file, and the amount of data that needs to be read will be (n-1) additional timestamp columns, resulting in poor storage and query efficiency for multivariate time series data. Summary of the Invention

[0005] This invention provides a method, apparatus, and computer device for hybrid processing of unit and multi-dimensional time-series data, in order to overcome the problem of poor storage and retrieval efficiency of multi-dimensional time-series data.

[0006] In a first aspect, the present invention provides a method for hybrid processing of unit and multi-dimensional time-series data, comprising:

[0007] Retrieve the data to be stored;

[0008] If the data to be stored is determined to be multi-dimensional time-series data, then the timestamp column and the value column of each component of the multi-dimensional time-series data are serialized;

[0009] The serialized timestamp column is stored as a time column block, and the value column of each serialized component is stored as a value column block, so that the multivariate time series data is stored as multiple data blocks in the data area of ​​the storage file;

[0010] Update the metadata information of all data blocks of the aforementioned multi-dimensional time-series data;

[0011] Based on the metadata information of the time column blocks in all the data blocks, generate the corresponding file time series metadata information of the time column blocks; based on the metadata information of the value column blocks in all the data blocks, generate the file time series metadata information of each value column block.

[0012] The file time series metadata information and metadata information of the time column block, as well as the file time series metadata information and metadata information of each of the value column blocks, are serialized and stored in the index area of ​​the storage file;

[0013] The metadata tree is generated by using the multivariate time-series data as the key of the metadata tree of the stored file.

[0014] In one possible design, the serialized timestamp column is stored as a time column block, and the value column of each serialized component is stored as a value column block, including:

[0015] Write the page data header of the first data page, and write the time data of the serialized timestamp column into the first data page;

[0016] When the size of the serialized timestamp column written in the first data page reaches a set threshold, a new data page is flushed until all the serialized timestamp columns are flushed to generate the time column block.

[0017] Write the page data header of the first page of the data page containing the value column block, and write the serialized value columns into the first page of the data page;

[0018] When the size of the data written in the first data page reaches a set threshold, a new data page is flushed until all the serialized value columns are flushed to generate each value column block.

[0019] In one possible design, after acquiring the data to be stored, the process further includes:

[0020] If the data to be stored is determined to be unit time-series data, then the timestamp column and value column of the unit time-series data are serialized;

[0021] The serialized timestamp column and value column are stored as unit sequence data blocks, so that the unit time series data is stored as the unit sequence data blocks in the data area of ​​the storage file;

[0022] Update the metadata information of the unit sequence data block;

[0023] Based on the metadata information of the unit sequence data block, generate file time series metadata information for the unit sequence data block;

[0024] The file time series metadata information and metadata information of the unit sequence data block are serialized and stored in the index area of ​​the storage file;

[0025] A metadata tree is generated using the name of the unit's time-series data as the key.

[0026] In one possible design, storing the serialized timestamp column and value column as unit sequence data blocks, and storing the unit time-series data as unit sequence data blocks in the data area of ​​the storage file, includes:

[0027] Write the page data header of the first data page, and write the serialized timestamp column and value column into the first data page;

[0028] When the size of the serialized timestamp column written in the first data page reaches a set threshold, a new data page is flushed until the serialized timestamp column and value column are all flushed to generate the unit sequence data block.

[0029] In one possible design, the hybrid processing method for unit and multi-dimensional time-series data provided in the first aspect of the present invention further includes:

[0030] Obtain a query command, wherein the query command contains the key of the multivariate sequence metadata to be queried and the set of components to be queried;

[0031] Based on the key of the metadata tree, obtain the file time series metadata information and metadata information of all time column blocks in the metadata tree;

[0032] Starting from the file time series metadata information of the time column block, traverse the file time series metadata information of the value column block of the multivariate time series data;

[0033] If it is determined that the component name in the file time series metadata information of the value column block exists in the set of components to be queried, then the file time series metadata information of the value column block is retained to obtain the metadata information of all value column blocks in the component containing the value column block, until the metadata information of all value column blocks has been traversed.

[0034] Traverse the metadata information of the time column block to obtain the time column block corresponding to the metadata information of the time column block, and decode the time column block to obtain the timestamp column;

[0035] Based on the metadata information of the value column block, each of the components to be queried is traversed sequentially to obtain the value column block corresponding to the metadata information of the value column block, and each value column block is decoded to obtain the value column of each component to be queried;

[0036] The timestamp column and the value columns of each of the components to be queried are returned as the query result, which is a matrix composed of a timestamp column and the value columns of each of the components to be queried.

[0037] In one possible design, the hybrid processing method for unit and multi-dimensional time-series data provided in the first aspect of the present invention further includes:

[0038] If it is determined that the set of components to be queried contains the component names in the file time series metadata information of all the value column blocks, then after obtaining the file time series metadata information and metadata information of all the time column blocks in the metadata tree according to the key of the metadata tree, the metadata information of the time column blocks is directly traversed to obtain the time column blocks corresponding to the metadata information of the time column blocks, the time column blocks are decoded to obtain the timestamp column and subsequent steps.

[0039] In one possible design, the hybrid processing method for unit and multi-dimensional time-series data provided in the first aspect of the present invention further includes:

[0040] Retrieve query criteria;

[0041] Based on the query conditions and the key of the metadata tree, obtain the file time series metadata information of the unit sequence data block;

[0042] Based on the file time series metadata information of the unit sequence data block, obtain the metadata information of each unit sequence data block;

[0043] Based on the metadata information of each unit sequence data block, obtain all the unit sequence data blocks corresponding to the time series data in the data area of ​​the storage file;

[0044] Each data page in the unit sequence data block is decoded to obtain a column of timestamps and a column of values ​​corresponding to each data page;

[0045] Determine whether each timestamp and the value corresponding to the timestamp column satisfy the query conditions;

[0046] If so, the timestamp column and the corresponding value column will be returned as the query result;

[0047] If not, check the timestamp and the value corresponding to the next data page until all data pages have been queried.

[0048] In a second aspect, the present invention provides a time-series data processing apparatus, comprising:

[0049] The acquisition module is used to acquire data to be stored.

[0050] The judgment and processing module is used to determine whether the data to be stored is multi-dimensional time-series data. If so, the timestamp column and the value column of each component of the multi-dimensional time-series data are serialized.

[0051] The storage module is used to store the serialized timestamp column as a time column block and the value column of each serialized component as a value column block, so as to store the multivariate time series data as multiple data blocks in the data area of ​​the storage file;

[0052] The metadata information update module is used to update the metadata information of all data blocks of the multi-dimensional time-series data;

[0053] The metadata information processing module is used to generate the corresponding file time series metadata information of the time series block based on the metadata information of the time series block in all the data blocks, and to generate the file time series metadata information of each value series block based on the metadata information of each value series block in all the data blocks.

[0054] The metadata information storage module is used to serialize the file time series metadata information and metadata information of the time column block, as well as the file time series metadata information and metadata information of each value column block, and store them in the index area of ​​the storage file, and use the multivariate time series data as the key of the metadata tree of the storage file to generate a metadata tree.

[0055] Thirdly, the present invention provides a computer device, comprising: at least one processor and a memory;

[0056] The memory stores computer-executed instructions;

[0057] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform a hybrid processing method for unit and multi-dimensional time-series data as described in the first aspect and the respective possible designs of the first aspect.

[0058] Fourthly, the present invention provides a computer storage medium storing computer-executable instructions, which, when executed by a processor, implement the network slicing deployment method as described in the first aspect and various possible designs of the first aspect.

[0059] This invention provides a method, apparatus, device, and storage medium for hybrid processing of unit and multi-dimensional time-series data. By generating data blocks and storing them in a data area, and generating metadata trees and storing them in an index area, this invention stores time-series data in a storage file, improving storage and query efficiency. Specifically, in the process of generating data blocks, the timestamp column of the multi-dimensional time-series data only needs to be stored once, eliminating the need for repeated writing. This not only reduces the size of the storage file but also improves the storage efficiency of multi-dimensional time-series data. Simultaneously, the index area of ​​the storage file stores the file time-series metadata information and metadata information for both the time column blocks and the value column blocks. During querying, the corresponding data block's metadata information is first indexed based on the file time-series metadata information, and then the corresponding data block is indexed again based on the data block's metadata information. Only a single independent query is needed to retrieve the corresponding value column and timestamp column, resulting in higher query efficiency. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 A schematic diagram illustrating an application scenario of the hybrid processing method for unit and multi-dimensional time-series data provided in an embodiment of the present invention;

[0062] Figure 2 A flowchart illustrating the hybrid processing method for unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 1 ;

[0063] Figure 2a This is a schematic diagram of the structure of a storage file for time-series data provided in an embodiment of the present invention;

[0064] Figure 3 A flowchart illustrating the hybrid processing method for unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 2 ;

[0065] Figure 4 A flowchart illustrating the hybrid processing method for unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 3 ;

[0066] Figure 5 A flowchart illustrating the hybrid processing method for unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 4 ;

[0067] Figure 6 A flowchart illustrating the hybrid processing method for unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 5 ;

[0068] Figure 7 A flowchart illustrating the hybrid processing method for unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 6 ;

[0069] Figure 8 This is a schematic diagram of the structure of a time-series data processing device provided in an embodiment of the present invention;

[0070] Figure 9 This is a schematic diagram of the hardware structure of a time-series data processing device provided in an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] With the popularization of the Internet of Things and the continuous development of industrial technology, the volume of time-series data will become increasingly massive. Existing time-series data is mainly divided into two types: unit-based time-series data and multi-dimensional time-series data. Unit-based time-series data refers to data containing only one timestamp and one value column; multi-dimensional time-series data refers to data containing one timestamp and multiple components, with each component containing one or more value columns. Because time-series data is collected and stored in chronological order, it is characterized by high write frequency, large data volume, and frequent mixing of unit-based and multi-dimensional time-series data. The inventors discovered during the mixed processing of time-series data that in existing technologies, when processing mixed unit time-series data and multivariate time-series data, each component of the multivariate time-series data is stored as a unit time-series data. Since each component of the multivariate time-series data has the same sampling time point, multiple columns of values ​​correspond to one column of timestamps. Furthermore, when writing the multivariate time-series data to disk, multiple components are already aligned according to their timestamps. This storage method results in the timestamp column being stored repeatedly when writing the multivariate time-series data to disk. During querying, all components of a multivariate time series are usually queried together for analysis. Therefore, the amount of data that needs to be read by this storage method is more than the number of sum value columns minus one column of timestamps, resulting in poor storage and query efficiency for multivariate time-series data.

[0073] To address the poor storage and query efficiency issues arising from the mixed processing of multivariate and unit time-series data, the inventors proposed a technical solution. This solution treats unit time-series data as a special type of multivariate time-series data storage, and then treats all time series as multivariate time-series data storage. However, since unit time-series data only contains a timestamp column and a value column, and does not contain null values, a bitmap is still stored to indicate null values ​​when storing it as a multivariate time series. Furthermore, the timestamp column and the value columns of each component in the multivariate time series are stored in multiple data blocks, each with its own metadata information. For unit time-series data, storing metadata information separately for the timestamp column obviously occupies excessive storage space. Under this storage method, queries for unit time series data are also split into two independent queries for the timestamp column and the value column, still resulting in poor storage and query efficiency.

[0074] To address the aforementioned problems, this invention proposes the following technical solution: different storage and query processes are executed based on the characteristics of unit time-series data and multivariate time-series data. Specifically, for multivariate time-series data, a data block-based storage method is proposed. That is, the timestamp column of the multivariate time-series data is stored as a time column block, and the value columns of each component of the multivariate data are stored as value column blocks, thus storing the multivariate time-series data as multiple data blocks. Simultaneously, metadata information for each data block is generated, and corresponding file time-series metadata information is generated based on the data block metadata information. Metadata keys are also generated based on the multivariate time-series data. The metadata tree keys are used to locate the multivariate time-series data during queries, while the file time-series metadata information and data block metadata information are used to locate the time column blocks or value column blocks of the multivariate time-series data during queries. Compared to existing technologies, this method stores less data and has higher storage efficiency. Furthermore, the corresponding data blocks and time column blocks can be quickly retrieved using the metadata information, improving the query efficiency of multivariate time-series data.

[0075] Figure 1 This is a schematic diagram illustrating an application scenario of the hybrid processing method for unit and multi-dimensional time-series data provided in the embodiments of the present invention, such as... Figure 1 As shown, it includes: terminal 101 and server 102.

[0076] The terminal 101 is used for users to query and display query results. The server 102 is used to acquire data to be stored and store the data, and to store the data in a storage file for the terminal 101 according to the mixed processing method of unit and multi-dimensional time-series data provided in this embodiment.

[0077] Figure 2 A flowchart illustrating the method for hybrid processing of unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 1 The execution entity in this embodiment can be Figure 1 The server 102 in the illustrated embodiment can also be other computer-related devices, and this embodiment is not particularly limited.

[0078] Figure 2a This is a schematic diagram of the data area structure in the storage file of time-series data provided in an embodiment of the present invention. The data area in this embodiment includes value column block 2a1, time column block 2a2, and unit sequence data block 2a3.

[0079] in

[0080] Value column block 2a1 is used to store the value column data of multivariate time series data. The value column block 2a1 includes at least one value data page, and the value data page consists of a page data header and a value data sequence.

[0081] Time column block 2a2 is used to store the timestamp column data of multi-dimensional time series data. The time column block 2a2 includes at least one time data page, and each time data page consists of a data page header and a time data sequence.

[0082] The cell sequence data block 2a3 is used to store the timestamp column and value column data of the cell time series data. The cell sequence data block includes a data block header and at least one data page. The data page includes a page data header, timestamp column data and value column data.

[0083] like Figure 2 As shown, the hybrid processing method for this unit and multi-dimensional time series data includes:

[0084] S201: Obtain the data to be stored.

[0085] In this embodiment of the invention, the data to be stored is generally the data generated after the user has modeled the data.

[0086] Taking multivariate time-series data to be stored as an example, suppose: there are multiple devices on a production line, and each device corresponds to a component. Each device has multiple sensors, and the value column obtained by each sensor can be one or more. The timestamp column corresponding to multiple value columns is the same. Therefore, the data to be stored at this time is the value column of each component obtained by the sensor and a timestamp column. This time-series data will be transmitted to server 102, which is the process of obtaining the data to be stored.

[0087] S202: If it is determined that the data to be stored is multi-dimensional time-series data, then serialize the timestamp column and the value column of each component of the multi-dimensional time-series data.

[0088] In this embodiment of the invention, when determining whether the data to be stored is multivariate time-series data, the server 102 can directly determine whether the data to be stored is multivariate time-series data based on the tags pre-stored during multivariate time-series data modeling, or it can determine whether the data to be stored is multivariate time-series data based on the number of timestamp columns and value columns obtained by the server when acquiring the data to be stored. If the data to be stored is determined to be multivariate time-series data, the timestamp columns and value columns of each component of the multivariate time-series data are first serialized.

[0089] For example, suppose that c1 has N components, and the names of each component are s1, s2, s3...s N The timestamp column of c1 and each component s1, s2, s3...s N Value serialization can serialize timestamp columns and value columns into binary sequences or other computer-readable sequences.

[0090] S203: Store the serialized timestamp column as a time column block, and store the value column of each serialized component as a value column block, so as to store the multivariate time series data as multiple data blocks in the data area of ​​the storage file.

[0091] In this embodiment of the invention, combined with Figure 2a The storage file contains both unit time-series data and multi-dimensional time-series data. The storage file is divided into a data area and an index area. The data area contains data blocks, which are divided into three types: value blocks for storing multi-dimensional time-series data value columns, time blocks for storing multi-dimensional time-series data timestamp columns, and unit sequence data blocks for storing unit time-series data.

[0092] Each value column block includes at least one data page, which is the basic unit for storing time-series data. Each data page consists of a page header and serialized value data written after the page header. Additionally, when a value column contains null values, a bitmap structure representing the null value is stored. Storing a bitmap uses fewer bytes than storing null values, thus reducing the storage space occupied by the file. The time column block includes multiple timestamp pages, each of which includes a page header and serialized timestamp data written after the page header.

[0093] In this embodiment of the invention, a time column block is generated by writing the serialized timestamp column into a storage medium such as a disk. After the time column block is generated, the serialized value column is continuously written into the storage medium such as a disk to generate a value column block. The value column blocks of each component of the multivariate time series data are physically stored immediately after the corresponding time column block. Specifically, the server 102 continuously writes the serialized value column into the disk after the timestamp column on the disk track, ensuring that the data of the same multivariate time series data in the same time domain are continuous on the disk.

[0094] S204: Update the metadata information for all data blocks of the multivariate time series data.

[0095] In this embodiment of the invention, metadata information is data used to describe information such as the location and size of data blocks. The metadata information of a data block corresponds to the data block. The metadata information of a data block includes the metadata information of the time column block and the metadata information of the value column block. When executing step 203, the metadata information of all data blocks of multivariate time series data is updated simultaneously. Each time a data page is written, the location and size information of the values ​​in the data page are updated in real time.

[0096] For example, if the maximum value in the first data page is 3, the maximum value in the second data page is 2, and the maximum value in the third data page is 4, then the maximum value recorded in the updated metadata information will be 4, and so on, until the metadata information of all data blocks is updated.

[0097] S205: Generate the corresponding file time series metadata information for the time series blocks based on the metadata information of the time series blocks in all data blocks; generate the file time series metadata information for each value series block based on the metadata information of each value series block in all data blocks.

[0098] In this embodiment of the invention, the metadata information of the time column block is written in memory but not yet written to the storage file. After the metadata information is written to the storage file, file-level metadata information will be generated, that is, the file time series metadata information corresponding to the metadata information of the time column block in this embodiment of the invention.

[0099] During a query, if a component of a multivariate time-series data contains multiple value columns in a storage file, the number of file time-series metadata information corresponding to the multivariate time-series data is the number of value columns plus one. The file time-series metadata information of a time column block can index the metadata information of one or more time column blocks corresponding to that timestamp column, and then index the metadata information of the value column block corresponding to that time column block. The server 102 indexes the location of the data to be queried in the storage file based on the metadata information of the time column block or the metadata information of the value column block.

[0100] In this embodiment of the invention, based on the metadata information of all data blocks updated in step 204, the server 102 will automatically generate file time series metadata information corresponding to the metadata information of all data blocks in memory. This file time series metadata information includes file time series metadata information of time column blocks and file time series metadata information of value column blocks.

[0101] S206: Serialize the file time series metadata information and metadata information of the time column block, as well as the file time series metadata information and metadata information of each value column block, and store them in the index area of ​​the multi-dimensional storage file.

[0102] In embodiments of the present invention, after generating the file time series metadata information of the time column blocks and the file time series metadata information of the value column blocks, this information needs to be stored in the index area of ​​the storage file. Therefore, it is necessary to serialize the file time series metadata information and metadata information of the time column blocks and the file time series metadata information and metadata information of each value column block. Here, serialization can be to serialize various types of metadata information into binary sequence form, and then write the serialized file time series metadata information and metadata information of the time column blocks and the serialized file time series metadata information and metadata information of each value column block into the disk and store them in the index area of ​​the storage file.

[0103] S207: Generate a metadata tree by using multivariate time-series data as the key of the metadata tree of the stored file.

[0104] In embodiments of the present invention, the metadata tree is generated from the metadata in the storage file, and the metadata tree is generally stored at the end of the storage file. Each timestamp column and each value column of the multivariate time series data is used as the key of the metadata tree. After the metadata tree is generated, it is merged with the metadata tree at the end of the storage file to form a new metadata tree.

[0105] In summary, the hybrid processing method for unit and multivariate time-series data provided in this embodiment stores the timestamp columns of the multivariate time-series data as time column blocks in the data area of ​​the storage file, and stores the value columns of each component as value column blocks in the data area of ​​the storage file. During the data block generation process, the timestamp columns of the multivariate time-series data only need to be stored once, eliminating the need for repeated writing. This not only reduces the size of the storage file but also improves the storage efficiency of the multivariate time-series data. Simultaneously, the file time-series metadata information and metadata information of the time column blocks, as well as the file time-series metadata information and metadata information of the value column blocks, are stored in the index area of ​​the storage file. During querying, the metadata information of the corresponding data block is first indexed based on the file time-series metadata information, and then the corresponding data block is indexed based on the metadata information of the data block. Only a single independent query is needed to retrieve the corresponding value column and timestamp column, resulting in higher query efficiency.

[0106] Figure 3 A flowchart illustrating the hybrid processing method for unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 2 In this embodiment of the invention, in Figure 2 Based on the provided embodiments, the specific implementation methods for storing the serialized timestamp column as a time column block and storing the value column of each serialized component as a value column block in S203 are described in detail. Please refer to [link to relevant documentation]. Figure 3 The method includes:

[0107] S301: Write the page data header of the first data page and write the time data of the serialized timestamp column to the first data page.

[0108] In this embodiment of the invention, a data page is the basic unit of data storage. For time column blocks of multivariate time-series data, a time column block consists of multiple timestamp pages storing time data. Each timestamp page includes a page data header and data serialized from a timestamp column. In this embodiment of the invention, after the timestamp column is serialized into a binary sequence, it needs to be written to a storage medium such as a disk. First, the page data header of a timestamp column is written to the disk. Then, the serialized timestamp column is written after the page data header of the first data page. At this time, the time data of the serialized timestamp column is also written to the disk.

[0109] S302: When the size of the serialized timestamp column written to the first data page reaches the set threshold, a new data page is flushed until all the serialized timestamp columns are flushed to generate a time column block.

[0110] In this embodiment of the invention, the storage capacity of each data page is fixed, namely the set threshold of the data page. When the size of the data written into the first data page in the serialized timestamp column reaches the set threshold of the data page, the serialized timestamp column can no longer be written into the currently flushed data page. Then, a new data page is opened, and the flushing operation of the deserialized timestamp column to generate the timestamp page is performed until all the serialized timestamp columns are flushed to generate a time column block stored in the data area of ​​the storage file.

[0111] S303: Write the page data header of the first page of the value column block and write the serialized value columns to the first page of the data page.

[0112] In this embodiment of the invention, for a value column block of multivariate time-varying data, the value column block consists of multiple data pages storing value data. Each data page includes a page data header and data serialized from the value column. In this embodiment of the invention, after the value column is serialized into a binary sequence, it needs to be written to a storage medium such as a disk. First, the page data header of a data page is written to the disk. Then, the serialized value column is written after the page data header of the first data page. At this time, the value data of the serialized value column is also written to the disk.

[0113] S304: When the size of the data written to the first data page reaches the set threshold, a new data page is flushed until all serialized value columns are flushed to generate value column blocks.

[0114] In summary, in this embodiment, the value columns and timestamp columns of multivariate time-series data are stored by continuously writing data pages to the disk. The value column blocks of all components of the multivariate time-series data are physically stored immediately following the corresponding time column blocks, ensuring that the data of the same multivariate sequence in the same time domain are continuous on the disk, thereby saving the size of the storage file.

[0115] Figure 4 A flowchart illustrating the hybrid processing method for unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 3 In this embodiment of the invention, in Figure 2 Based on the provided embodiments, after obtaining the data to be stored in step S201, if the data to be stored is determined to be cell time-series data, this embodiment of the invention discloses a hybrid processing method for cell and multi-element time-series data for storing cell time-series data. Please refer to [link to relevant documentation]. Figure 4 The method also includes:

[0116] S401: If it is determined that the data to be stored is cell time-series data, then serialize the timestamp column and value column of the cell time-series data.

[0117] In this embodiment of the invention, taking the unit time series data s1 as an example, the time column and value column of the unit time series data s1 need to be serialized into binary sequence form in order to write it to the disk.

[0118] S402: Store the serialized timestamp column and value column as a cell sequence data block, so as to store the cell time series data as a cell sequence data block in the data area of ​​the storage file.

[0119] Combination Figure 2a As indicated in step S203, there are three types of data blocks: time column blocks of multivariate time series data, value column blocks of multivariate time series data, and cell sequence data blocks. Among them, cell sequence data blocks are used to store cell time series data.

[0120] Specifically, a unit sequence data block includes a data block header and multiple data pages. Each data page consists of a page data header and a binary sequence. In this embodiment of the invention, the binary sequence can be the timestamp column and value column serialized in step S401. The serialized timestamp column and value column are stored as one or more unit sequence data blocks in the data area of ​​the storage file.

[0121] S403: Update the metadata information of the unit sequence data block.

[0122] While performing step S402, server 102 automatically updates the metadata information of the unit sequence data block. This metadata information refers to the data that is generated after each unit time-series data of a certain size is written, which describes the information of these unit time-series data, such as the starting position and size of the unit sequence data block. This data is the metadata information of the unit sequence data block.

[0123] For example, if the maximum value in the first data written to the disk is 1, and the maximum value in the subsequent data written is 2, then the maximum value recorded in the metadata information in memory needs to be updated to 2. When subsequent data writing is completed, if a maximum value greater than 2 appears, then the maximum value greater than 2 will be updated in the metadata information; otherwise, it will not be updated.

[0124] S404: Generate file time series metadata information for the unit sequence data block based on the metadata information of the unit sequence data block.

[0125] In an embodiment of the present invention, the metadata information of the unit sequence data block is generated in memory at the same time as the unit sequence data block is generated, and has not yet been written to the storage file on the disk. When the metadata information is written to the disk, file time series metadata information is generated and stored in the index area of ​​the storage file.

[0126] S405: Serialize the file time series metadata information and metadata information of the unit time series data and store them in the index area of ​​the storage file.

[0127] Before writing the file time series metadata information and metadata information of the unit sequence data block to the disk, it is necessary to serialize the file time series metadata information and metadata information of the unit sequence data block before performing the flush operation. For example, a unit time series data corresponds to a file time series metadata information in the index area. A file time metadata information can index the metadata information of one or more unit sequence data blocks corresponding to the timestamp column of the unit time series data. The specific location of the unit sequence data block in the storage file can be located through the metadata information of the unit sequence data block.

[0128] S406: Generate a metadata tree using the name of the cell time series data as the key.

[0129] In an embodiment of the present invention, the metadata tree is generated from the metadata in the storage file, and the metadata tree is generally stored at the end of the storage file. The metadata tree is generated by using s1 of the unit time sequence data s1 as the key of the metadata tree. Then, the generated metadata tree is merged with the metadata tree at the end of the storage file to form a new metadata tree and stored at the end of the storage file.

[0130] In summary, in this embodiment, during the storage of unit time-series data, the unit time-series data is stored as unit sequence data blocks, and file time-series metadata information and metadata information of the unit sequence data blocks are generated during the storage process. In the mixed processing of unit time-series data and multi-dimensional time-series data, the value column and timestamp column of the unit time-series data are no longer stored separately according to the storage method of multi-dimensional time-series data, so that the storage file occupies less storage space.

[0131] Figure 5 A flowchart illustrating the hybrid processing method for unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 4 ,exist Figure 4 Based on the provided embodiments, the specific implementation method of storing the serialized timestamp column and value column as a unit sequence data block in S402, so as to store the unit time series data as a unit sequence data block in the data area of ​​the storage file, is described in detail:

[0132] S501: Write the page data header of the first data page, and write the serialized timestamp column and value column to the first data page;

[0133] S502: When the size of the serialized timestamp column written to the first data page reaches the set threshold, a new data page is flushed until the serialized timestamp column and value column are all flushed to generate a unit sequence data block.

[0134] In an embodiment of the present invention, for unit time-series data, the serialized timestamp column and value column are written to the disk together to generate a unit sequence data block. The writing starts from the first data page. When the size of the serialized timestamp column and value column written in the first data page reaches the set threshold of the data page, a new data page is written until all the serialized timestamp column and value column are written to the disk. The generated unit sequence data block is stored in the data area of ​​the storage file.

[0135] Figure 6 A flowchart illustrating the hybrid processing method for unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 5 In this embodiment of the invention, in Figure 2 Based on the provided embodiments, the method for mixed processing of unit and multi-dimensional time-series data during querying, for multi-dimensional time-series data stored in storage files, further includes:

[0136] S601: Obtain a query command, wherein the query command contains the key of the metadata of the multivariate sequence to be queried and the set of components to be queried.

[0137] In the embodiments of the present invention, it is assumed that: multivariate time series data c1 has N components, each component is named s1, s2, s3...sN, and the data to be queried involves M components. Let the set of components to be queried be QSet={q1, q2, q3...qm}, where M and N are both natural numbers greater than 1, and M is less than or equal to N.

[0138] For example, with N=4 and M=3, a device contains 4 components. The component names of the multivariate time-series data stored by this device over a period of time are, in order, temperature s1, operating speed s2, load s3, and humidity s4. This multivariate time-series data stored by the device is denoted as c1. The components to be queried involve temperature, operating speed, and load. The set of components to be queried is denoted as Qset, where Qset = {temperature q1, operating speed q2, load q3}.

[0139] S602: Based on the key of the metadata tree, retrieve the file time series metadata information and metadata information of all time column blocks in the metadata tree.

[0140] For example, based on step 601, using the multivariate time-series data c1 as the key, the file time-series metadata information and the location of the metadata of the time column blocks are indexed from the metadata tree at the end of the storage file, thereby obtaining the file time-series metadata information of all time column blocks in the metadata tree.

[0141] S603: Starting from the file time series metadata information of the time column block, traverse the file time series metadata information of the value column block of the multivariate time series data.

[0142] Based on the generation process of the file time series metadata information of the time column blocks, starting from the file time series metadata information of the time column blocks, the file time series metadata information of the corresponding multivariate time series data value column blocks is read backwards. The file time series metadata information of the value column blocks corresponds to the metadata of the value column blocks.

[0143] S604: If the component name in the file time series metadata information of the value column block exists in the set of components to be queried, then retain the file time series metadata information of the value column block to obtain the metadata information of all value column blocks in the component containing the value column block, until the metadata information of all value column blocks has been traversed.

[0144] Before obtaining the metadata information of each value column of the multivariate time series data, it is necessary to first check whether the component name recorded in the file time series metadata information of each value column block exists in the component set to be queried. The determination is based on whether the recorded component name is the same as the name in the component set to be queried. If they are the same, they are kept in the component set to be queried; otherwise, they are not saved. The above operation is repeated until the metadata information of all value column blocks has been traversed.

[0145] S605: Traverse the metadata information of the time column block to obtain the time column block corresponding to the metadata information of the time column block, decode the time column block to obtain the timestamp column.

[0146] In an embodiment of the present invention, the metadata information of the time column block can be indexed to the location of the time column block in the storage file. Then, by decoding the time column block, the timestamp column can be obtained, and the metadata information of all value column blocks corresponding to the metadata information of one traversal of the time column block can be obtained. The corresponding value column blocks include the metadata information of all value column blocks with query categories.

[0147] S606: Based on the metadata information of the value column block, traverse each component to be queried in sequence to obtain the value column block corresponding to the metadata information of the value column block, decode each value column block to obtain the value column of each component to be queried.

[0148] When the data written to the first data page in the serialized value column reaches the set threshold of the data page, the remaining serialized timestamp columns can no longer be written to the currently flushed data page. A new data page is then opened, and the deserialized value columns are flushed to the disk. This process continues until all serialized value columns are flushed, generating a time column block stored in the data area of ​​the storage file. In this embodiment, when a bitmap exists in a value column block of a component being flushed, this bitmap is used to represent null values ​​in the value column. In this case, the existing bitmap needs to be serialized after the page data header, and then the serialized value columns after the bitmap are flushed to the data page, thus storing the value column block in the storage file.

[0149] In this embodiment of the invention, the metadata information of the acquired time column blocks is first traversed, and the information and value columns are obtained according to the information recorded in the metadata information. The information recorded in the metadata information may be the starting position and size of the data block in the file. The time column blocks in the file can be indexed and read through this information. The read time column data blocks are decoded to obtain the timestamp column. Then, the metadata information of the value column data blocks of all components to be queried corresponding to this iteration is obtained.

[0150] S607: Returns the timestamp column and the value columns of each component to be queried as the query results.

[0151] In this embodiment of the invention, the query result in step S605 is a matrix composed of a column of timestamps and the value columns of each queried component. For example, the metadata information of each queried component is traversed sequentially. Based on the starting position and size of the recorded value column data blocks, the corresponding value column data blocks are indexed and read. Then, the value column data blocks are decoded. Since decoding the value column data blocks yields value columns and a bitmap, during the decoding process, empty values ​​are automatically added to the positions of the bitmap in the returned value of the value column based on the content of the bitmap. After obtaining the time column and all the value columns filled with empty values, no timestamp alignment is needed; the timestamp column and the M value columns are directly concatenated into a matrix, and the query result is returned.

[0152] Furthermore, when M=N, i.e., when all components of the multivariate time series data are available, according to the storage method of multivariate time series data, the value columns of all components of the multivariate time series data are physically stored immediately following the corresponding time columns, ensuring that the data of the same multivariate sequence in the same time domain are contiguous on the disk. Therefore, combined with... Figure 6 The hybrid processing method for unit and multi-dimensional time series data provided in this embodiment of the invention can be optimized as follows when M=N:

[0153] If it is determined that the set of components to be queried contains the component names in the file time series metadata information of all value column blocks, then after obtaining the file time series metadata information and metadata information of all time column blocks in the metadata tree according to the key of the metadata tree, there is no need to execute the step of traversing the metadata information of the time column blocks to obtain the time column blocks corresponding to the metadata information of the time column blocks, decoding the time column blocks to obtain the timestamp column and subsequent steps. This can save the time of querying multivariate time series data.

[0154] In summary, this embodiment, through traversal, firstly, the component to be queried is retained from all the components of the multivariate time-series data into the set of components to be queried. Then, the file time-series metadata information of the data block of the component to be queried is read out. Finally, the position of the data block in the file is found based on the metadata information of the data block. Then, the queried data block is decoded to obtain a timestamp column and a value column corresponding to the timestamp column and padded with null values. Finally, without the need for timestamp alignment operations, the timestamp column and the value columns can be concatenated into a matrix as the query structure and returned to terminal 101. This query does not require timestamp alignment operations, making the query more efficient.

[0155] Figure 7 A flowchart illustrating the hybrid processing method for unit and multi-dimensional time-series data provided in this embodiment of the invention. Figure 6 In this embodiment of the invention, in Figure 2 Based on the provided embodiments, when querying cell time-series data stored in a storage file, the method for mixed processing of cell and multi-dimensional time-series data further includes:

[0156] S701: Get query conditions.

[0157] S702: Obtain the file time series metadata information of the cell sequence data block based on the query conditions and the key of the metadata tree.

[0158] S703: Obtain the metadata information for each unit sequence data block based on the file time series metadata information of the unit sequence data block.

[0159] In this embodiment of the invention, only unit sequence data blocks are generated during the storage of unit time-series data. Therefore, when querying unit time-series data, the query conditions can be a specific time point or a specific unit time-series data name. For example, the unit time-series data to be queried is unit time-series data s1. This unit time-series data s1 can be data detected by the device at one of the measurement points. This measurement point can be one of the sensors in the device. When querying the data obtained by this sensor, the file time-series metadata information of the unit sequence data block in the index area is first indexed from the metadata tree at the end of the storage file using s1 as the key, thereby obtaining the file time-series metadata information of the unit sequence data block. Since the file time-series metadata information of the unit sequence data block corresponds to the metadata information of the unit sequence data block, the metadata information of the unit sequence data block can be further indexed and obtained.

[0160] S704: Based on the metadata information of each unit sequence data block, obtain all the unit sequence data blocks corresponding to the unit time series in the data area of ​​the storage file.

[0161] In this embodiment of the invention, the metadata information of each unit sequence data block records the basic characteristic information describing the unit sequence data block, such as the starting position and size of each unit sequence data block. Therefore, based on this descriptive information, all unit sequence data blocks in the data area of ​​the disk can be read based on the metadata information of each unit sequence data block.

[0162] S705: Decode each data page in the unit sequence data block to obtain a column of timestamps and a column of values ​​corresponding to each data page.

[0163] S706: Determine whether each timestamp and the value corresponding to the timestamp column meet the query conditions.

[0164] S707: If so, the timestamp column and the corresponding value column of the timestamp column will be returned as the query result.

[0165] S708: If not, check the timestamp and the corresponding value for the next data page until all data pages have been queried.

[0166] In the embodiments of the present invention, since the unit time-series data only contains a column of timestamps and a column of values ​​when modeling, and there are no null values, each page will generate a column of timestamps and a column of values ​​in memory after decoding, without null values. Then, it is only necessary to determine whether each data point meets the query filtering conditions. The data point can be a certain timestamp and / or a certain value, and the filtering conditions can be a time point and / or a value. If the data point meets the filtering conditions, the query timestamp column and value column obtained are returned as the query result.

[0167] In summary, in this embodiment, when querying unit time series data, the file time series metadata information of the unit sequence data block can be quickly indexed through the key of the metadata tree, thereby obtaining the metadata information of the unit sequence data block. Finally, based on the metadata information of the unit sequence data block, the unit sequence data block is obtained. Then, the unit sequence data block is decoded to obtain a column of timestamps and a column of values. Finally, based on the query conditions and filtering conditions, the timestamp column and value column that meet the conditions can be obtained as the query result and returned to the terminal 101.

[0168] Figure 8 This is a schematic diagram of the structure of a time-series data processing device provided in an embodiment of the present invention. Figure 8 As shown, the time-series data processing device 80 includes: an acquisition module 801, a judgment processing module 802, a storage module 803, a metadata information update module 804, a metadata information processing module 805, and a metadata information storage module 806.

[0169] Module 801 is used to acquire data to be stored.

[0170] The judgment and processing module 802 is used to determine whether the data to be stored is multi-dimensional time series data. If so, the timestamp column and the value column of each component of the multi-dimensional time series data are serialized.

[0171] Storage module 803 is used to store the serialized timestamp column as a time column block and the value column of each serialized component as a value column block, so as to store the multi-dimensional time series data as multiple data blocks in the data area of ​​the storage file;

[0172] Metadata information update module 804 is used to update the metadata information of all data blocks of multivariate time series data;

[0173] The metadata information processing module 805 is used to generate the corresponding file time series metadata information of the time column block based on the metadata information of the time column block in all data blocks, and to generate the file time series metadata information of each value column block based on the metadata information of each value column block in all data blocks.

[0174] The metadata information storage module 806 is used to serialize the file time series metadata information and metadata information of the time column block, as well as the file time series metadata information and metadata information of each value column block, and store them in the index area of ​​the multi-dimensional storage file. The multi-dimensional time series data is used as the key of the metadata tree of the storage file to generate a metadata tree.

[0175] In one possible implementation, storage module 803 is specifically used to write the time data of the serialized timestamp column into the first data page after writing the page data header. When the size of the serialized timestamp column written into the first data page reaches a set threshold, a new data page is flushed until all the serialized timestamp columns are flushed to generate the time column. Storage module 803 is also specifically used to write the page data header of the first data page of the value column block and write each serialized value column into the first data page. When the size of the data written into the first data page reaches a set threshold, a new data page is flushed until all the serialized value columns are flushed to generate the value column blocks.

[0176] In one possible implementation, after the acquisition module 801 acquires the data to be stored, the judgment and processing module 802 is further used to determine that the data to be stored is unit time-series data, and serialize the timestamp column and value column of the unit time-series data; the storage module 803 is further used to store the serialized timestamp column and value column as unit sequence data blocks, so as to store the unit time-series data as unit sequence data blocks in the data area of ​​the storage file; the metadata information update module 804 is further used to generate file time-series metadata information of the unit sequence data blocks according to the metadata information of the unit sequence data blocks; the metadata information processing module 805 is further used to serialize the file time-series metadata information and metadata information of the unit sequence data blocks and store them in the index area of ​​the storage file; the metadata information storage module 806 is further used to generate a metadata tree with the name of the unit time-series data as the key.

[0177] In one possible implementation, for the processing of unit time-series data, the storage module 803 in the time-series data processing device is specifically used to write the page data header of the first page of data, and write the serialized timestamp column and value column into the first page of data. When the size of the serialized timestamp column written into the first page of data reaches a set threshold, a new data page is flushed until all the serialized timestamp column and value column are flushed to generate a unit sequence data block.

[0178] In one possible implementation, the time-series data processing device further includes a query module 807.

[0179] The query module 807 is used to retrieve the file time series metadata information and metadata information of all time column blocks in the metadata tree based on the key of the metadata tree; starting from the file time series metadata information of the time column blocks, it traverses the file time series metadata information of the value column blocks of the multivariate time series data; if it is determined that the component name in the file time series metadata information of the value column block exists in the set of components to be queried, the file time series metadata information of the value column block is retained to obtain the metadata information of all value column blocks in the component containing the value column block, until the metadata information of all value column blocks has been traversed; it traverses the metadata information of the time column blocks to obtain the time column blocks corresponding to the metadata information of the time column blocks, decodes the time column blocks to obtain the timestamp column; it traverses each component to be queried in sequence according to the metadata information of the value column blocks to obtain the value column blocks corresponding to the metadata information of the value column blocks, decodes each value column block to obtain the value column of each component to be queried; and returns the timestamp column and the value column of each component to be queried as the query result, which is a matrix composed of a column of timestamps and the value columns of each component to be queried.

[0180] In one possible implementation, if it is determined that the set of components to be queried contains the component names in the file time series metadata information of all value column blocks, then when querying multivariate time series data, after obtaining the file time series metadata information and metadata information of all time column blocks in the metadata tree according to the key of the metadata tree, the query module 807 directly executes the traversal of the metadata information of the time column blocks to obtain the time column blocks corresponding to the metadata information of the time column blocks, decodes the time column blocks to obtain the timestamp column and subsequent steps.

[0181] In one possible implementation, for unit time-series data, the query module 807 of the time-series data processing device is further used to query the unit time-series data. The query module is used to obtain query conditions; obtain file time-series metadata information of unit sequence data blocks according to the query conditions and the key of the metadata tree; obtain metadata information of each unit sequence data block according to the file time-series metadata information of the unit sequence data blocks; obtain all unit sequence data blocks corresponding to the unit time-series data in the data area of ​​the storage file according to the metadata information of each unit sequence data block; decode each data page in the unit sequence data block to obtain a column of timestamps and a column of values ​​corresponding to each data page; determine whether each timestamp and the value corresponding to the timestamp meet the query conditions; if so, return the timestamp column and the value column corresponding to the timestamp as the query result; if not, check the timestamp and the value corresponding to the timestamp of the next data page until all data pages have been queried.

[0182] The apparatus provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0183] Figure 9 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. (See diagram below.) Figure 9 As shown, the computer device 90 of this embodiment includes: a processor 901 and a memory 902; wherein

[0184] Processor 901 is used to store instructions for computer execution;

[0185] Memory 902; used to execute computer execution instructions stored in the memory to implement the various steps executed by server 102 in the above embodiments. See the relevant descriptions in the foregoing method embodiments for details.

[0186] Alternatively, the memory 902 can be either standalone or integrated with the processor 901.

[0187] When the memory 902 is set up independently, the computer device also includes a bus 903 for connecting the memory 902 and the processor 901.

[0188] This invention also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the above-described method for mixed processing of unit and multi-dimensional time-series data.

[0189] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for mixed processing of unit and multi-dimensional time-series data.

[0190] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0191] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0192] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0193] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute partial steps of the methods of the various embodiments of this application.

[0194] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0195] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0196] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0197] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0198] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0199] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of mixed processing of single and multiple time series data, characterized by, include: Retrieve the data to be stored; If the data to be stored is determined to be multi-dimensional time-series data, then the timestamp column and the value column of each component of the multi-dimensional time-series data are serialized; The serialized timestamp column is stored as a time column block, and the value column of each serialized component is stored as a value column block, so that the multivariate time series data is stored as multiple data blocks in the data area of ​​the storage file; Update the metadata information of all data blocks of the aforementioned multi-dimensional time-series data; Based on the metadata information of the time column blocks in all the data blocks, generate the corresponding file time series metadata information of the time column blocks; based on the metadata information of the value column blocks in all the data blocks, generate the file time series metadata information of each value column block. The file time series metadata information and metadata information of the time column block, as well as the file time series metadata information and metadata information of each of the value column blocks, are serialized and stored in the index area of ​​the storage file; The metadata tree is generated by using the multivariate time-series data as the key of the metadata tree of the stored file; The step of storing the serialized timestamp column as a time column block and storing the value column of each serialized component as a value column block includes: Write the page data header of the first data page, and write the time data of the serialized timestamp column into the first data page; When the size of the serialized timestamp column written in the first data page reaches a set threshold, a new data page is flushed until all the serialized timestamp columns are flushed to generate the time column block. Write the page data header of the first page of the data page containing the value column block, and write the serialized value columns into the first page of the data page; When the size of the data written in the first data page reaches a set threshold, a new data page is flushed until all the serialized value columns are flushed to generate each value column block.

2. The method of claim 1, wherein, After acquiring the data to be stored, the process also includes: If the data to be stored is determined to be unit time-series data, then the timestamp column and value column of the unit time-series data are serialized; The serialized timestamp column and value column are stored as unit sequence data blocks, so that the unit time series data is stored as the unit sequence data blocks in the data area of ​​the storage file; Update the metadata information of the unit sequence data block; Based on the metadata information of the unit sequence data block, generate file time series metadata information for the unit sequence data block; The file time series metadata information and metadata information of the unit sequence data block are serialized and stored in the index area of ​​the storage file; A metadata tree is generated using the name of the unit's time-series data as the key.

3. The method of claim 2, wherein, The step of storing the serialized timestamp column and value column as unit sequence data blocks, so as to store the unit time-series data as unit sequence data blocks in the data area of ​​the storage file, includes: Write the page data header of the first data page, and write the serialized timestamp column and value column into the first data page; When the size of the serialized timestamp column written in the first data page reaches a set threshold, a new data page is flushed until the serialized timestamp column and value column are all flushed to generate the unit sequence data block.

4. The method of claim 1, wherein, Also includes: Obtain a query command, wherein the query command contains the key of the multivariate sequence metadata to be queried and the set of components to be queried; Based on the key of the metadata tree, obtain the file time series metadata information and metadata information of all time column blocks in the metadata tree; Starting from the file time series metadata information of the time column block, traverse the file time series metadata information of the value column block of the multivariate time series data; If it is determined that the component name in the file time series metadata information of the value column block exists in the set of components to be queried, then the file time series metadata information of the value column block is retained to obtain the metadata information of all value column blocks in the component containing the value column block, until the metadata information of all value column blocks has been traversed. Traverse the metadata information of the time column block to obtain the time column block corresponding to the metadata information of the time column block, and decode the time column block to obtain the timestamp column; Based on the metadata information of the value column block, each of the components to be queried is traversed sequentially to obtain the value column block corresponding to the metadata information of the value column block, and each value column block is decoded to obtain the value column of each component to be queried; The timestamp column and the value columns of each of the components to be queried are returned as the query result, which is a matrix composed of a timestamp column and the value columns of each of the components to be queried.

5. The method of claim 4, wherein, Also includes: If it is determined that the set of components to be queried contains the component names in the file time series metadata information of all the value column blocks, then after obtaining the file time series metadata information and metadata information of all the time column blocks in the metadata tree according to the key of the metadata tree, the metadata information of the time column blocks is directly traversed to obtain the time column blocks corresponding to the metadata information of the time column blocks, the time column blocks are decoded to obtain the timestamp column and subsequent steps.

6. The method of claim 3, wherein, Also includes: Retrieve query criteria; Based on the query conditions and the key of the metadata tree, obtain the file time series metadata information of the unit sequence data block; Based on the file time series metadata information of the unit sequence data block, obtain the metadata information of each unit sequence data block; Based on the metadata information of each unit sequence data block, obtain all the unit sequence data blocks corresponding to the unit time series data in the data area of ​​the storage file; Each data page in the unit sequence data block is decoded to obtain a column of timestamps and a column of values ​​corresponding to each data page; Determine whether each timestamp and the value corresponding to the timestamp satisfy the query condition; If so, the timestamp and the value corresponding to the timestamp will be returned as the query result; If not, check the timestamp and the value corresponding to the next data page until all data pages have been queried.

7. An apparatus for processing time series data, characterized by include: The acquisition module is used to acquire data to be stored. The judgment and processing module is used to determine whether the data to be stored is multi-dimensional time-series data. If so, the timestamp column and the value column of each component of the multi-dimensional time-series data are serialized. The storage module is used to store the serialized timestamp column as a time column block and the value column of each serialized component as a value column block, so as to store the multivariate time series data as multiple data blocks in the data area of ​​the storage file; The metadata information update module is used to update the metadata information of all data blocks of the multi-dimensional time-series data; The metadata information processing module is used to generate the corresponding file time series metadata information of the time series block based on the metadata information of the time series block in all the data blocks, and to generate the file time series metadata information of each value series block based on the metadata information of each value series block in all the data blocks. The metadata information storage module is used to serialize the file time series metadata information and metadata information of the time column block, as well as the file time series metadata information and metadata information of each value column block, and store them in the index area of ​​the storage file, and use the multivariate time series data as the key of the metadata tree of the storage file to generate a metadata tree; The storage module is specifically configured to: write the time data of the serialized timestamp column into the first data page after writing the page data header of the first data page; when the size of the serialized timestamp column written in the first data page reaches a set threshold, refresh the new data page until all the serialized timestamp columns are refreshed to generate the time column block; write the page data header of the first data page of the value column block, and write each serialized value column into the first data page; when the size of the data written in the first data page reaches a set threshold, refresh the new data page until all the serialized value columns are refreshed to generate each value column block.

8. A computer device, comprising: include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the hybrid processing method of unit and multi-dimensional time-series data as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by the processor, implement the hybrid processing method of unit and multi-dimensional time-series data as described in any one of claims 1 to 6.

Images