Time sequence database multi-storage management method and time sequence database
By defining the data table structure in the time-series database and using soft link technology, devices are evenly distributed to multiple EntityGroups, solving the problems of data load imbalance and performance improvement in time-series databases with multi-disk storage, and achieving efficient IO utilization and improved query performance.
Patent Information
- Application Number
- CN202510554141.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-29
- Publication Date
- 2025-10-21
AI Technical Summary
When using multiple disks for storage, time series databases face problems such as unbalanced data load, insufficient write performance, low query performance, irrational data organization, and poor disk scalability, which limits overall performance improvements.
By defining a time-series data table structure, time-series data and write-ahead logs (WAL) are divided into multiple EntityGroup subdirectories according to devices, and a round-robin algorithm is used to evenly distribute the data across devices. Combined with soft link technology, data and index information are stored in different time partitions and disk directories, achieving efficient utilization of disk I/O.
It improves the concurrent insertion and query performance of time-series databases, ensures that the amount of data read from each disk is similar, and supports dynamic disk expansion and efficient data organization.
Smart Images
Figure CN120821714A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data storage, and in particular to a time series database multi-storage management method and a time series database. Background Art
[0002] Time series data is a series of data generated continuously over time. Simply put, it is timestamped data. A time series database (TSDB) is a database used to ingest, process, and store timestamped data. A time series database stores a continuous stream of time series data. As time passes, the amount of time series data continues to increase, and the total amount of data stored in the time series database will eventually exceed the maximum capacity of a single disk.
[0003] By mounting multiple disks on a server, a time series database can use these disks to store time series data, addressing the capacity limitations of a single disk. The disk I / O of a server with N disks mounted on it is N times higher than that of a server with only one disk. Therefore, these disks cannot be simply used as storage media; instead, it is important to consider how to fully utilize the disk I / O.
[0004] Storing time series data on multiple disks has the following difficulties:
[0005] (1) How to ensure data load balance and disk space usage balance;
[0006] (2) How to maximize disk writes to improve the insertion performance of time series databases;
[0007] (3) How to evenly distribute hot data to improve the query performance of time series databases;
[0008] (4) How to integrate and organize data from multiple disks;
[0009] (4) How to support the continued addition of disks.
[0010] When a time series database uses multiple disks to store data, how to maximize the IO of all disks to improve the overall performance of the time series database is a technical problem that needs to be solved. Summary of the Invention
[0011] The technical task of the present invention is to address the above shortcomings and provide a time series database multi-storage management method and a time series database to solve the technical problem of how to maximize the IO of all disks to improve the overall performance of the time series database.
[0012] In a first aspect, the present invention provides a multi-storage management method for a time series database, comprising the following steps:
[0013] Define the time series data table structure: Each time series table TsTable in the time series database corresponds to a subdirectory as the TsTable directory. The time series data and WAL data in the time series table TsTable are divided into multiple EntityGroup subdirectories according to the device. The time series data in the EntityGroup is divided into different time partition Partition subdirectories. The data in the Partition is split into multiple segment subdirectories according to a fixed size.
[0014] Configure the disk list: Configure the root directory TSDB for all disks of the time series database and create three subdirectories under the root directory of the disk: Segments, TableDeviceIndex, and EntityGroups. The Segments directory is used to store the segment data subdirectories under the partition, the TableDeviceIndex directory is used to store the Device Index data subdirectory of the TsTable object, and the EntityGroups directory is used to store an EntityGroup subdirectory under the TsTable. The disk where the root directory TSBS is located is the primary storage disk.
[0015] Create a time series table TsTable: Select a disk using the metadata disk selection algorithm, create a subdirectory under the TableDeviceIndex directory of the disk, and use the created subdirectory to store the DeviceIndex information of the time series table TsTable. Create a subdirectory under the EntityGroups directory of each storage disk, and use the created subdirectory to store the EntityGroup of the time series table TsTable. Soft link the created subdirectory to the TSDB directory or TsTable directory of the primary storage disk.
[0016] Device allocation: According to the polling algorithm, newly added devices are assigned to different EntityGroups in turn, and the device's EntityGroup and entity_id information are written to the Device Index;
[0017] Time series data writing: Use the time series data disk selection algorithm to select a disk, create a subdirectory under the Segments directory of the disk, store new segment data in the created subdirectory, and link the created subdirectory to the corresponding Partition directory through a soft link. When the current segment is full, a new segment is created according to the time series data disk selection algorithm;
[0018] New disk processing: Add a new disk record in the configuration file and set the disk status. Regularly check the configuration file with time series data. After the disk is found, start the disk balancing background task, calculate the number of segments that need to be migrated for each disk, select the appropriate segments for migration, create a new EntityGroup directory for each table on the new disk, soft link it to the TsTable directory, and update the disk status.
[0019] Data query: Access the Device Index and EntityGroup of the time series table through the TSDB directory of the primary storage disk. Locate the specific Partition and Segment based on the query conditions, and access the data actually stored on different disks through soft links.
[0020] Preferably, for the defined time series data table structure, a subdirectory is configured under the time series table TsTable directory as a Device Index directory. The Device Index directory stores all device information in the time series table TsTable, the EntityGroup to which the device belongs, entity_id, and index. Entity_id indicates the number of times the corresponding device is inserted into the EntityGroup.
[0021] The time series table TsTable is split into multiple EntityGroups. The number of EntityGroups is equal to the number of storage disks.
[0022] As a best practice, when allocating devices, TsTable evenly distributes the devices to different EntityGroups. The allocation algorithm is to allocate newly added devices to each EntityGroup in turn: the first new device is allocated to the first EntityGroup, the second new device is allocated to the second EntityGroup, and the Nth new device is allocated to the Mth EntityGroup. The calculation formula is as follows:
[0023] M=(N-1)%S+1,
[0024] Where S represents the number of EntityGroups and N represents the Nth new device.
[0025] As a preferred option, the EntityGroup stores the write-ahead log (WAL) and forms partitions based on data time. The algorithm for partitioning the time series data is as follows:
[0026] Partition = ts% 864000,
[0027] Here, ts represents the timestamp of this time series data, in seconds.
[0028] Preferably, the metadata disk selection algorithm includes the following steps:
[0029] Get the available disk list through the available disk list algorithm;
[0030] For each available disk, use Linux commands to calculate the total disk space used by the TableDeviceIndex directory on each disk.
[0031] Select the disk with the smallest total amount as the result of this selection;
[0032] The time series data disk selection algorithm includes the following steps:
[0033] Get the available disk list through the available disk list algorithm;
[0034] For each available disk, use Linux commands to calculate the total disk space used by each disk's Segments directory;
[0035] Select the disk with the smallest total amount as the result of this selection;
[0036] Correspondingly, the available disk list algorithm includes the following steps:
[0037] Obtain a list of all disks through the time series database configuration file disklist.cfg;
[0038] Filter out disks whose status is not normal;
[0039] Use Linux commands to get the used space and free space of each disk;
[0040] Analyze whether the remaining space of each disk is less than the alarm threshold W. If so, filter it out;
[0041] The filtered disks are assembled into a list of available disks.
[0042] Preferably, when configuring the disk list, the time series database initialization startup logic includes the following operations:
[0043] Configure the management configuration file disklist.cfg, add disk records, and set the initial record status field value to NEW;
[0044] When the program starts, it reads the configuration file disklist.cfg. If it finds that the disk record status is NEW, it creates three directories, Segments, TableDeviceIndex, and EntityGroups, in the disk root directory. Then the program sets the disk record status field value to NORM.
[0045] As a preference, the newly added disk processing logic includes the following:
[0046] The user adds a new row of disk records at the end of the configuration file disklist.cfg, and the value of the status field of the record is NEW;
[0047] The time series database program regularly checks the configuration file and creates a disk balancing background task when new disk records are found.
[0048] The disk balancing background task migrates some segment data to the new disk. The number of segments migrated to each disk is:
[0049]
[0050] Among them, S represents the number of segments stored by the disk, N represents the number of newly added disks, and T represents the number of all disks;
[0051] The segment selection method for disk migration is: first sort the subdirectories under the segments directory by creation time, and the i-th segment to be migrated is the M-th segment in the sorted segment list. i indivual:
[0052]
[0053] In the formula, S represents the number of segments stored on the disk, i represents the i-th segment to be migrated, and NUM represents the number of segments to be migrated on the disk;
[0054] After selecting the segment list to be migrated, process them one by one. First, copy a segment subdirectory to the segments directory of the new disk. Then, find the soft link corresponding to the segment in the TSBS root directory and modify the subdirectory pointed to by the soft link.
[0055] If N new disks are added at the same time, the selected segment subdirectory list is split and the i-th segment subdirectory is migrated to the F-th new disk. The calculation formula is as follows:
[0056] F=i%N+1,
[0057] The disk balancing background task creates a new EntityGroup directory for each table on the new disk and soft links it to the TsTable directory;
[0058] Disk balancing background task, the program automatically sets the status of newly added disk records to normal status;
[0059] The disk balancing background task has completed.
[0060] In a second aspect, the present invention provides a time series database, which performs multi-disk data storage based on a time series database multi-storage management method as described in any one of the first aspects.
[0061] The time series database multi-storage management method and time series database of the present invention have the following advantages: by splitting the WAL of the write operation to multiple disks, the performance of concurrent insertion is improved; by evenly distributing the Device Index data of all tables to multiple disks, the performance of concurrent query of multiple tables is improved; by evenly distributing the time series data to multiple disks, the total amount of data read by each disk is similar during high-concurrency query of the program. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0063] The present invention will be further described below with reference to the accompanying drawings.
[0064] Figure 1 This is a schematic diagram of the structure of a time series table TsTable in a time series database multi-storage management method in Example 1;
[0065] Figure 2 This is a schematic diagram of multiple disk directory hierarchies of a time series database in a multi-storage management method for a time series database in Example 1;
[0066] Figure 3 Example 1 is a schematic diagram of a configuration file in a time series database multi-storage management method in which three disks are configured. DETAILED DESCRIPTION
[0067] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments given are not intended to limit the present invention. Unless there is a conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.
[0068] Embodiments of the present invention provide a time series database multi-storage management method and a time series database, which are used to solve the technical problem of how to maximize the IO of all disks to improve the overall performance of the time series database.
[0069] Example 1:
[0070] The present invention provides a multi-storage management method for a time series database, which includes seven steps: defining a time series data table structure, configuring a disk list, creating a time series table TsTable, device allocation, writing time series data, processing newly added disks, and data query.
[0071] Step S100 defines the time series data table structure: each time series table TsTable in the time series database corresponds to a subdirectory as the TsTable directory. The time series data and WAL data in the time series table TsTable are divided into multiple EntityGroup subdirectories according to the device. The time series data in the EntityGroup is divided into different time partition Partition subdirectories. The data in the Partition is split into multiple segment subdirectories according to a fixed size.
[0072] In this embodiment, for the defined time series data table structure, a subdirectory is configured under the time series table TsTable directory as the Device Index directory. The Device Index directory stores all device information in the time series table TsTable, the EntityGroup to which the device belongs, the entity_id, and the index. The entity_id indicates the number of times the corresponding device is inserted into the EntityGroup; the time series table TsTable is split into multiple EntityGroups, and the number of EntityGroups is equal to the number of storage disks.
[0073] like Figure 1 As shown in the figure, the root directory of the time series database is the TSBS directory. Each time series table corresponds to a TsTable subdirectory. The time series data and WAL data in the time series table TsTable are divided into multiple EntityGroup subdirectories according to the device. The time series data in the EntityGroup is divided into different time partition Partition subdirectories. The data in the Partition is split into multiple segment subdirectories according to a fixed size.
[0074] The time series table TsTable directory contains a Device Index subdirectory, which stores all device information in the time series table, the Entity Group to which the device belongs, the entity_id (the device's insertion order in the Entity Group), and index information. When creating a time series table, the metadata disk selection algorithm is used to store the Device Index subdirectory on a disk and link it to the TsTable directory using the Linux system's soft link mechanism.
[0075] The time series table TsTable is split into multiple EntityGroups. The number of EntityGroups is equal to the number of storage disks. One EntityGroup is stored on each disk and linked to the TsTable directory through the Linux system's soft link mechanism. Each EntityGroup contains a WAL (write-ahead log), and the WAL of the time series table can be written concurrently to multiple disks.
[0076] The time series table TsTable generally contains many devices. The TsTable object has a Device Index subdirectory for storing device information and the EntityGroup and entity_id information to which the device belongs. All information about the device can be quickly retrieved based on the primary key of the device.
[0077] Step S200 configures the disk list: configure the root directory TSDB of all disks of the time series database, and create three subdirectories under the root directory of the disk, namely the Segments directory, the TableDeviceIndex directory, and the EntityGroups directory. The Segments directory is used to store the segment data subdirectory under the partition, the TableDeviceIndex directory is used to store the Device Index data subdirectory of the TsTable object, and the EntityGroups directory is used to store an EntityGroup subdirectory under the TsTable. The disk where the root directory TSBS is located is the primary storage disk.
[0078] In this embodiment, when configuring the disk list, the time series database initialization startup logic includes the following operations:
[0079] (1) Configure the management configuration file disklist.cfg, add a disk record, and set the initial record status field value to NEW;
[0080] (2) When the program starts, it reads the configuration file disklist.cfg. If it finds that the disk record status is NEW, it creates three directories, Segments, TableDeviceIndex, and EntityGroups, in the disk root directory. Then the program sets the disk record status field value to NORM.
[0081] like Figure 2As shown in the figure, when starting a time series database, you need to configure the storage path for the database root directory TSBS. The disk where TSBS resides is the master disk. The master disk stores the root directory TSBS for time series data. Under the root directory TSBS is a storage disk configuration file, disklist.cfg, which configures the root directory for each disk. When the database program starts, it verifies whether the three subdirectories Segments, TableDeviceIndex, and EntityGroups exist in each disk's root directory. If they do not exist, they are created.
[0082] Master Disk represents the primary storage disk, and Disk2 represents a common storage disk. Master Disk has an additional directory, the TSDB root directory, compared to Disk2. The directory hierarchy under this directory is TsTable / EntityGroup / Partition / segment, in that order.
[0083] Each storage disk has three subdirectories: Segments, TableDeviceIndex, and EntityGroups. The Segments directory stores the segment data subdirectories under the partition, the TableDeviceIndex directory stores the Device Index data subdirectories of the TsTable object, and the EntityGroups directory stores an EntityGroup subdirectory under the TsTable.
[0084] Step S300 creates the time series table TsTable: select a disk through the metadata disk selection algorithm, create a subdirectory under the TableDeviceIndex directory of the disk, store the DeviceIndex information of the time series table TsTable through the created subdirectory, create a subdirectory under the EntityGroups directory of each storage disk, store the EntityGroup of the time series table TsTable through the created subdirectory, and soft link the created subdirectory to the TSDB directory or TsTable directory of the main storage disk.
[0085] In this embodiment, the metadata disk selection algorithm includes the following steps:
[0086] (1) Obtain the available disk list through the available disk list algorithm;
[0087] (2) For each available disk, use Linux commands to calculate the total disk space used by the TableDeviceIndex directory of each disk;
[0088] (3) Select the disk with the smallest total amount as the result of this selection.
[0089] Correspondingly, the available disk list algorithm includes the following steps:
[0090] (1) Obtain a list of all disks through the time series database configuration file disklist.cfg;
[0091] (2) Filter out disks whose disk status is not normal;
[0092] (3) Use Linux commands to obtain the used space and free space of each disk;
[0093] (4) Analyze whether the remaining space of each disk is less than the alarm threshold W. If so, filter it out;
[0094] (5) The filtered disks are combined into a list of available disks.
[0095] In this embodiment, when a new time series table TsTable is created, a storage disk is selected using the metadata disk selection algorithm. A subdirectory is created under the TableDeviceIndex subdirectory of this storage disk to store the device index information for the table. This subdirectory is then soft-linked to the corresponding table subdirectory in the TSDB directory of the primary storage disk. A subdirectory is then created under the EntityGroups directory of each storage disk to store the EntitiGroups under the TsTable. These subdirectories are then soft-linked to the corresponding table subdirectories in the TSDB directory of the primary storage disk.
[0096] Step S400: Device allocation: According to the polling algorithm, newly added devices are sequentially allocated to different EntityGroups, and the EntityGroup and entity_id information of the device are written into the Device Index.
[0097] In this embodiment, when allocating devices, TsTable evenly distributes the devices to different EntityGroups. The allocation algorithm is to allocate newly added devices to each EntityGroup in turn: the first new device is allocated to the first EntityGroup, the second new device is allocated to the second EntityGroup, and the Nth new device is allocated to the Mth EntityGroup. The calculation formula is as follows:
[0098] M=(N-1)%S+1,
[0099] Where S represents the number of EntityGroups and N represents the Nth new device.
[0100] Step S500: Time series data writing: Use the time series data disk selection algorithm to select a disk, create a subdirectory under the Segments directory of the disk, store new Segment data through the created subdirectory, and link the created subdirectory to the corresponding Partition directory through a soft link. When the current Segment is full, create a new Segment according to the time series data disk selection algorithm.
[0101] The time series data disk selection algorithm includes the following steps:
[0102] (1) Obtain the available disk list through the available disk list algorithm;
[0103] (2) For each available disk, use Linux commands to calculate the total disk space used by each disk Segments directory;
[0104] (3) Select the disk with the smallest total amount as the result of this selection.
[0105] EntityGroup stores the write-ahead log (WAL) and partitions formed by data time. The algorithm for partitioning time series data is as follows:
[0106] Partition = ts% 864000,
[0107] Here, ts represents the timestamp of this time series data, in seconds.
[0108] In this embodiment, when a new Segment is created, a storage disk is selected using the time series data disk selection algorithm, and a subdirectory (the directory name is: tableid_parition_segmentid) is created under the Segments subdirectory of the storage disk to store the Segment data, and this subdirectory is soft-linked to the corresponding Partition directory.
[0109] All subdirectories under the Segments, TableDeviceIndex, and EntityGroups directories on each disk are linked to a certain level of the TSBS directory through soft links (ln-s command). This ensures that all data directories and files can be accessed through the TSBS root directory, regardless of which disk the file is stored on.
[0110] like Figure 3As shown, the configuration file configures three disks and creates three tables. Based on the disk configuration order in the disklist.cfg configuration file, the disk array is composed of [master disk, disk1, disk2]. The first EntityGroup of the first table, TsTable1, is stored on the second disk, disk1; the second EntityGroup is stored on the third disk, disk2; and the third EntityGroup is stored on the first disk, masterdisk. The first EntityGroup of the second table, TsTable2, is stored on the second disk, disk2; the second EntityGroup is stored on the first disk, masterdisk; and the third EntityGroup is stored on the second disk, disk1. The first EntityGroup of the third table, TsTable3, is stored on the first disk, masterdisk; the second EntityGroup is stored on the second disk, disk1; and the third EntityGroup is stored on the third disk, disk2. This distribution method evenly distributes the first EntityGroup of all Tstables across all disks, improving insert performance in scenarios where each TsTable has only one device.
[0111] Step S600 adds new disk processing: add a new disk record in the configuration file, and set the disk status. The time series data regularly detects the configuration file. After the disk is found, the disk balancing background task is started, the number of segments that need to be migrated for each disk is calculated, and the appropriate segments are selected for migration. A new EntityGroup directory is created for each table on the new disk, and a soft link is made to the TsTable directory, and the disk status is updated.
[0112] In this embodiment, the newly added disk processing logic includes the following:
[0113] (1) The user adds a new row of disk records at the end of the configuration file disklist.cfg, and the status field value of the record is NEW;
[0114] (2) The time series database program regularly checks the configuration file and creates a disk balancing background task when a new disk record is found;
[0115] (3) Disk balancing background task, migrates some segment data to the new disk. The number of segments migrated to each disk is:
[0116]
[0117] Among them, S represents the number of segments stored by the disk, N represents the number of newly added disks, and T represents the number of all disks;
[0118] (4) The segment selection method for disk migration is: first, sort the subdirectories under the segments directory by creation time, and the i-th segment to be migrated is the M-th segment in the sorted segment list. i indivual:
[0119]
[0120] In the formula, S represents the number of segments stored on the disk, i represents the i-th segment to be migrated, and NUM represents the number of segments to be migrated on the disk;
[0121] (4) After selecting the segment list to be migrated, process them one by one. First, copy a segment subdirectory to the segments directory of the new disk, then find the soft link corresponding to the segment in the TSBS root directory and modify the subdirectory pointed to by the soft link;
[0122] (5) If there are N newly added disks at the same time, the selected segment subdirectory list is split and the i-th segment subdirectory is migrated to the F-th newly added disk. The calculation formula is as follows:
[0123] F=i%N+1,
[0124] (6) The disk balancing background task creates a new EntityGroup directory for each table on the new disk and soft links it to the TsTable directory;
[0125] (7) Disk balancing background task, the program automatically sets the status of newly added disk records to normal status;
[0126] (8) The disk balancing background task ends.
[0127] Step S700: Data query: Access the Device Index and EntityGroup of the time series table through the TSDB directory of the primary storage disk. According to the query conditions, locate the specific Partition and Segment, and access the data actually stored on different disks through soft links.
[0128] When a new segment needs to be created under the Partition directory, the segment subdirectory is stored on a disk selected by the time series data disk selection algorithm, and then linked to the Partition directory through the Linux system's soft link mechanism.
[0129] The method of this embodiment separates the WAL, metadata, and time series data in a time series database into independent subdirectories and uses different disk selection algorithms to store them in different disk directories, fully utilizing the I / O resources of all disks. The independent subdirectories are linked to the storage hierarchy of the time series database root directory using the Linux system's soft link (ln-s) command. Accessing directory files in the root directory allows direct access to independent subdirectories on different disks.
[0130] Example 2:
[0131] The present invention provides a time series database, which performs multi-disk data storage based on the method disclosed in Example 1.
[0132] The above is a detailed introduction to the time series database multi-storage management method and the time series database provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A multi-storage management method for a time series database, characterized in that: The steps include: Define the time series data table structure: Each time series table TsTable in the time series database corresponds to a subdirectory as the TsTable directory. The time series data and WAL data in the time series table TsTable are divided into multiple EntityGroup subdirectories according to the device. The time series data in the EntityGroup is divided into different time partition Partition subdirectories. The data in the Partition is split into multiple segment subdirectories according to a fixed size. Configure the disk list: Configure the root directory TSDB for all disks of the time series database and create three subdirectories under the root directory of the disk: Segments, TableDeviceIndex, and EntityGroups. The Segments directory is used to store the segment data subdirectories under the partition, the TableDeviceIndex directory is used to store the Device Index data subdirectory of the TsTable object, and the EntityGroups directory is used to store an EntityGroup subdirectory under the TsTable. The disk where the root directory TSBS is located is the primary storage disk. Create a time series table TsTable: Select a disk using the metadata disk selection algorithm, create a subdirectory under the TableDeviceIndex directory of the disk, and use the created subdirectory to store the DeviceIndex information of the time series table TsTable. Create a subdirectory under the EntityGroups directory of each storage disk, and use the created subdirectory to store the EntityGroup of the time series table TsTable. Soft link the created subdirectory to the TSDB directory or TsTable directory of the primary storage disk. Device allocation: According to the polling algorithm, newly added devices are assigned to different EntityGroups in turn, and the device's EntityGroup and entity_id information are written to the Device Index; Time series data writing: Use the time series data disk selection algorithm to select a disk, create a subdirectory under the Segments directory of the disk, store new segment data in the created subdirectory, and link the created subdirectory to the corresponding Partition directory through a soft link. When the current segment is full, a new segment is created according to the time series data disk selection algorithm; New disk processing: Add a new disk record in the configuration file and set the disk status. Regularly check the configuration file with time series data. After the disk is found, start the disk balancing background task, calculate the number of segments that need to be migrated for each disk, select the appropriate segments for migration, create a new EntityGroup directory for each table on the new disk, soft link it to the TsTable directory, and update the disk status. Data query: Access the Device Index and EntityGroup of the time series table through the TSDB directory of the primary storage disk. Locate the specific Partition and Segment based on the query conditions, and access the data actually stored on different disks through soft links.
2. The multi-storage management method for a time series database according to claim 1, characterized in that: For the defined time series data table structure, a subdirectory is configured under the time series table TsTable directory as the Device Index directory. The DeviceIndex directory stores all device information in the time series table TsTable, the EntityGroup to which the device belongs, entity_id, and index. The entity_id indicates the number of times the corresponding device is inserted into the EntityGroup. The time series table TsTable is split into multiple EntityGroups. The number of EntityGroups is equal to the number of storage disks.
3. The multi-storage management method for a time series database according to claim 1, characterized in that: When allocating devices, TsTable evenly divides the devices into different EntityGroups. The allocation algorithm is to allocate newly added devices to each EntityGroup in turn. The first new device is allocated to the first EntityGroup, the second new device is allocated to the second EntityGroup, and the Nth new device is allocated to the Mth EntityGroup. The calculation formula is as follows: M=(N-1)%S+1, Where S represents the number of EntityGroups and N represents the Nth new device.
4. The multi-storage management method for a time series database according to claim 1, characterized in that: EntityGroup stores the write-ahead log (WAL) and partitions formed by data time. The algorithm for partitioning time series data is as follows: Partition = ts% 864000, Here, ts represents the timestamp of this time series data, in seconds.
5. The multi-storage management method for a time series database according to claim 1, characterized in that: The metadata disk selection algorithm includes the following steps: Get the available disk list through the available disk list algorithm; For each available disk, use Linux commands to calculate the total disk space used by the TableDeviceIndex directory on each disk. Select the disk with the smallest total amount as the result of this selection; The time series data disk selection algorithm includes the following steps: Get the available disk list through the available disk list algorithm; For each available disk, use Linux commands to calculate the total disk space used by each disk's Segments directory; Select the disk with the smallest total amount as the result of this selection; Correspondingly, the available disk list algorithm includes the following steps: Obtain a list of all disks through the time series database configuration file disklist.cfg; Filter out disks whose status is not normal; Use Linux commands to get the used space and free space of each disk; Analyze whether the remaining space of each disk is less than the alarm threshold W. If so, filter it out; The filtered disks are assembled into a list of available disks.
6. The multi-storage management method for a time series database according to claim 1, characterized in that: When configuring the disk list, the time series database initialization startup logic includes the following operations: Configure the management configuration file disklist.cfg, add disk records, and set the initial record status field value to NEW; When the program starts, it reads the configuration file disklist.cfg. If it finds that the disk record status is NEW, it creates three directories, Segments, TableDeviceIndex, and EntityGroups, in the disk root directory. Then the program sets the disk record status field value to NORM.
7. The multi-storage management method for a time series database according to claim 1, characterized in that: The newly added disk processing logic includes the following: The user adds a new row of disk records at the end of the configuration file disklist.cfg, and the value of the status field of the record is NEW; The time series database program regularly checks the configuration file and creates a disk balancing background task when new disk records are found. The disk balancing background task migrates some segment data to the new disk. The number of segments migrated to each disk is: Among them, S represents the number of segments stored by the disk, N represents the number of newly added disks, and T represents the number of all disks; The segment selection method for disk migration is: first sort the subdirectories under the segments directory by creation time, and the i-th segment to be migrated is the M-th segment in the sorted segment list. i indivual: In the formula, S represents the number of segments stored on the disk, i represents the i-th segment to be migrated, and NUM represents the number of segments to be migrated on the disk; After selecting the segment list to be migrated, process them one by one. First, copy a segment subdirectory to the segments directory of the new disk. Then, find the soft link corresponding to the segment in the TSBS root directory and modify the subdirectory pointed to by the soft link. If N new disks are added at the same time, the selected segment subdirectory list is split and the i-th segment subdirectory is migrated to the F-th new disk. The calculation formula is as follows: F=i%N+1 The disk balancing background task creates a new EntityGroup directory for each table on the new disk and soft links it to the TsTable directory; Disk balancing background task, the program automatically sets the status of newly added disk records to normal status; The disk balancing background task has completed.
8. A time series database, characterized in that: The time series database performs multi-disk data storage based on a time series database multi-storage management method according to any one of claims 1 to 7.