Hash index-based online ddl implementation method and system for time-series tag table
By employing multi-version management and a global hash index in the time-series tag table, the problem of write and query blocking during DDL operations is solved, achieving high efficiency and stability for online DDL operations.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 上海沄熹科技有限公司
- Filing Date
- 2026-03-20
- Publication Date
- 2026-08-04
AI Technical Summary
In existing technologies, time-series tag tables based on KaiwuDB require the addition of a locking mechanism during the DDL process, which leads to blocking of writes and queries, affecting the efficiency of time-series writes.
It adopts a multi-version management approach, generating a new partition label table with each DDL operation, and using a global hash index for physical data isolation and querying, supporting online DDL operations.
It improves query and write efficiency, supports online DDL operations without blocking record writing and querying in the Tag table, and ensures system continuity and stability.
Smart Images
Figure CN122507724A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database technology, specifically to a method, system, apparatus, and storage medium for online DDL implementation of time-series tag tables based on hash indexes. Background Technology
[0002] The time-series tag table based on KaiwuDB uses a version-based storage method (see logical structure). Figure 1 The hash index is used to store PTags and each column independently. The metadata file of the tag table has only one version file, which cannot distinguish the metadata information of different versions. During the DDL process, a locking mechanism needs to be added, which will severely block the writing and querying of tags and affect the timing of the write. Summary of the Invention
[0003] The technical objective of this invention is to address the above shortcomings by providing a method, system, apparatus, and storage medium for online DDL implementation of time-series Tag tables based on hash indexes. By employing a multi-version management approach for the Tag table, each DDL operation generates a new partitioned tag table, and each DDL operation generates a new metadata file stored in a new partition. The hash index serves as the overall index, improving query and write efficiency. This supports online DDL operations on the Tag table without blocking the writing and querying of Tag records.
[0004] In a first aspect, the present invention provides an online DDL implementation method for time-series tag tables based on hash indexes, comprising the following steps: S1: Establish a versioned partitioned storage mechanism. Each time a DDL operation is performed on the Tag table, an incrementing table version number is generated. An independent partitioned tag table is created for the new version. Data of different versions is physically partitioned and stored according to the table version number to achieve physical isolation of data between versions. Each partitioned tag table contains the complete set of Tag records under that version. S2: A global hash index is used as the total index for storing all versions of the Tag table. The hash index includes the main tag record, the table version number, the record row number of the partition tag table, and the location information of the partition tag table corresponding to each table version number. The total index serves as a unified query entry point and automatically routes to the corresponding partition tag table based on the table version number. S3: Save the metadata file for each DDL operation of the Tag table, distinguishing and storing it by the newly added table version number, so that online DDL operations on the Tag table will not block the normal writing and querying of records in the Tag table.
[0005] One possible solution is to use atomic operations to obtain the version number of the newly added table.
[0006] One possible solution is that if the creation of the partition tag table with table version number V(n+1) and the metadata file storing the DDL fails, the current table version number of the Tag table is Vn, and the data written will be written to the partition tag table with table version number Vn.
[0007] One possible approach is that if the partition label table with version number V(n+1) and the metadata file storing the DDL are successfully created, the data to be written will be written to the partition label table with version number V(n+1).
[0008] One possible approach is to first create a table with version number V1 for the Tag table, which has a partition tag table V1 and DDL metadata V1. The actual row numbers stored for the two records written are row 1 and row 2 in the partition tag table V1.
[0009] Another aspect of the present invention provides an online DDL implementation system for time-series tag tables based on hash indexes, comprising: Partitioning mechanism establishment unit: used to establish a versioned partitioned storage mechanism. Each time a DDL operation is performed on the Tag table, an incrementing table version number is generated, and an independent partition tag table is created for the new version. Data of different versions is physically partitioned and stored according to the table version number to achieve physical isolation of data between versions. Each partition tag table contains the complete set of Tag records under that version. Hash Index Storage Unit: A global hash index is used as the total index for storing all versions of the Tag table. The hash index includes the main tag record, the table version number, the record row number of the partition tag table, and the location information of the partition tag table corresponding to each table version number. The total index serves as a unified query entry point and automatically routes to the corresponding partition tag table based on the table version number. Metadata processing unit: Used to store metadata files for each DDL operation of the Tag table. It is stored by distinguishing the new table version number, so that online DDL operations on the Tag table will not block the normal writing and querying of records in the Tag table.
[0010] In another aspect, the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described online DDL implementation method for a time-series tag table based on a hash index.
[0011] In another aspect, the present invention provides a computer-readable storage medium comprising a computer program that, when executed by a processor, implements the steps of the above-described online DDL implementation method for a time-series tag table based on a hash index.
[0012] The present invention provides a method, system, apparatus, and storage medium for online DDL implementation of time-series tag tables based on hash indexes, which has the following advantages: Given the storage characteristics of tag tables, the tag table hash index is used as the table's overall index. Each DDL version corresponds to an independent partition tag table. Simultaneously, the hash index stores the primary tag, tag table version number, and record row number of the partition tag table. This enables online DDL implementation of tag tables without blocking normal writing and querying of the tag table, and supports online DDL even under conditions of large-scale continuous writing of time-series data. Attached Figure Description
[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0014] The invention will be further described below with reference to the accompanying drawings.
[0015] Figure 1 This is a flowchart of an online DDL implementation method for a time-series tag table based on a hash index according to Embodiment 1 of the present invention; Figure 2 This is a logical structure diagram of the Tag table DDL in the prior art; Figure 3 The following is a logical diagram of the online DDL implementation method of time-series tag table based on hash index according to Embodiment 1 of the present invention; Figure 4 This is a logical structure diagram of the online DDL implementation method for time-series tag tables based on hash index according to Embodiment 1 of the present invention, which executes multiple DDL operations. Figure 5 A flowchart illustrating the support for online DDL in the Tag table according to Embodiment 1 of the present invention; Figure 6 Example diagram of writing to the Tag table during the DDL process according to Embodiment 1 of the present invention; Figure 7 This is a block diagram of the logical structure of an online DDL implementation system for a time-series tag table based on a hash index, according to Embodiment 2 of the present invention. Detailed Implementation
[0016] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.
[0017] Definitions: Primary Tag: Tag is a tag field in time series data. Primary Tag represents a tag field that can uniquely identify a device. Multiple tag fields can be defined as a Primary Tag. In the KaiwuDB database, the Primary Tag is used to identify a unique device. In this article, it is simply referred to as PTag.
[0018] Tag table version number: This is the version number of the record in the Tag table. It is incremented by 1 with each DDL operation to ensure that there are no duplicates.
[0019] Online DDL (Data Definition Language) refers to the ability of a database to perform table structure changes without interrupting existing business operations, stopping database services, and ensuring data consistency. Example
[0020] like Figures 1-6 As shown, the online DDL implementation method for time-series tag tables based on hash indexes provided in this embodiment includes the following steps, considering the storage characteristics of tag tables: S1: Establish a versioned partitioned storage mechanism. Each time a DDL operation is performed on the Tag table, an incrementing table version number is generated. An independent partitioned tag table is created for the new version. Data of different versions is physically partitioned and stored according to the table version number to achieve physical isolation of data between versions. Each partitioned tag table contains the complete set of Tag records under that version.
[0021] Partition Tag table: Tag columns defined by the time-series engine columnar storage. Each PartitionTag contains only a single version of the record. Each Partition Tag table has its own independent data directory (consisting of TableID + Table version number).
[0022] To better manage data and ensure its independence and traceability, we need to establish a versioned partitioned storage mechanism. Specifically, each time a Data Definition Language (DDL) operation is performed on a Tag table, the system automatically generates an incrementing table version number. This incrementing version number acts as a unique identifier, clearly distinguishing different versions of the Tag table. After a new table version number is generated, a dedicated partitioned tag table is created for that new version. This dedicated partitioned tag table acts as a dedicated container, storing only the relevant data for that version. Data from different versions is physically partitioned according to the table version number; that is, each version's data is stored in a different physical location. This method achieves physical isolation between versions, preventing interference and confusion between different versions. Each partitioned tag table contains the complete set of Tag records for that version. Thus, this partitioned tag table covers all Tag-related record information for that version; whether records are added, modified, or deleted, the corresponding content can be found in this partitioned tag table, facilitating subsequent data querying, analysis, and management. Atomic operations are used to obtain the updated table version number.
[0023] S2: A global hash index is used as the total index for storing all versions of the Tag table. The hash index includes the main tag record, the table version number, the record row number of the partition tag table, and the location information of the partition tag table corresponding to each table version number. The total index serves as a unified query entry point and automatically routes to the corresponding partition tag table based on the table version number.
[0024] Throughout the system, a global hash index covering all versions is established, serving as the master index for the Tag table. This hash index contains several key pieces of information: the primary tag record, which is the core identifier for a specific tag; the table version number, used to clearly identify different table versions and distinguish data states at different stages; the row number of the partition tag table, which allows precise location of the specific row within the partition tag table; and the location information of the partition tag table corresponding to each table version number, which accurately indicates the specific location of the corresponding partition tag table in the storage system. This master index serves as a unified query entry point. When a query is requested, the system automatically and accurately routes the query to the corresponding partition tag table based on the input table version number through its internal routing mechanism, thereby efficiently retrieving the required data.
[0025] S3: Save the metadata file for each DDL operation of the Tag table, distinguishing and storing it by the newly added table version number, so that online DDL operations on the Tag table will not block the normal writing and querying of records in the Tag table.
[0026] During system operation, for the Tag table, it is necessary to save the metadata files generated each time a Data Definition Language (DDL) operation is performed. Specifically, to clearly distinguish the metadata files corresponding to different DDL operations, a new table version number is used for differentiated storage. After each DDL operation, a unique table version number is assigned to the corresponding metadata file to ensure the independence and identifiability of each metadata file. Through this series of operations, the system ultimately enables online DDL operations on the Tag table without blocking normal write and query operations on the Tag table, ensuring the continuous and stable operation of the entire system's data processing flow.
[0027] In step S3, the Tag table is initially created with version number V1, containing a partition tag table V1 and DDL metadata V1. The actual stored row numbers for the two records written are row 1 and row 2 in the partition tag table V1. If the creation of the partition tag table with version number V(n+1) and the storage of the DDL metadata file fails, the current version number of the Tag table becomes Vn, and the written data will be written to the partition tag table with version number Vn. If the creation of the partition tag table with version number V(n+1) and the storage of the DDL metadata file succeeds, the written data will be written to the partition tag table with version number V(n+1). An atomic operation is used to obtain the updated table version number.
[0028] like Figure 4The diagram shows the logical structure of the Tag table after two DDL operations. Initially, the Tag table is created as version V1, containing only Partition Tag (V1) and meta file (V1). At this stage, the current version is V1, and the two records written are actually stored as row 1 and row 2 in Partition Tag V1. Then, after executing the DDL to add column k+1, Partition Tag V2 and meta file V2 are added. At this point, the current version is V2, and the two records written are actually stored as row 1 and row 2 in Partition Tag V2. Finally, after executing the DDL to delete column 1, Partition Tag V3 and meta file V3 are added. At this point, the current version is V3. The actual row numbers stored for the two records written are row1 and row2 in Partition Tag V3. When the Primary Tag is specified during the write or query process, the record in the Hash index will be queried in O(1). Once the record is found, the Partition Tag will be found according to the version number corresponding to the Primary Tag, and then the corresponding entire record will be found according to the rowid (row identifier) of the Partition Tag.
[0029] like Figure 5 As shown, a specific process for non-blocking online DDL read / write in this embodiment is as follows: atomic operations are used to obtain the table version number after the addition, such as... Figure 6 As shown, the creation of the Partition Tag V2 object and meta file V2 is not yet complete, so the current actual table version of the Tag table is V1, and data will still be written to the Partition Tag V1 of V1 (e.g., ...). Figure 6 (The middle line points to the two dashed lines of V1). If both Partition Tag V2 and meta file V2 succeed, the atomic operation updates the current actual table version number to V2; otherwise, it fails and returns, and the next table version number added will be V3. Example
[0030] Figure 7 This is a logical structure block diagram of an online DDL implementation system for time-series tag tables based on hash indexes according to Embodiment 2 of the present invention. like Figure 7 As shown, the online DDL implementation system for time-series tag tables based on hash indexes provided in this embodiment is a system that corresponds one-to-one with the method in Embodiment 1, including: Partitioning mechanism establishment unit: used to establish a versioned partitioned storage mechanism. Each time a DDL operation is performed on the Tag table, an incrementing table version number is generated, and an independent partition tag table is created for the new version. Data of different versions is physically partitioned and stored according to the table version number to achieve physical isolation of data between versions. Each partition tag table contains the complete set of Tag records under that version. Hash Index Storage Unit: A global hash index is used as the total index for storing all versions of the Tag table. The hash index includes the main tag record, the table version number, the record row number of the partition tag table, and the location information of the partition tag table corresponding to each table version number. The total index serves as a unified query entry point and automatically routes to the corresponding partition tag table based on the table version number. Metadata processing unit: Used to store metadata files for each DDL operation of the Tag table. It is stored by distinguishing the new table version number, so that online DDL operations on the Tag table will not block the normal writing and querying of records in the Tag table. Example
[0031] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the online DDL implementation method for a time-series tag table based on a hash index as described in Embodiment 1. Example
[0032] A computer-readable storage medium includes a computer program that, when executed by a processor, implements the steps of the online DDL implementation method for time-series tag tables based on hash indexes in Embodiment 1.
[0033] The online DDL implementation method, system, apparatus, and storage medium for time-series tag tables based on hash indexes according to the present invention have been described above by way of example with reference to the accompanying drawings. However, those skilled in the art should understand that various modifications can be made to the online DDL implementation method, system, apparatus, and storage medium for time-series tag tables based on hash indexes proposed in the present invention without departing from the scope of the invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.
Claims
1. A method for online DDL implementation of time-series tag tables based on hash indexes, characterized in that, Includes the following steps: S1: Establish a versioned partitioned storage mechanism. Each time a DDL operation is performed on the Tag table, an incrementing table version number is generated. An independent partitioned tag table is created for the new version. Data of different versions is physically partitioned and stored according to the table version number to achieve physical isolation of data between versions. Each partitioned tag table contains the complete set of Tag records under that version. S2: A global hash index is used as the total index for storing all versions of the Tag table. The hash index includes the main tag record, the table version number, the record row number of the partition tag table, and the location information of the partition tag table corresponding to each table version number. The total index serves as a unified query entry point and automatically routes to the corresponding partition tag table based on the table version number. S3: Save the metadata file for each DDL operation of the Tag table, distinguishing and storing it by the newly added table version number, so that online DDL operations on the Tag table will not block the normal writing and querying of records in the Tag table.
2. The online DDL implementation method for time-series tag tables based on hash indexes according to claim 1, characterized in that, Use atomic operations to obtain the version number of the table after the addition.
3. The online DDL implementation method for time-series tag tables based on hash indexes according to claim 1, characterized in that, If the creation of the partition tag table with table version number V(n+1) and the metadata file storing the DDL fails, the current table version number of the Tag table will be Vn, and the written data will be written to the partition tag table with table version number Vn.
4. The online DDL implementation method for time-series tag tables based on hash indexes according to claim 1, characterized in that, If the partition label table with version number V(n+1) and the metadata file storing the DDL are created successfully, the data to be written will be written to the partition label table with version number V(n+1).
5. The online DDL implementation method for time-series tag tables based on hash indexes according to claim 1, characterized in that, First, the Tag table is created with version number V1, and has partition tag table V1 and DDL metadata V1. The actual stored row numbers of the two records written are row 1 and row 2 in partition tag table V1.
6. An online DDL implementation system for time-series tag tables based on hash indexes, characterized in that, include: Partitioning mechanism establishment unit: used to establish a versioned partitioned storage mechanism. Each time a DDL operation is performed on the Tag table, an incrementing table version number is generated, and an independent partition tag table is created for the new version. Data of different versions is physically partitioned and stored according to the table version number to achieve physical isolation of data between versions. Each partition tag table contains the complete set of Tag records under that version. Hash Index Storage Unit: A global hash index is used as the total index for storing all versions of the Tag table. The hash index includes the main tag record, the table version number, the record row number of the partition tag table, and the location information of the partition tag table corresponding to each table version number. The total index serves as a unified query entry point and automatically routes to the corresponding partition tag table based on the table version number. Metadata processing unit: Used to store metadata files for each DDL operation of the Tag table. It is stored by distinguishing the new table version number, so that online DDL operations on the Tag table will not block the normal writing and querying of records in the Tag table.
7. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the steps of the online DDL implementation method for a time-series tag table based on a hash index as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when executed by a processor, implements the steps of the online DDL implementation method for time-series tag tables based on hash indexes as described in any one of claims 1 to 6.