Method for dynamically modifying Tdensine table structure

Abstract

The invention relates to the technical field of databases, in particular to a method for dynamically modifying a Tdensine table structure. According to the method, a shadow table is used as an intermediate carrier, through the full-process design of shadow table creation, data double writing, historical data migration, data consistency verification, flow switching and old table cleaning, on the premise that business services are not stopped and real-time writing and query of data are not affected, safe and efficient dynamic updating of a Tdensine table structure is achieved, a strategy of changing space into time is adopted, and the dynamic updating efficiency of the Tdensine table structure is improved. Business blocking caused by table locking is avoided, data integrity and consistency are guaranteed through multiple mechanisms, and the method has remarkable practicability and technical innovation.

Description

Technical Field

[0001] This invention relates to the field of database technology, and more specifically, to a method for dynamically modifying the structure of a Tdengine table. Background Technology

[0002] Tdengine, a high-performance time-series database, is widely used in various scenarios requiring the storage of massive amounts of time-series data due to its advantages such as high write throughput, low storage cost, and support for functions specific to time-series data. In practical applications, as business requirements iterate (such as adding equipment monitoring indicators or adjusting data collection dimensions), it is often necessary to modify the table structure (especially the super table structure) in Tdengine. Common operations include adding columns, deleting columns, and adjusting field data types.

[0003] In existing technologies, modifying the Tdengine table structure typically uses SQL statements such as ALTER TABLE tb_name ADD COLUMNfield_name data_type or ALTER TABLE tb_name DROP COLUMN field_name.

[0004] The existing technology has the following technical defects, specifically:

[0005] 1. Serious business interruption issues: When Tdengine executes the ALTER TABLE command, it locks the target table with an exclusive lock. During the lock period, any write operations are prohibited, and only partial read operations are allowed. The larger the amount of data stored in the table, the longer the table lock time. During this period, the business system cannot write real-time data to the table, resulting in interruption of data collection, which in turn causes problems such as monitoring system alarm failure and missing business statistics data, seriously affecting business continuity.

[0006] 2. High risk of data inconsistency: To avoid the impact of prolonged table locking on business operations, some technical personnel may choose to stop business services first and then execute the ALTER TABLE command to modify the table structure. This method will cause business data generated during the service stoppage to be unable to be written to the database in real time. If the data is not temporarily stored through caching or other means, it will directly cause data loss. Even if it is temporarily stored through caching, when the cached data is written to the database in batches after the service is restored, the database may be overloaded due to the large amount of data, which may further cause data write failures and damage data integrity and consistency.

[0007] 3. No safe rollback mechanism: If an abnormal situation such as database service failure or network interruption occurs during the execution of the existing ALTER TABLE command, the table structure may be in a "semi-modified" state (such as some columns have been added but have not taken effect, or the original column data is corrupted). At this time, it is not possible to directly roll back to the table structure before the modification. Technical personnel need to restore it by backing up the data, which is complicated and time-consuming, further extending the business interruption time. Summary of the Invention

[0008] The purpose of this invention is to provide a method for dynamically modifying the Tdengine table structure to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention aims to provide a method for dynamically modifying the Tdengine table structure, including: S1, shadow table creation: parsing the table structure update request, generating a shadow table containing change information based on the original table structure, and naming it according to rules.

[0010] S2, Dual Data Write: After the shadow table is created, configure the business system to write real-time data to both the original table and the shadow table simultaneously, while query requests are still handled by the original table.

[0011] S3. Historical Data Migration: Based on the dual-write data structure, historical data from the original table before the creation of the shadow table is migrated to the shadow table in batches and at a controlled rate.

[0012] S4. Data Consistency Verification: After the historical data migration is completed, verify whether the data in the original table and the shadow table are consistent from two dimensions: historical data and real-time data. If the verification is inconsistent, repair and re-verify or trigger rollback.

[0013] S5. Traffic Switching: After the data consistency verification passes, a gradual strategy is adopted to switch business traffic from the original table to the shadow table. After the full switch, the shadow table becomes the new master table.

[0014] S6. Cleaning up and confirming the status of old tables: After the traffic switch is completed and the new master table is running stably, the dual-write logic is taken offline, and the original tables are archived or deleted as needed.

[0015] S7. Full process rollback: If an anomaly occurs at any stage, stop the relevant tasks, delete the shadow table, and restore the original table route to enable the business to recover quickly.

[0016] As a further improvement to this technical solution, the shadow table creation is specifically implemented as follows: receiving a table structure update request submitted by the user, which includes the target original table name, specific structure change instructions and attribute information of the changed fields; based on the target original table name, executing the SQL instruction SHOWCREATESTABLE[original table name] to query and obtain the complete table creation statement of the original table, which includes the field list, field attributes, tag information and database affiliation of the original table.

[0017] According to the structure change instruction, the original table creation statement is modified: if the change instruction is to add a column, the field to be added and its attributes are added to the original table field list; if the change instruction is to delete a column, the definition of the field to be deleted is removed from the original table field list.

[0018] Generate a shadow table name according to the naming rule "original table name_shadow_v{version number}", execute the modified table creation statement, and complete the creation of the shadow table.

[0019] As a further improvement to this technical solution, the dual data writing is specifically implemented as follows: Configure the data writing logic of the business system, and while keeping the normal writing process of the original table unchanged, start an additional asynchronous writing thread. The asynchronous writing thread and the original table writing thread are executed in parallel, and the same business data is synchronously written to the shadow table created by S1 in real time. The execution priority of the asynchronous writing thread is lower than that of the original table writing thread. Keep the routing strategy of business query requests unchanged, and all business query requests are directed to the original table for processing.

[0020] As a further improvement to this technical solution, the historical data migration is specifically implemented as follows:

[0021] S3.1. Start an independent historical data migration task. The migration task is deployed on a dedicated processing node that is separate from the core business server to avoid occupying business operation resources.

[0022] S3.2 Determine the scope of historical data migration: Take the creation timestamp of the shadow table in S1 as the base time point, denoted as t0, and the migration scope is all historical data in the original table whose timestamp field ts <= t0.

[0023] S3.3. Adopt a "batch-based, rate-controlled" migration strategy: Divide historical data into multiple migration batches according to time intervals. After each batch is completed, pause for a preset time. At the same time, set a migration rate threshold and monitor the CPU utilization, memory usage, and network bandwidth of the business server in real time. If the CPU utilization or memory usage exceeds the preset threshold, the migration rate will be automatically reduced and restored to the original rate after the resource load returns to normal.

[0024] S3.4 Migration tasks support breakpoint resumption: After each migration batch is completed, the current migration progress is automatically recorded. If the migration task is interrupted due to an abnormality, it can be restarted and the previously recorded migration progress can be read to continue the migration from that progress without having to start from the beginning.

[0025] As a further improvement to this technical solution, the data consistency verification is specifically implemented as follows:

[0026] S4.1 After the historical data migration task is completed, take the baseline time point t0 as the verification boundary point and perform consistency verification on the data of the original table and the shadow table respectively.

[0027] S4.2, Verification is divided into two dimensions:

[0028] S4.2.1 Historical Data Validation: By executing a comparison query SQL, the total number of data records in the original table and the shadow table within the time interval is counted. If the number of records is inconsistent, the data is directly determined to be inconsistent. If the number of records is consistent, a preset number of data records are randomly selected, and all field values ​​of each data record are compared row by row.

[0029] S4.2.2 Real-time data verification: Randomly sample a preset number of real-time data generated after t0, query the corresponding data in the original table and the shadow table respectively, compare the field values ​​of the data with the write timestamp, and ensure that the real-time data generated by the dual-write logic is consistent.

[0030] S4.3 If the above two dimensions of verification pass, it is determined that the data in the original table and the shadow table are consistent, and S5 is executed. If either dimension of verification fails, the cause of the anomaly is investigated, the problem is fixed, and the historical data migration and consistency verification in S3 is re-executed. If multiple consecutive verifications fail, the shadow table is deleted, and the process is returned to S1 to re-initiate the table structure update process.

[0031] As a further improvement to this technical solution, the traffic switching is specifically implemented as follows: S5.1, the routing rules are configured using a "gradual switching" strategy: a preset proportion of business write requests and query requests are routed to the shadow table, while the original table still processes the remaining proportion of requests.

[0032] S5.2 Monitor the data write success rate, query response time, and error-free operation status of the business system after the monitoring switch, and the monitoring duration shall not be less than the preset basic duration.

[0033] S5.3 If there are no abnormalities during the monitoring period, gradually increase the traffic ratio of the shadow table: increase the traffic ratio in sequence according to the preset gradient, and maintain the preset monitoring time after each increase. After the traffic ratio increases to 100% and there are no abnormalities in the monitoring, complete the full traffic switch.

[0034] S5.4 Once 100% of the traffic has been switched to the shadow table and it is running stably, the shadow table officially becomes the new main business table, and the original table stops receiving new business requests.

[0035] As a further improvement to this technical solution, the old table cleanup and status confirmation are specifically implemented as follows: S6.1 After the traffic is fully switched to the shadow table, the running status of the new business main table is continuously observed. The observation period is no less than the preset observation period. The focus is on confirming that there is no data loss during writing, the query results are accurate, and the business logic is normal.

[0036] S6.2 After the observation period ends and no abnormalities are found, the asynchronous write thread in S2 is taken offline, and the dual-write logic is turned off.

[0037] S6.3. Based on business data storage requirements, perform subsequent processing on the original table: If historical data needs to be retained, export the original table data to the distributed storage system for archiving; if it does not need to be retained, directly execute the DROPTABLE [original table name] command to delete the original table and release the storage space of the Tdengine database.

[0038] As a further improvement to this technical solution, the full-process rollback is specifically implemented as follows: if a business anomaly or system failure is detected at any stage from S1 to S6, a rollback operation is immediately triggered. The rollback operation includes: stopping the data dual-write logic in S2 and the historical data migration task in S3, deleting the shadow table created in S1, and restoring the business request route to the original table to ensure that the business is quickly restored to the normal operating state before the table structure change.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] 1. Reduce business interruption time: This invention adopts the idea of ​​"trading space for time" and modifies the table structure by creating a shadow table. There is no need to perform table locking operations on the original table. Even for Tdengine super tables with a large amount of data, it can avoid long-term business interruption and greatly reduce business interruption time.

[0041] 2. Reduce the risk of changes: The entire table structure modification process is based on the shadow table. All operations do not affect the normal operation of the original table and the original business. Rollback operations can be performed at any stage. Stopping dual writes and deleting the shadow table can restore the business to its initial state, which greatly reduces the risk of table structure changes.

[0042] 3. Ensure data integrity and consistency: Real-time data synchronization is guaranteed through a dual-write mechanism, historical data migration ensures the complete transfer of historical data, and data comparison and verification steps ensure that the original table and the shadow table are completely consistent before traffic switching is performed, effectively guaranteeing data integrity and consistency. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the implementation steps of the method of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Example: Please refer to Figure 1 As shown, a method for dynamically modifying the Tdengine table structure is provided, including: S1, shadow table creation: parsing the table structure update request, generating a shadow table containing change information based on the original table structure and naming it according to rules.

[0047] In one specific embodiment, the shadow table creation is implemented by: receiving a table structure update request submitted by a user, the request including the target original table name, specific structure change instructions and attribute information of the changed fields; based on the target original table name, executing the SQL instruction SHOWCREATESTABLE[original table name], querying and obtaining the complete table creation statement of the original table, the table creation statement including the field list, field attributes, tag information and database affiliation of the original table.

[0048] The structure change instructions include adding columns and deleting columns, and the attribute information of the changed fields includes, but is not limited to, data type and field length.

[0049] According to the structure change instruction, the original table creation statement is modified: if the change instruction is to add a column, the field to be added and its attributes are added to the original table field list; if the change instruction is to delete a column, the definition of the field to be deleted is removed from the original table field list.

[0050] When the structure change instruction is to delete a column, while modifying the original table creation statement to generate the shadow table statement, all historical data of the column to be deleted in the original table is retained, and no column deletion operation is performed in the original table. This ensures that the complete data of the original table can be directly restored during rollback without the need for additional data repair.

[0051] Generate a shadow table name according to the naming rule "original table name_shadow_v{version number}", execute the modified table creation statement, and complete the creation of the shadow table.

[0052] The version number increments according to the number of changes to the original table structure. The initial state of the original table corresponds to version v1, the shadow table version generated by the first change is v2, and so on.

[0053] S2, Dual Data Write: After the shadow table is created, configure the business system to write real-time data to both the original table and the shadow table simultaneously, while query requests are still handled by the original table.

[0054] In one specific embodiment, the dual data writing is implemented as follows: Configure the data writing logic of the business system, and while keeping the normal writing process of the original table unchanged, start an additional asynchronous writing thread. The asynchronous writing thread and the original table writing thread are executed in parallel, and the same business data is synchronously written to the shadow table created by S1 in real time. The execution priority of the asynchronous writing thread is lower than that of the original table writing thread to avoid occupying the core resources of the original table writing. Keep the routing strategy of business query requests unchanged, and all business query requests are directed to the original table for processing.

[0055] The normal write process involves writing business data to the original table along the existing path. All business query requests include historical data queries and real-time data queries. After the business data is generated, the main thread writes it to the original table along the existing path. The newly added asynchronous thread reads the same data and synchronously writes it to the shadow table after the main thread finishes writing. To avoid the asynchronous thread preempting the main thread's resources, the asynchronous thread's priority is set to "low," and a non-blocking I / O mode is used to ensure that the performance of the main thread writing to the original table is not affected.

[0056] The asynchronous write thread configuration retry mechanism is as follows: If a single data write to the shadow table fails, a unique identifier of the failed data (composed of a TS timestamp and the original table tag field) is recorded, and a retry is automatically initiated after a preset retry interval, with the number of retries not exceeding the preset maximum number of retries; if the maximum number of retries is reached and the process still fails, an alarm notification is immediately triggered (by email, WeChat, or SMS to the technical contact person), and the retry process for that data is suspended until the technical personnel have investigated and resolved the problem, at which point the data rewrite operation is manually triggered.

[0057] S3. Historical Data Migration: Based on the dual-write data structure, historical data from the original table before the creation of the shadow table is migrated to the shadow table in batches and at a controlled rate.

[0058] In one specific embodiment, the historical data migration is implemented as follows:

[0059] S3.1. Start an independent historical data migration task. The migration task is deployed on a dedicated processing node that is separate from the core business server to avoid occupying business operation resources.

[0060] S3.2 Determine the scope of historical data migration: Take the creation timestamp of the shadow table in S1 as the base time point, denoted as t0, and the migration scope is all historical data in the original table whose timestamp field ts <= t0.

[0061] S3.3. Adopt a "batch-based, rate-controlled" migration strategy: Divide historical data into multiple migration batches according to time intervals (e.g., one batch for each hour of data). Pause for a preset time after each batch is completed. At the same time, set a migration rate threshold (e.g., migrate 1000-3000 data entries per second). Monitor the CPU utilization, memory usage, and network bandwidth of the business server in real time. If the CPU utilization or memory usage exceeds the preset threshold, automatically reduce the migration rate (to 50% of the original rate). Restore the original rate after the resource load returns to normal.

[0062] The migration rate threshold can be dynamically adjusted based on the original table's data volume: different intervals are divided according to the original table's data volume, and each interval corresponds to a different migration rate threshold, ensuring that migration tasks do not affect normal business operations under different data volume scenarios. The preset time is dynamically adjusted according to peak and off-peak business periods: it can be set to 5 seconds during off-peak periods to speed up the migration process, and extended to 10 seconds during peak periods to reduce resource consumption on real-time business, ensuring that migration tasks and business operations are staggered in terms of resource allocation. The preset threshold is determined based on the daily load baseline of the business server (taking 80% of the peak load as the threshold value), which both reserves sufficient resources to ensure normal business operation and avoids excessive restrictions on migration tasks that lead to excessively long migration cycles.

[0063] S3.4 Migration tasks support breakpoint resumption: After each migration batch is completed, the current migration progress is automatically recorded. If the migration task is interrupted due to an abnormality, it can be restarted and the previously recorded migration progress can be read to continue the migration from that progress without having to start from the beginning.

[0064] The current migration progress includes the maximum timestamp of the migrated data and the current batch number.

[0065] S4. Data Consistency Verification: After the historical data migration is completed, verify whether the data in the original table and the shadow table are consistent from two dimensions: historical data and real-time data. If the verification is inconsistent, repair and re-verify or trigger rollback.

[0066] In one specific embodiment, the data consistency verification is implemented as follows:

[0067] S4.1 After the historical data migration task is completed, take the baseline time point t0 as the verification boundary point and perform consistency verification on the data of the original table and the shadow table respectively.

[0068] S4.2, Verification is divided into two dimensions:

[0069] S4.2.1 Historical Data Validation: By executing a comparison query SQL, the total number of data records in the original table and the shadow table within the time interval is counted respectively. If the number of records is inconsistent, the data is directly determined to be inconsistent. If the number of records is consistent, a preset number of data records are randomly selected (if the total number of data records is less than the preset number, all records are selected), and the values ​​of all fields of each data record are compared row by row.

[0070] S4.2.2 Real-time data verification: Randomly sample a preset number of real-time data generated after t0, query the corresponding data in the original table and the shadow table respectively, compare the field values ​​of the data with the write timestamp, and ensure that the real-time data generated by the dual-write logic is consistent.

[0071] Historical data verification can also be aided by using Tdengine's built-in CHECKSUM_TABLE([table name], 'ts<=t0') function to calculate the checksum of the data in the original table and the shadow table within the ts<=t0 interval. If the checksums are consistent, the sample size for subsequent sampling comparison can be simplified; if the checksums are inconsistent, a full count comparison and standard sampling comparison must be performed according to the requirements of S4.2.1 to balance verification efficiency and accuracy.

[0072] S4.3 If the above two dimensions of verification pass, it is determined that the data in the original table and the shadow table are consistent, and S5 is executed. If either dimension of verification fails, the cause of the anomaly is investigated, the problem is fixed, and the historical data migration and consistency verification in S3 is re-executed. If multiple consecutive verifications fail, the shadow table is deleted, and the process is returned to S1 to re-initiate the table structure update process.

[0073] The reasons for the anomalies include, but are not limited to, missed migration tasks, failure of dual-write threads, and data conversion errors caused by mismatched field types.

[0074] S5. Traffic Switching: After the data consistency verification passes, a gradual strategy is adopted to switch business traffic from the original table to the shadow table. After the full switch, the shadow table becomes the new master table.

[0075] In a specific embodiment, the traffic switching is implemented as follows: S5.1, the routing rules are configured using a "gradual switching" strategy: a preset proportion of business write requests and query requests are routed to the shadow table, while the original table still processes the remaining proportion of requests.

[0076] The preset ratio is set by professionals, following a principle of "from small to large, step-by-step increase": The initial preset ratio is typically 10%-20%, determined based on the size of the smallest load unit of the business system (e.g., device, user group), ensuring that the initial switch only affects a small number of services, facilitating rapid identification and remediation of potential problems. Subsequent ratios increase in a gradient of "50%→80%→100%", with each gradient's preset ratio covering different load scenarios of the business system (e.g., normal load, peak load), and the difference between adjacent gradients controlled within 30%-40%, ensuring a smooth switchover rhythm while avoiding excessively long switchover cycles due to too small gradient intervals. Routing rule configuration: Traffic switching is achieved by modifying the routing logic of the business system's DAO layer, without needing to adjust the underlying Tdengine database configuration, reducing the complexity and risk of system transformation. Modifications to routing rules take effect dynamically through the configuration center, without requiring a business system restart, ensuring business continuity.

[0077] The routing rules are configured by modifying the routing logic of the business system's database access layer (DAO layer), without requiring any changes to the underlying configuration of the Tdengine database, thus reducing the complexity of system transformation. Furthermore, during the route switching process, all business requests are unaware of the changes, and there is no risk of request loss or duplicate processing.

[0078] S5.2 Monitoring the data write success rate after the switch (must maintain the preset success standard), query response time (fluctuation should not exceed the preset fluctuation range of the original response time), and the business system's error-free operation status, with monitoring duration not less than the preset basic duration.

[0079] S5.3 If there are no abnormalities during the monitoring period, gradually increase the traffic ratio of the shadow table: increase the traffic ratio in sequence according to the preset gradient, and maintain the preset monitoring time after each increase. After the traffic ratio increases to 100% and there are no abnormalities in the monitoring, complete the full traffic switch.

[0080] S5.4 Once 100% of the traffic has been switched to the shadow table and it is running stably, the shadow table officially becomes the new main business table, and the original table stops receiving new business requests.

[0081] S6. Cleaning up and confirming the status of old tables: After the traffic switch is completed and the new master table is running stably, the dual-write logic is taken offline, and the original tables are archived or deleted as needed.

[0082] In one specific embodiment, the old table cleanup and status confirmation are implemented as follows: S6.1 After the traffic is fully switched to the shadow table, the running status of the new business main table is continuously observed. The observation period is no less than the preset observation period. The focus is on confirming that there is no data loss during writing, the query results are accurate, and the business logic is normal.

[0083] The preset observation period is set by professionals. For example, it needs to cover a complete operating cycle of the business system (e.g., 24 hours), ensuring that it includes peak business periods, off-peak periods, and special time periods (e.g., data aggregation in the early morning, hourly statistics), to comprehensively verify the stability of the new master table under various scenarios. At the same time, the observation period needs to be adjusted according to the timeliness requirements of the business data: for scenarios with high real-time requirements (e.g., power monitoring), it can be set to 24 hours; for scenarios with higher tolerance for data latency (e.g., offline statistics), it can be extended to 48-72 hours to ensure sufficient time to discover potential data consistency issues.

[0084] S6.2 After the observation period ends and no abnormalities are found, the asynchronous write thread in S2 is taken offline, and the dual-write logic is turned off.

[0085] S6.3. Based on business data storage requirements, perform subsequent processing on the original table: If historical data needs to be retained, export the original table data to the distributed storage system for archiving; if it does not need to be retained, directly execute the DROPTABLE [original table name] command to delete the original table and release the storage space of the Tdengine database.

[0086] S7. Full process rollback: If an anomaly occurs at any stage, stop the relevant tasks, delete the shadow table, and restore the original table route to enable the business to recover quickly.

[0087] In one specific embodiment, the full-process rollback is implemented as follows: if a business anomaly or system failure is detected at any stage from S1 to S6, a rollback operation is immediately triggered. The rollback operation includes: stopping the data dual-write logic in S2 and the historical data migration task in S3, deleting the shadow table created in S1, and restoring the business request route to the original table to ensure that the business is quickly restored to the normal operating state before the table structure change.

[0088] The business anomalies include, but are not limited to, the original table write failure rate exceeding the preset failure threshold, the shadow table query response time exceeding the preset response threshold, and a large number of data dual write failures. The system failures include Tdengine database service downtime and server power outage.

[0089] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for dynamically modifying the structure of a Tdengine table, characterized in that, include: S1. Shadow Table Creation: Parse the table structure update request, generate a shadow table containing change information based on the original table structure, and name it according to the rules. S2, Dual Data Write: After the shadow table is created, configure the business system to write real-time data to both the original table and the shadow table simultaneously, while query requests are still handled by the original table; S3. Historical data migration: Based on the dual-write data, historical data in the original table before the creation of the shadow table is migrated to the shadow table in batches and at a controlled rate. S4. Data consistency verification: After the historical data migration is completed, verify whether the data in the original table and the shadow table are consistent from two dimensions: historical data and real-time data. If the verification is inconsistent, repair and re-examine or trigger rollback. S5. Traffic Switching: After the data consistency verification passes, a gradual strategy is adopted to switch business traffic from the original table to the shadow table. After the full switch, the shadow table becomes the new master table. S6. Cleaning up and confirming the status of old tables: After the traffic switch is completed and the new master table is running stably, the dual-write logic is taken offline, and the original tables are archived or deleted as needed. S7. Full process rollback: If an anomaly occurs at any stage, stop the relevant tasks, delete the shadow table, and restore the original table route to enable the business to recover quickly.

2. The method for dynamically modifying the Tdengine table structure according to claim 1, characterized in that, The creation of the shadow table is specifically implemented as follows: Receive a table structure update request submitted by a user. The request includes the target original table name, specific structure change instructions and attribute information of the changed fields. Based on the target original table name, execute the SQL instruction SHOWCREATESTABLE[original table name] to query and obtain the complete table creation statement of the original table. The table creation statement includes the field list, field attributes, tag information and database affiliation of the original table. According to the structure change instruction, modify the original table creation statement: if the change instruction is to add a column, add the field to be added and its attributes to the original table field list; If the change instruction is to delete a column, then the definition of the field to be deleted will be removed from the original table's field list; Generate a shadow table name according to the naming rule "original table name_shadow_v{version number}", execute the modified table creation statement, and complete the creation of the shadow table.

3. The method for dynamically modifying the Tdengine table structure according to claim 1, characterized in that, The specific implementation method for the dual data writing is as follows: Configure the data writing logic of the business system. While keeping the normal writing process of the original table unchanged, start an additional asynchronous writing thread. The asynchronous writing thread executes in parallel with the original table writing thread, and writes the same business data to the shadow table created by S1 in real time. The execution priority of the asynchronous writing thread is lower than that of the original table writing thread. Keep the routing strategy of business query requests unchanged, and all business query requests are directed to the original table for processing.

4. The method for dynamically modifying the Tdengine table structure according to claim 1, characterized in that, The historical data migration is specifically implemented as follows: S3.

1. Start an independent historical data migration task. The migration task is deployed on a dedicated processing node that is separate from the core business server to avoid occupying business operation resources. S3.2 Determine the scope of historical data migration: Take the creation timestamp of the shadow table in S1 as the base time point, denoted as t0, and the migration scope is all historical data in the original table whose timestamp field ts <= t0; S3.

3. Adopt a "batch-based, rate-controlled" migration strategy: Divide historical data into multiple migration batches according to time intervals. After each batch is migrated, pause for a preset time. At the same time, set a migration rate threshold and monitor the CPU utilization, memory usage, and network bandwidth of the business server in real time. If the CPU utilization or memory usage exceeds the preset threshold, the migration rate will be automatically reduced and restored to the original rate after the resource load returns to normal. S3.4 Migration tasks support breakpoint resumption: After each migration batch is completed, the current migration progress is automatically recorded. If the migration task is interrupted due to an abnormality, it can be restarted and the previously recorded migration progress can be read to continue the migration from that progress without having to start from the beginning.

5. The method for dynamically modifying the Tdengine table structure according to claim 1, characterized in that, The specific implementation method for the data consistency verification is as follows: S4.1 After the historical data migration task is completed, take the baseline time point t0 as the verification boundary point and perform consistency verification on the data of the original table and the shadow table respectively. S4.2, Verification is divided into two dimensions: S4.2.1 Historical Data Validation: By executing a comparison query SQL, the total number of data records in the original table and the shadow table within the time interval is counted respectively. If the number of records is inconsistent, the data is directly determined to be inconsistent. If the number of records is consistent, a preset number of data records are randomly selected and the values ​​of all fields of each data record are compared row by row. S4.2.2 Real-time data verification: Randomly sample a preset amount of real-time data generated after t0, query the corresponding data in the original table and the shadow table respectively, compare the field values ​​of the data with the write timestamp, and ensure that the real-time data generated by the dual-write logic is consistent; S4.3 If the above two dimensions of verification pass, it is determined that the data in the original table and the shadow table are consistent, and S5 is executed. If either dimension of verification fails, the cause of the anomaly is investigated, the problem is fixed, and the historical data migration and consistency verification in S3 is re-executed. If multiple consecutive verifications fail, the shadow table is deleted, and the process is returned to S1 to re-initiate the table structure update process.

6. The method for dynamically modifying the Tdengine table structure according to claim 1, characterized in that, The traffic switching is specifically implemented as follows: S5.

1. Configure routing rules using a "progressive switching" strategy: route a preset proportion of business write requests and query requests to the shadow table, while the original table still processes the remaining proportion of requests; S5.2 Monitor the data write success rate, query response time and error-free operation status of the business system after the monitoring switch, and the monitoring duration shall not be less than the preset basic duration; S5.3 If there are no abnormalities during the monitoring period, gradually increase the traffic ratio of the shadow table: increase the traffic ratio in sequence according to the preset gradient, and maintain the preset monitoring time after each increase. After the traffic ratio is increased to 100% and there are no abnormalities in the monitoring, complete the full traffic switch. S5.4 Once 100% of the traffic has been switched to the shadow table and it is running stably, the shadow table officially becomes the new main business table, and the original table stops receiving new business requests.

7. The method for dynamically modifying the Tdengine table structure according to claim 1, characterized in that, The specific implementation method for old table cleanup and status confirmation is as follows: S6.1 After the traffic is fully switched to the shadow table, continuously observe the running status of the new business main table. The observation period shall not be less than the preset observation period. The key is to confirm that there is no data loss during writing, the query results are accurate, and there are no abnormalities in the business logic. S6.2 After the observation period ends and there are no abnormalities, the asynchronous write thread in S2 is taken offline, and the dual-write logic is turned off. S6.

3. Based on business data storage requirements, perform subsequent processing on the original table: If historical data needs to be retained, export the original table data to the distributed storage system for archiving; if it does not need to be retained, directly execute the DROPTABLE [original table name] command to delete the original table and release the storage space of the Tdengine database.

8. The method for dynamically modifying the Tdengine table structure according to claim 1, characterized in that, The specific implementation method for the full-process rollback is as follows: If a business anomaly or system failure is detected at any stage from S1 to S6, a rollback operation is immediately triggered. The rollback operation includes: stopping the data dual-write logic in S2 and the historical data migration task in S3, deleting the shadow table created in S1, and restoring the business request route to the original table to ensure that the business is quickly restored to the normal operating state before the table structure change.

Images