Database upgrading method

Abstract

The invention relates to the technical field of database analysis, in particular to a database upgrading method, which comprises the following steps of: establishing a double-engine parallel environment, starting new and old version database engines and constructing a cross-version communication channel; in response to the established communication channel, the client request is routed to the new and old engines in parallel for execution through the transaction dispatcher; based on a transaction operation log generated by an old engine, data change features are analyzed in real time, and an increment synchronization instruction set is generated; implementing version-coordinated transaction lock management when the new engine performs data playback according to the generated incremental synchronization instruction set; after the data consistency verification of the preset period is completed, switching a transaction route to a new engine and closing an old engine write-in channel; through double-engine parallel processing, incremental data synchronization and a cross-version lock coordination mechanism, the purposes of no service interruption and high data consistency in the database version upgrading process are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of database analysis, and particularly to an upgrade method for a database. Background Art

[0002] Traditional database version upgrade solutions generally face the dual challenges of poor service continuity and data consistency risks. In the prior art, the shutdown migration mode will cause the interruption of critical services. Especially in scenarios with high real-time requirements such as financial transactions and the Internet of Things, each hour of shutdown may cause millions of economic losses. Although the online hot upgrade method reduces the downtime, due to the serial processing mechanism of the single-engine architecture, there are still problems such as a long data migration window period and poor cross-version transaction compatibility, which are extremely likely to cause data loss or inconsistent states. When it comes to large-scale data table structure changes or storage engine replacements, traditional solutions lack effective concurrent control means, often resulting in business response delays due to overly coarse lock granularity, or deadlocks between versions during fine-grained lock management.

[0003] In addition, the static resource configuration strategy is difficult to cope with the dynamic fluctuations of the loads of the old and new engines during the upgrade process, and it is easy to cause the coexistence of resource idleness and bottlenecks. In the prior art, data verification mostly uses full-scale comparison after the event and cannot capture incremental differences in real time, resulting in a lag in rollback decisions. These defects seriously restrict the availability and upgrade reliability of critical business systems. Summary of the Invention

[0004] The purpose of the present invention is to provide an upgrade method for a database to solve the problems raised in the above background art.

[0005] To achieve the above purpose, the present invention provides the following technical solution: An upgrade method for a database, the method steps of which include:

[0006] S1. Establish a dual-engine parallel environment, start the old and new version database engines, and build a cross-version communication channel;

[0007] S2. In response to the communication channel established in step S1, parallelly route client requests to the old and new engines for execution through a transaction dispatcher;

[0008] S3. Based on the transaction operation logs generated by the old engine in step S2, real-time analyze the data change characteristics and generate an incremental synchronization instruction set;

[0009] S4. According to the incremental synchronization instruction set generated in step S3, implement version-coordinated transaction lock management when the new engine performs data replay;

[0010] S5. After the data consistency verification for the preset period in step S4 is completed, switch the transaction routing to the new engine and close the write channel of the old engine; steps S1 to S5 achieve the goal of uninterrupted service and strong data consistency during the database version upgrade through the dual-engine parallel processing, incremental data synchronization, and cross-version lock coordination mechanism.

[0011] As a further improvement of this technical solution, the process of S1 specifically includes:

[0012] Deploy the old and new version database instances in parallel in the computing cluster, allocate initial computing resources and set up load-aware probes, and collect the performance metrics of the dual engines according to the load-aware probes, where the performance metrics of the dual engines include CPU utilization rate;

[0013] Establish a bidirectional data pipeline including multi-level buffer queues, and the buffer queues include three levels: emergency, normal, and batch; dynamically monitor the key channel metrics according to the bidirectional data pipeline, where the key channel metrics include buffer fill rate;

[0014] Trigger resource rebalancing according to the CPU utilization rate difference threshold, and dynamically adjust the network bandwidth allocation based on the buffer fill rate.

[0015] As a further improvement of this technical solution, the operations performed by the transaction dispatcher in S2 include:

[0016] Perform dual-engine synchronous execution and result comparison on write operations, and trigger transaction rollback and record exceptions when the difference exceeds the preset threshold.

[0017] As a further improvement of this technical solution, the process of S3 specifically includes:

[0018] Parse the transaction operation log into atomic data change units;

[0019] Merge and generate a batch operation instruction set including operation sequence constraints according to the table dimension as the incremental synchronization instruction set.

[0020] As a further improvement of this technical solution, the transaction lock management in S4 includes dynamically selecting lock strategies according to the operation characteristics in the incremental synchronization instruction set, specifically including:

[0021] Dynamically select table-level locks and row-level locks according to the operation characteristics;

[0022] Set a lock timeout mechanism that dynamically matches the instruction execution time;

[0023] Synchronize the dual-engine lock status through the cross-version communication channel.

[0024] As a further improvement of this technical solution, the synchronization of the dual-engine lock status includes:

[0025] When the dual engines mutually exclude the lock for the same resource request, the old version of the lock is automatically released according to the preset priority;

[0026] Allow the dual engines to add shared locks to the same resource simultaneously, supporting parallel reading.

[0027] As a further improvement of this technical solution, the data consistency verification of the preset cycle includes:

[0028] Execute full - volume data comparison and real - time comparison of incremental changes within the preset cycle;

[0029] Set the fault - tolerance threshold for consistency, and terminate the verification process when the fault - tolerance threshold for consistency is exceeded.

[0030] As a further improvement of this technical solution, the process of S5 specifically includes:

[0031] Execute write - operation suspension, transaction routing switching, and old - engine write - permission closing in stages;

[0032] Retain the old - engine reading function for monitoring and verification after the switch;

[0033] Release the network bandwidth related to the old engine and perform new - engine performance tuning.

[0034] As a further improvement of this technical solution, the process of triggering resource re - balancing according to the CPU utilization rate difference threshold specifically includes:

[0035] By collecting the CPU utilization rate metrics of the dual engines in real - time, periodically calculate the difference between the two and compare it with the preset CPU utilization rate difference threshold; when the difference exceeds the CPU utilization rate difference threshold, based on the current load weight coefficients and transaction - processing priorities of the dual engines, use the feedback control model to dynamically calculate the resource re - allocation ratio to preferentially ensure the basic resource supply of the high - load engine; send a quota adjustment instruction to the resource scheduler according to the allocation ratio.

[0036] As a further improvement of this technical solution, the process of dynamically adjusting network bandwidth allocation based on the buffer filling rate specifically includes:

[0037] Real - time monitor the buffer filling rate. When the preset buffer filling rate threshold is reached, dynamically calculate the bandwidth allocation ratio according to the current load and growth trend, optimize the transmission parameters using the sliding - window algorithm, and adjust the bandwidth weights of each channel in real - time through the traffic controller to form a dynamic adaptation mechanism between the buffer capacity and network resources.

[0038] Compared with the prior art, the beneficial effects of the present invention are:

[0039] The upgrade method of this database realizes zero interruption of business during the database upgrade through a dual-engine parallel processing architecture. Combining a cross-version communication channel with an intelligent transaction distribution mechanism ensures the compatibility and integrity of the coordinated execution of transactions by the old and new version engines; the incremental data synchronization technology generates a synchronization instruction set based on the transaction operation logs parsed in real time, and cooperates with the dynamic lock strategy management to effectively reduce the risk of data conflicts between versions; the multi-level buffer queue and the load-aware resource allocation mechanism realize the dynamic adaptation of network bandwidth and computing resources, ensuring the stability of the system throughput during the upgrade.

[0040] In addition, through a consistency verification process that combines full volume and increment, data differences are quickly identified and fault tolerance processing is triggered within a preset period. At the same time, the read function of the old engine is retained for abnormal rollback verification; on the basis of ensuring strong data consistency, the overall solution significantly improves the resource utilization efficiency and system reliability during the upgrade process, providing a smooth upgrade guarantee for critical business systems. Brief Description of the Drawings

[0041] Figure 1 It is a schematic diagram of the method steps of the present invention. Detailed Embodiments

[0042] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0043] Please refer to Figure 1 , the present invention provides a technical solution: an upgrade method for a database, including the following method steps:

[0044] S1. Establish a dual-engine parallel environment, start the old and new version database engines, and build a cross-version communication channel, specifically including:

[0045] Deploy the old and new version database instances in parallel in the computing cluster, allocate initial computing resources (CPU core count, memory quota, storage volume) for the dual engines, enable the load-aware probe, and collect the performance metrics of the dual engines in real time, including CPU utilization; where the dual engines refer to the old and new version database engines.

[0046] Establish a two-way data pipeline to realize cross-version data interaction between the dual engines, and set up a multi-level buffer queue, where the multi-level buffer queue includes three levels of queues: emergency, normal, and batch.

[0047] Dynamically monitor the key metrics of the channel according to the bidirectional data pipeline, where the key metrics of the channel include the buffer fill rate; the buffer fill rate is a quantitative metric that directly reflects the channel load status and is used to trigger dynamic bandwidth adjustment;

[0048] Periodically calculate the CPU utilization difference between the two engines. When the difference exceeds the set threshold, trigger resource rebalancing and adjust the resource quotas of each engine according to the dynamic algorithm; Trigger resource rebalancing according to the CPU utilization difference threshold; Specifically include:

[0049] By collecting the CPU utilization metrics of the two engines in real time, periodically calculate the difference between them and compare it with the preset CPU utilization difference threshold; When the difference exceeds the CPU utilization difference threshold, based on the current load weight coefficients and transaction processing priorities of the two engines, use the feedback control model to dynamically calculate the resource reallocation ratio, which is used to preferentially ensure the basic resource supply of the high-load engine; Send a quota adjustment instruction to the resource scheduler according to the allocation ratio, synchronously update parameters such as the number of CPU cores and memory allocation, and re-monitor the index fluctuation trend after adjustment to form a closed-loop resource self-adaptive adjustment mechanism, ensuring that the resource allocation of the two engines always matches the real-time load demand and maintaining the performance balance state of the overall system;

[0050] When the buffer fill rate of the bidirectional data pipeline rises to the preset level, automatically increase the network bandwidth allocation proportionally, reflecting the self-adaptive supply ability of the channel resources; Specifically include:

[0051] By real-time monitoring the change of the fill rate of the data buffer, when it is detected that the fill rate exceeds the preset buffer fill rate threshold, dynamically construct a bandwidth allocation model based on the current network load status and buffer growth trend, and preferentially increase the bandwidth quota for the high-fill-rate buffer channel;

[0052] At the same time, combined with historical transmission efficiency data, use the sliding window algorithm to calculate the optimal bandwidth ratio, and adjust the bandwidth allocation weights of each channel in real time through the flow controller to form a dynamic balance mechanism between the buffer capacity and network resource supply, effectively preventing buffer overflow or idle state and continuously optimizing the data transmission efficiency.

[0053] The established communication channel realizes real-time data interaction and resource dynamic coordination between the new and old engines through the bidirectional data pipeline and the multi-level buffer queue, ensuring that the two engines maintain the ability to work together under load fluctuations, and providing infrastructure support for the subsequent two-way flow control and data consistency guarantee.

[0054] S2. In response to the communication channel established in step S1, use the transaction dispatcher to parallelly route the client requests to the new and old engines for execution, specifically including:

[0055] Identify the transaction type (query, update, or schema change) requested by the client, and extract key features (such as SQL operation mode, table object version dependency);

[0056] Combine the CPU utilization rate and buffer fill rate of the dual engines monitored in stage S1 to calculate the real-time service capacity index of the dual engines;

[0057] Send the write operation to both engines for execution simultaneously, compare the results returned by the two engines field by field. When the difference in execution time between the two engines exceeds the preset threshold, trigger transaction rollback and record the exception to ensure operation consistency through result comparison; Dynamically select the optimal execution node based on the cache hit rate and response latency of the dual engines;

[0058] In the old version of the database engine, whenever a transaction operation (such as query, update, or schema change) occurs, record the transaction operation log. This log contains detailed transaction information; Use a log collection tool or proxy program to collect transaction operation logs from the old version of the database engine in real-time. Log collectors such as Fluentd and Logstash can read and forward log files in real-time.

[0059] S3. Based on the transaction operation logs generated by the old engine in step S2, parse the data change characteristics in real-time and generate an incremental synchronization instruction set, specifically including:

[0060] Continuously collect the transaction operation logs generated by the old version in the dual engines, and parse the log entries in the transaction operation logs into atomic data change units;

[0061] Merge the parsed atomic data change units into batch operation instructions by table dimension, and mark the operation sequence constraints in the generated synchronization instruction set as the incremental synchronization instruction set.

[0062] S4. According to the incremental synchronization instruction set generated in step S3, implement version-coordinated transaction lock management during data replay in the new engine, specifically including:

[0063] S4.1. Dynamically select the lock strategy according to the operation characteristics (such as batch write, single-row update) marked in the incremental synchronization instruction set generated in S3, where:

[0064] Table-level lock: Applicable to schema changes or full-table data migration, lock the entire table to ensure operation atomicity;

[0065] Row-level lock: For single-row data operations, only lock the target data row to maximize concurrency performance;

[0066] Lock timeout mechanism: Set the upper limit of the lock holding time, dynamically match it with the estimated execution time of the instruction to prevent deadlocks or long-term blocking.

[0067] S4.2. Through the cross-version communication channel established in the S1 stage, the lock states (such as lock type, locking range, and holding time) held by the old version in the dual engines are synchronously updated to the new version in real time, where:

[0068] When the dual engines request mutex locks for the same resource, the old version lock is automatically released according to the preset priority (new version first);

[0069] The dual engines are allowed to add shared locks to the same resource simultaneously to support parallel reading;

[0070] S4.3. Define a preset time period (such as 24 hours, 48 hours, etc.). During this period, data consistency verification is continuously performed. At the start of the verification period, a full-scale data comparison is carried out to ensure that all tables and data in the old and new engines are completely consistent;

[0071] During the verification period, the incremental data changes between the old and new engines are continuously monitored and recorded, and real-time comparison is carried out, including transaction log parsing, data synchronization instruction set generation, real-time comparison, and consistency determination, where:

[0072] Transaction log parsing: Continuously collect the transaction operation logs generated by the old version in the dual engines, and parse the log entries into atomic data change units;

[0073] Data synchronization instruction set generation: Merge the parsed atomic data change units into batch operation instructions by table dimension, and mark the operation sequence constraints in the generated synchronization instruction set;

[0074] Real-time comparison: Compare the incremental synchronization instructions executed by the new engine with the actual data of the old engine in real time to ensure that each change is correct;

[0075] Consistency determination: Define the fault tolerance threshold for data consistency, such as the maximum allowable number or proportion of inconsistencies. During the verification period, continuously monitor the number or proportion of inconsistencies. Once the set threshold is exceeded, immediately stop the verification process.

[0076] S5. When the data consistency verification in step S4 is completed for the preset period, switch the transaction routing to the new engine and close the write channel of the old engine, that is, perform write operation suspension, transaction routing switch, and old engine write permission closure in stages; After the switch, retain the read function of the old engine for monitoring and verification; Release the network bandwidth related to the old engine and perform performance tuning on the new engine; The specific steps are as follows:

[0077] Suspend the write operation requests of all clients to ensure that no new data changes occur;

[0078] Stop the operation mode of sending write operations to both the old and new engines simultaneously. At this time, all read and write requests are temporarily suspended;

[0079] Modify the configuration of the transaction dispatcher so that it routes all subsequent transaction requests (including reads and writes) only to the new version of the database engine;

[0080] After confirming that the new engine can independently handle all types of operation requests, set it to exclusive mode, at which time it will assume the entire workload;

[0081] Cut off the write permission of the old engine to prevent any accidental data writing, but still retain its read function for a period of time for monitoring and verification.

[0082] Remove the part related to the old engine in the two-way data pipeline, release the network bandwidth and other resources that are no longer used. At the same time, perform performance tuning on the new engine.

[0083] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only preferred examples of the present invention and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements fall within the scope of the present invention claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for upgrading a database, characterized in that: The method steps are as follows: S1. Establish a dual-engine parallel environment, start the old and new versions of the database engine, and build a cross-version communication channel; S2, in response to the communication channel established in step S1, routing the client request to the new and old engines for execution in parallel through the transaction distributor; S3. Based on the transaction operation log generated by the old engine in step S2, the data change characteristics are analyzed in real time and an incremental synchronization instruction set is generated; S4. Implementing version-coordinated transaction lock management when the new engine executes data replay according to the incremental synchronization instruction set generated in step S3; S5. After step S4 completes the data consistency verification for the preset period, switch the transaction routing to the new engine and close the old engine write channel; steps S1 to S5 use dual-engine parallel processing, incremental data synchronization and cross-version lock coordination mechanism to ensure that the service is not interrupted and the data maintains strong consistency during the database version upgrade process.

2. The database upgrade method according to claim 1, characterized in that: The process of S1 specifically includes: Deploy the old and new versions of database instances in parallel in the computing cluster, allocate initial computing resources and set up load-aware probes. Collect the performance indicators of the dual engines based on the load-aware probes, including CPU utilization. Establishing a bidirectional data pipeline including a multi-level buffer queue, wherein the buffer queue includes three levels of queues: emergency, ordinary, and batch; dynamically monitoring channel key indicators according to the bidirectional data pipeline, wherein the channel key indicators include a buffer fill rate; Resource rebalancing is triggered based on CPU utilization difference thresholds, and network bandwidth allocation is dynamically adjusted based on buffer fill rates.

3. The database upgrade method according to claim 1, characterized in that: The operations performed by the transaction distributor in S2 include: The write operation is executed synchronously by two engines and the results are compared. When the difference exceeds the preset threshold, the transaction is rolled back and the exception is recorded.

4. The database upgrade method according to claim 1, characterized in that: The process of S3 specifically includes: Parse transaction operation logs into atomic data change units; A batch operation instruction set containing operation order constraints is generated by merging table dimensions as an incremental synchronization instruction set.

5. The database upgrade method according to claim 1, characterized in that: The transaction lock management in S4 includes dynamically selecting a lock strategy according to the operation characteristics in the incremental synchronization instruction set, specifically including: Dynamically select table-level locks and row-level locks based on operation characteristics; Set a lock timeout mechanism that dynamically matches the instruction execution time; The dual-engine lock status is synchronized through the cross-version communication channel.

6. The database upgrade method according to claim 5, characterized in that: The synchronous dual-engine lock state includes: When two engines request a mutex lock for the same resource, the old version lock is automatically released according to the preset priority; Allows dual engines to add shared locks to the same resource at the same time, supporting parallel reading.

7. The database upgrade method according to claim 1, characterized in that: The data consistency verification of the preset period includes: Perform full data comparison and real-time comparison of incremental changes within a preset period; Set a consistency tolerance threshold and terminate the verification process when the consistency tolerance threshold is exceeded.

8. The database upgrade method according to claim 1, characterized in that: The process of S5 specifically includes: Suspending write operations, switching transaction routing, and shutting down old engine write permissions are performed in stages; Keep the old engine reading function for monitoring verification after switching; Release the network bandwidth related to the old engine and perform performance tuning for the new engine.

9. The database upgrade method according to claim 2, characterized in that: The process of triggering resource rebalancing according to the CPU utilization difference threshold specifically includes: By collecting dual-engine CPU utilization indicators in real time, the difference between the two is periodically calculated and compared with the preset CPU utilization difference threshold; when the difference exceeds the CPU utilization difference threshold, based on the current dual-engine load weight coefficient and transaction processing priority, the feedback control model is used to dynamically calculate the resource reallocation ratio to prioritize the basic resource supply of the high-load engine; and quota adjustment instructions are sent to the resource scheduler based on the allocation ratio.

10. The database upgrade method according to claim 2, characterized in that: The process of dynamically adjusting network bandwidth allocation based on buffer fill rate specifically includes: Monitor the buffer fill rate in real time. When the preset buffer fill rate threshold is reached, dynamically calculate the bandwidth allocation ratio based on the current load and growth trend, optimize the transmission parameters using the sliding window algorithm, and adjust the bandwidth weight of each channel in real time through the traffic controller to form a dynamic adaptation mechanism between buffer capacity and network resources.

Images