A database upgrade method

Abstract

The present invention relates to the technical field of database analysis, and specifically to a database upgrade method. In the present invention, a dual-engine parallel environment is established, new and old version database engines are started, and a cross-version communication channel is constructed; in response to the established communication channel, client requests are routed in parallel to the new and old engines for execution through a transaction distributor; based on the transaction operation log generated by the old engine, data change characteristics are analyzed in real time and an incremental synchronization instruction set is generated; according to the generated incremental synchronization instruction set, version-coordinated transaction lock management is implemented when the new engine executes data replay; after completing the data consistency verification of a preset period, the transaction route is switched to the new engine and the old engine write channel is closed; through the dual-engine parallel processing, incremental data synchronization and cross-version lock coordination mechanism, the goals of uninterrupted service and strong data consistency during the database version upgrade process are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of database analysis, and in particular to a database upgrading method. Background Art

[0002] Traditional database version upgrade solutions generally face the dual challenges of poor service continuity and data consistency risks. In existing technologies, the use of a downtime migration model will lead to interruptions in critical businesses, especially in scenarios with high real-time requirements such as financial transactions and the Internet of Things. Every hour of downtime may cause millions in economic losses. Although the online hot upgrade method reduces downtime, it is limited by the serial processing mechanism of the single-engine architecture. There are still problems such as long data migration windows and poor cross-version transaction compatibility, which can easily lead to data loss or inconsistent status. When it comes to large-scale data table structure changes or storage engine replacements, traditional solutions lack effective concurrency control measures, often resulting in business response delays due to coarse lock granularity, or deadlocks between versions when managing fine-grained locks.

[0003] Furthermore, static resource allocation strategies struggle to cope with the dynamic fluctuations in load between the old and new engines during the upgrade process, easily leading to both idle resources and bottlenecks. Existing data verification techniques often rely on post-hoc full-data comparisons, which fail to capture incremental differences in real time and result in delayed rollback decisions. These shortcomings severely restrict the availability and upgrade reliability of critical business systems. Summary of the Invention

[0004] The purpose of the present invention is to provide a database upgrade method to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for upgrading a database, the method comprising the following steps:

[0006] S1. Establish a dual-engine parallel environment, start the old and new versions of the database engine, and build a cross-version communication channel;

[0007] S2. In response to the communication channel established in step S1, the client request is routed to the new and old engines for execution in parallel through the transaction distributor;

[0008] S3. Based on the transaction operation log generated by the old engine in step S2, the data change characteristics are analyzed in real time and an incremental synchronization instruction set is generated;

[0009] S4. Implementing version-coordinated transaction lock management when the new engine executes data replay based on the incremental synchronization instruction set generated in step S3.

[0010] S5. After step S4 completes the data consistency verification for the preset period, switch the transaction routing to the new engine and close the old engine write channel; steps S1 to S5 achieve the goal of uninterrupted service and strong data consistency during the database version upgrade process through dual-engine parallel processing, incremental data synchronization and cross-version lock coordination mechanism.

[0011] As a further improvement of this technical solution, the process of S1 specifically includes:

[0012] Deploy the new and old database versions in parallel in the computing cluster, allocate initial computing resources, and set up load-aware probes. Use the load-aware probes to collect dual-engine performance metrics, including CPU utilization.

[0013] Establishing a bidirectional data pipeline comprising multiple levels of buffer queues, the buffer queues comprising three levels of queues: emergency, normal, and batch; dynamically monitoring key channel indicators based on the bidirectional data pipeline, wherein the key channel indicators include buffer fill rate;

[0014] Resource rebalancing is triggered based on the CPU utilization difference threshold, and network bandwidth allocation is dynamically adjusted based on the buffer fill rate.

[0015] As a further improvement of the present technical solution, the transaction distributor in S2 performs the following operations:

[0016] The write operation is executed synchronously by two engines and the results are compared. When the difference exceeds the preset threshold, the transaction is rolled back and the exception is recorded.

[0017] As a further improvement of this technical solution, the process of S3 specifically includes:

[0018] Parse transaction operation logs into atomic data change units;

[0019] Merge by table dimension to generate a batch operation instruction set containing operation order constraints as the incremental synchronization instruction set.

[0020] As a further improvement of this technical solution, the transaction lock management in S4 includes dynamically selecting a lock strategy based on the operation characteristics in the incremental synchronization instruction set, specifically including:

[0021] Dynamically select table-level locks and row-level locks based on operation characteristics;

[0022] Set a lock timeout mechanism that dynamically matches instruction execution time;

[0023] Synchronize the dual-engine lock status through the cross-version communication channel.

[0024] As a further improvement of this technical solution, the synchronous dual-engine lock state includes:

[0025] When two engines request a mutex lock for the same resource, the old version lock is automatically released based on the preset priority.

[0026] Allows dual engines to simultaneously apply shared locks to the same resource, supporting parallel reading.

[0027] As a further improvement of this technical solution, the data consistency verification of the preset period includes:

[0028] Perform full data comparison and real-time comparison of incremental changes within a preset period;

[0029] Set a consistency tolerance threshold and terminate the verification process when the consistency tolerance threshold is exceeded.

[0030] As a further improvement of this technical solution, the process of S5 specifically includes:

[0031] The write operation suspension, transaction routing switch, and shutdown of the old engine write permission are performed in stages;

[0032] Retain the old engine reading function for monitoring verification after the switch;

[0033] Release network bandwidth related to the old engine and optimize the performance of the new engine.

[0034] As a further improvement of this technical solution, the process of triggering resource rebalancing according to the CPU utilization difference threshold specifically includes:

[0035] By collecting the CPU utilization indicators of the two engines in real time, the difference between the two is periodically calculated and compared with the preset CPU utilization difference threshold. When the difference exceeds the CPU utilization difference threshold, the feedback control model is used to dynamically calculate the resource reallocation ratio based on the current load weight coefficients and transaction processing priorities of the two engines to prioritize the basic resource supply of the high-load engine. Quota adjustment instructions are sent to the resource scheduler based on the allocation ratio.

[0036] As a further improvement of the present technical solution, the process of dynamically adjusting network bandwidth allocation based on the buffer fill rate specifically includes:

[0037] The buffer fill rate is monitored in real time. When the preset buffer fill rate threshold is reached, the bandwidth allocation ratio is dynamically calculated based on the current load and growth trend. The transmission parameters are optimized using the sliding window algorithm. The bandwidth weight of each channel is adjusted in real time through the traffic controller to form a dynamic adaptation mechanism between buffer capacity and network resources.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] This database upgrade method achieves zero business interruption during the database upgrade process through a dual-engine parallel processing architecture, and combines cross-version communication channels with an intelligent transaction distribution mechanism to ensure the compatibility and integrity of transactions executed collaboratively by the new and old version engines. The incremental data synchronization technology generates a synchronization instruction set based on real-time parsed transaction operation logs, and cooperates with dynamic lock strategy management to effectively reduce the risk of data conflicts between versions. The multi-level buffer queue and load-aware resource allocation mechanism realize dynamic adaptation of network bandwidth and computing resources, ensuring stable system throughput during the upgrade.

[0040] In addition, through a consistency verification process that combines full and incremental data, data differences can be quickly identified and fault-tolerant processing can be triggered within a preset period, while retaining the old engine reading function for abnormal fallback verification. The overall solution significantly improves the resource utilization efficiency and system reliability of the upgrade process on the basis of ensuring strong data consistency, providing smooth upgrade guarantees for critical business systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the method steps of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0043] See also Figure 1 The present invention provides a technical solution: a database upgrade method, comprising the following steps:

[0044] S1. Establish a dual-engine parallel environment, start the old and new versions of the database engine, and build a cross-version communication channel. Specifically, the following steps are required:

[0045] Deploy the new and old database versions in parallel in the computing cluster. Allocate initial computing resources (CPU cores, memory quota, and storage volumes) for the two engines. Enable load-aware probes to collect performance metrics for both engines in real time, including CPU utilization. The two engines represent the new and old database versions.

[0046] Establish a bidirectional data pipeline to achieve cross-version data interaction between the two engines, and set up a multi-level buffer queue, which includes emergency, normal, and batch queues.

[0047] Dynamically monitor key channel indicators based on bidirectional data pipelines, including buffer fill rate. Buffer fill rate directly reflects the channel load status and is used to trigger dynamic bandwidth adjustment.

[0048] The CPU utilization difference between the two engines is periodically calculated. When the difference exceeds the set threshold, resource rebalancing is triggered, and the resource quota of each engine is adjusted according to the dynamic algorithm. Resource rebalancing is triggered based on the CPU utilization difference threshold. Specifically, the following steps are performed:

[0049] By collecting dual-engine CPU utilization metrics in real time, the system periodically calculates the difference between the two and compares it with a preset CPU utilization difference threshold. When the difference exceeds the CPU utilization difference threshold, a feedback control model is used to dynamically calculate the resource reallocation ratio based on the current dual-engine load weight coefficients and transaction processing priorities to prioritize the supply of basic resources to the high-load engine. Based on the allocation ratio, quota adjustment instructions are sent to the resource scheduler, and parameters such as the number of CPU cores and memory allocation are simultaneously updated. After the adjustment, the indicator fluctuation trend is re-monitored, forming a closed-loop resource adaptive adjustment mechanism to ensure that dual-engine resource allocation always matches real-time load demand and maintains balanced performance of the entire system.

[0050] When the buffer fill rate of the bidirectional data pipeline reaches a preset level, network bandwidth allocation is automatically increased proportionally, demonstrating the adaptive supply capability of channel resources. Specifically,

[0051] By monitoring the fill rate changes of the data buffer in real time, when it is detected that the fill rate exceeds the preset buffer fill rate threshold, a bandwidth allocation model is dynamically built based on the current network load status and buffer growth trend, and bandwidth quota is increased preferentially for high fill rate buffer channels;

[0052] At the same time, combined with historical transmission efficiency data, the sliding window algorithm is used to calculate the optimal bandwidth ratio, and the bandwidth allocation weight of each channel is adjusted in real time through the traffic controller to form a dynamic balance mechanism between buffer capacity and network resource supply, effectively preventing buffer overflow or idle state, and continuously optimizing data transmission efficiency.

[0053] The established communication channel enables real-time data interaction and dynamic resource coordination between the new and old engines through bidirectional data pipelines and multi-level buffer queues, ensuring that the dual engines can maintain their ability to work together under load fluctuations, and providing infrastructure support for bidirectional flow control and data consistency assurance in subsequent stages.

[0054] S2. In response to the communication channel established in step S1, the client request is routed to the new and old engines for execution in parallel through the transaction distributor, specifically including:

[0055] Identify the transaction type (query, update, or structural change) requested by the client and extract key features (such as SQL operation mode and table object version dependency);

[0056] Combined with the dual-engine CPU utilization and buffer fill rate monitored during the S1 phase, the real-time service capability index of the dual-engine is calculated;

[0057] Write operations are sent to both engines simultaneously for execution. The results returned by the two engines are compared field by field. When the difference in execution time between the two engines exceeds a preset threshold, the transaction is rolled back and the exception is recorded. This result comparison ensures operational consistency. The optimal execution node is dynamically selected based on the cache hit rate and response latency of the two engines.

[0058] In older database engines, every time a transaction (such as a query, update, or schema change) occurs, a transaction log is recorded. This log contains detailed transaction information. Log collection tools or agents can be used to collect transaction logs from older database engines in real time. Log collectors such as Fluentd and Logstash can read and forward log files in real time.

[0059] S3. Based on the transaction operation log generated by the old engine in step S2, the data change characteristics are analyzed in real time and an incremental synchronization instruction set is generated, specifically including:

[0060] Continuously collect transaction operation logs generated by older versions of the dual engines and parse the log entries in the transaction operation logs into atomic data change units;

[0061] The parsed atomic data change units are merged into batch operation instructions according to the table dimension, and the operation sequence constraints are marked in the generated synchronization instruction set as the incremental synchronization instruction set.

[0062] S4. Based on the incremental synchronization instruction set generated in step S3, version-coordinated transaction lock management is implemented when the new engine executes data replay, specifically including:

[0063] S4.1. Dynamically select a locking strategy based on the operational characteristics (e.g., batch write, single row update) noted in the incremental synchronization instruction set generated in S3, where:

[0064] Table-level lock: Applicable to structural changes or full-table data migration, locking the entire table to ensure the atomicity of the operation;

[0065] Row-level lock: For single-row data operations, only the target data row is locked to maximize concurrency performance;

[0066] Lock timeout mechanism: Set an upper limit on the lock holding time, dynamically matching it with the estimated instruction execution time to prevent deadlock or long-term blocking.

[0067] S4.2. Using the cross-version communication channel established in S1, synchronize the lock status (e.g., lock type, lock range, and holding time) held by the old version in the dual-engine to the new version in real time.

[0068] When two engines request a mutex lock for the same resource, the old version lock is automatically released based on the preset priority (newer version first);

[0069] Allows dual engines to simultaneously apply shared locks to the same resource, supporting parallel reading;

[0070] S4.3. Define a preset time period (e.g., 24 hours, 48 hours, etc.) and continuously perform data consistency verification within this period. At the beginning of the verification period, perform a full data comparison to ensure that all tables and data in the new and old engines are completely consistent.

[0071] During the verification cycle, incremental data changes between the new and old engines are continuously monitored and recorded, and compared in real time. This includes transaction log parsing, data synchronization instruction set generation, real-time comparison, and consistency determination. The following steps are performed:

[0072] Transaction log parsing: Continuously collect transaction operation logs generated by older versions of the dual engines and parse log entries into atomic data change units;

[0073] Data synchronization instruction set generation: The parsed atomic data change units are merged into batch operation instructions according to table dimensions, and operation sequence constraints are marked in the generated synchronization instruction set;

[0074] Real-time comparison: The incremental synchronization instructions executed by the new engine are compared with the actual data of the old engine in real time to ensure that each change is correct;

[0075] Consistency determination: Define the tolerance threshold for data consistency, such as the maximum number or ratio of inconsistencies allowed. During the verification cycle, continuously monitor the number or ratio of inconsistencies. Once the set threshold is exceeded, the verification process is immediately stopped.

[0076] S5. After the data consistency verification for the preset period is completed in step S4, the transaction route is switched to the new engine and the old engine write channel is closed. That is, the write operation suspension, transaction route switching, and the old engine write permission closure are performed in stages; after the switch, the old engine read function is retained for monitoring and verification; the network bandwidth related to the old engine is released and the performance of the new engine is tuned. The specific steps include:

[0077] Suspend all client write requests to ensure that no new data changes occur;

[0078] Stop sending write operations to both the old and new engines at the same time. All read and write requests are temporarily suspended.

[0079] Modify the configuration of the transaction distributor so that it routes all subsequent transaction requests (both read and write) only to the new version of the database engine;

[0080] After confirming that the new engine can independently handle all types of operation requests, set it to exclusive mode, at which point it will take on the entire workload;

[0081] Cut off the old engine's write permissions to prevent any accidental data writing, but still retain its read functionality for a period of time to facilitate monitoring and verification.

[0082] Remove the old engine-related parts of the bidirectional data pipeline to free up unused network bandwidth and other resources, while also optimizing the performance of the new engine.

[0083] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for upgrading a database, characterized in that: The method steps are as follows: S1. Establish a dual-engine parallel environment, start the old and new versions of the database engine, and build a cross-version communication channel. Specifically, the following steps are required: Deploy the new and old database versions in parallel in the computing cluster, allocate initial computing resources, and set up load-aware probes. Use the load-aware probes to collect dual-engine performance metrics, including CPU utilization. Establishing a bidirectional data pipeline comprising multiple levels of buffer queues, the buffer queues comprising three levels of queues: emergency, normal, and batch; dynamically monitoring key channel indicators based on the bidirectional data pipeline, wherein the key channel indicators include buffer fill rate; Resource rebalancing is triggered based on CPU utilization difference thresholds, and network bandwidth allocation is dynamically adjusted based on buffer fill rates; S2. In response to the communication channel established in step S1, the client request is routed to the new and old engines for execution in parallel through the transaction distributor, specifically including: Identify the transaction type requested by the client and extract key features; Calculate the real-time service capability index of the dual engines based on the dual-engine CPU utilization and buffer fill rate monitored in step S1; Write operations are sent to both engines simultaneously for execution. The results returned by the two engines are compared field by field. When the difference in execution time between the two engines exceeds a preset threshold, the transaction is rolled back and the exception is recorded. This result comparison ensures operational consistency. The optimal execution node is dynamically selected based on the cache hit rate and response latency of the two engines. S3. Based on the transaction operation log generated by the old engine in step S2, the data change characteristics are analyzed in real time and an incremental synchronization instruction set is generated; S4. Implementing version-coordinated transaction lock management when the new engine executes data replay based on the incremental synchronization instruction set generated in step S3. S5. After step S4 completes the data consistency verification for the preset period, switch the transaction routing to the new engine and close the old engine write channel; steps S1 to S5 use dual-engine parallel processing, incremental data synchronization and cross-version lock coordination mechanism to ensure that services are not interrupted and data maintains strong consistency during the database version upgrade process.

2. The database upgrade method according to claim 1, characterized in that: The process of S3 specifically includes: Parse transaction operation logs into atomic data change units; Merge by table dimension to generate a batch operation instruction set containing operation order constraints as the incremental synchronization instruction set.

3. The database upgrade method according to claim 1, characterized in that: The transaction lock management in S4 includes dynamically selecting a lock strategy based on the operation characteristics in the incremental synchronization instruction set, specifically including: Dynamically select table-level locks and row-level locks based on operation characteristics; Set a lock timeout mechanism that dynamically matches instruction execution time; Synchronize the dual-engine lock status through the cross-version communication channel.

4. The database upgrade method according to claim 3, characterized in that: The synchronous dual-engine lock state includes: When two engines request a mutex lock for the same resource, the old version lock is automatically released based on the preset priority. Allows dual engines to simultaneously apply shared locks to the same resource, supporting parallel reading.

5. The database upgrade method according to claim 1, characterized in that: The data consistency verification of the preset period includes: Perform full data comparison and real-time comparison of incremental changes within a preset period; Set a consistency tolerance threshold and terminate the verification process when the consistency tolerance threshold is exceeded.

6. The database upgrade method according to claim 1, characterized in that: The process of S5 specifically includes: The write operation suspension, transaction routing switch, and shutdown of the old engine write permission are performed in stages; Retain the old engine reading function for monitoring verification after the switch; Release the network bandwidth associated with the old engine and perform performance tuning on the new engine.

7. The database upgrade method according to claim 1, characterized in that: The process of triggering resource rebalancing according to the CPU utilization difference threshold specifically includes: By collecting the CPU utilization indicators of the two engines in real time, the difference between the two is periodically calculated and compared with the preset CPU utilization difference threshold. When the difference exceeds the CPU utilization difference threshold, the feedback control model is used to dynamically calculate the resource reallocation ratio based on the current load weight coefficients and transaction processing priorities of the two engines to prioritize the basic resource supply of the high-load engine. Quota adjustment instructions are sent to the resource scheduler based on the allocation ratio.

8. The database upgrade method according to claim 1, characterized in that: The process of dynamically adjusting network bandwidth allocation based on the buffer fill rate specifically includes: The buffer fill rate is monitored in real time. When the preset buffer fill rate threshold is reached, the bandwidth allocation ratio is dynamically calculated based on the current load and growth trend. The transmission parameters are optimized using the sliding window algorithm. The bandwidth weight of each channel is adjusted in real time through the traffic controller to form a dynamic adaptation mechanism between buffer capacity and network resources.

Images