A database full quantity synchronization breakpoint continuation method and system and a storage medium

Abstract

The application relates to the field of information technology, in particular to a database full-amount synchronization breakpoint continuation transmission method and system and a storage medium. The method comprises the following steps: selecting a split field of each table of a database, splitting the split field into a plurality of batches, and sequencing the batches; recording a starting value and an ending value and persistently storing the starting value and the ending value to an external storage; establishing a subtask thread pool and a sending window matched with the number of subtask threads; when breakpoint recovery occurs, reading the starting value and the ending value of the split field of the corresponding batch from the external storage; constructing a query condition, querying all data meeting the starting value and the ending value of the split field from a corresponding table of the database, taking the data as a recovered batch, and submitting the recovered batch to a subtask thread for synchronization. The application has the beneficial technical effects that the granularity of retransmission can be reasonably controlled, the balance between the data amount of retransmission and the concurrent amount is achieved, the speed during breakpoint recovery is high, and the application is easy to practice.

Description

Technical Field

[0001] This invention relates to the field of information technology, specifically to a method, system, and storage medium for full-database synchronous breakpoint resume transmission. Background Technology

[0002] Current mainstream database full synchronization technologies cannot achieve breakpoint resumption or cannot effectively control the granularity of breakpoint resumption, failing to achieve a balance between concurrency and the amount of data retransmitted. Traditional database full synchronization methods rely on retries to re-insert data when partial data import fails, rendering them ineffective in scenarios such as program crashes or machine restarts, and unable to achieve breakpoint resumption. Although there are publicly available technologies for achieving breakpoint resumption in full database synchronization, one approach is to mark the synchronization progress in real time during the synchronization process and resume transmission from the marked breakpoint upon recovery. However, this method suffers from difficulties in implementing in multi-threaded concurrent synchronization scenarios and low efficiency in single-threaded synchronization scenarios. Therefore, there is a need to research database synchronization technologies that can improve data synchronization efficiency.

[0003] Existing technology discloses a high-speed, highly stable, and breakpoint-resume data synchronization method based on the HTTP protocol, including a central system data synchronization module and a child node data synchronization module. The central system's data entry and management module processes updated business data, stores it, and then calls the central system's data update notification module. This notification module sends a notification to the child node's data update listening module via a message queue link. The listening module then calls the child node's data synchronization request module to request the central system's synchronization data download module, obtains the requested version data, and repeats the request process until the maximum version number of this type of business data on the child node matches the maximum version number of the central system. Data synchronization then stops, waiting for the next update notification from the central system or reaching a scheduled interval before requesting new version data from the central system's synchronization data download module. This technical solution can quickly, efficiently, and stably perform incremental data synchronization between a central system and multiple child nodes even under unstable network conditions. However, this technical solution cannot control the granularity of breakpoint resumption and is not suitable for multi-threaded parallel execution. Summary of the Invention

[0004] The technical problem this invention aims to solve is the lack of a database synchronization solution that supports breakpoint resumption and is suitable for multi-threaded parallel execution. This invention proposes a method, system, and storage medium for full database synchronization with breakpoint resumption, which enables breakpoint resumption with controllable breakpoint granularity and is suitable for multi-threaded parallel execution.

[0005] To solve the above technical problems, the present invention adopts the following technical solution: a method for resuming interrupted full-database synchronization, comprising the following steps: Select the splitting field for each table in the database, split the table data into several batches according to the splitting field, and sort the batches; Record the start and end values ​​of the corresponding splitting field for each batch, and persistently store the batch sorting number, the corresponding table name, start value, and end value to external storage. Establish a subtask thread pool and a sending window that matches the number of subtask threads. Submit batches within the sending window to the subtask threads for synchronization. After the batch synchronization is completed, update the position of the sending window. When a breakpoint occurs and recovery is needed, the start and end values ​​of the corresponding batch's splitting field are read from external storage. Construct query conditions, retrieve all data from the corresponding table in the database that satisfy the start and end values ​​of the segmentation field, and use them as a batch for recovery. Submit the recovered batch to the subtask thread for synchronization.

[0006] Preferred methods for selecting the segmentation field include: If the table in the database is a single primary key table, then the splitting field of the table is selected as the primary key; If the database table has a composite primary key, then the field with the highest distinguishability is selected as the splitting field of the table.

[0007] As a preferred method, the method for calculating the distinctness of a field is: distinctness = count(distinct field_name) / count(field_name), where count(distinct field_name) refers to the number of entries in the table after removing duplicate field values, and count(field_name) refers to the number of entries in the table.

[0008] As a preferred method, the method of dividing table data into several batches based on the splitting field includes: Set the number of batches to n, and read the minimum value change step m of the segmentation field; Calculate the partitioning step size: step = max(count(segmentation field) / n, m); The values ​​of the splitting field are divided into several intervals according to the step size, and the entries of the table corresponding to each interval are treated as a batch.

[0009] Preferred methods for establishing a sending window include: Set the size of the sending window to be equal to the number of subtask threads; Set the starting position of the sending window to the first batch, establish an ack pointer pointing to the starting position of the sending window, and persistently store the ack pointer in external storage.

[0010] Preferably, the external storage also stores the synchronization status of batches. The initial value of the synchronization status is "waiting for scheduling". After the subtask thread pool is established, the synchronization status of the batches submitted to the subtask thread is set to "sending". After the batch synchronization is completed, the synchronization status of the corresponding batch is updated to "success". If the batch synchronization is interrupted, the synchronization is immediately retried until the synchronization is completed. The method for updating the position of the sending window is as follows: when the batch at the beginning position of the sending window is successfully synchronized, the ack pointer is incremented by 1 until the end position of the sending window reaches the last batch.

[0011] Preferably, when a breakpoint occurs, the position of the sending window is restored according to the ack pointer, the batch that needs to be restored is determined according to the restored position of the sending window, and the restored batch is submitted to the subtask thread for synchronization.

[0012] A database full synchronization breakpoint resume system is provided for executing a database full synchronization breakpoint resume method as described above. The system includes a management node cluster, a coordinator cluster, and a synchronization node cluster. The management node cluster includes at least one management node, the coordinator cluster includes at least one coordinator, and the synchronization node cluster includes at least two synchronization nodes. The management node selects the sharding field for each table in the database of the data source, divides the table data into several batches based on the sharding field, sorts the batches, records the start and end values ​​of the sharding field for each batch, and persistently stores the batch sorting number, the corresponding table name, start value, and end value to external storage. The two synchronization nodes run on the host containing the source database and the target database, respectively. The synchronization node corresponding to the source database establishes a subtask thread pool and a sending window matching the number of subtask threads. Batches within the sending window are submitted to the subtask threads for synchronization. After batch synchronization is complete, the synchronization node corresponding to the source database updates the position of the sending window. When a breakpoint occurs and recovery occurs, the synchronization node corresponding to the source database reads the start and end values ​​of the segmentation field for the corresponding batch from external storage. The synchronization node corresponding to the source database constructs query conditions and queries the corresponding table in the database for all data that satisfies the start and end values ​​of the segmentation field, which are then used as the batch to be recovered. The synchronization node corresponding to the source database submits the recovered batch to the subtask threads for synchronization. The coordinator establishes a network connection between the management node and several of the synchronization nodes.

[0013] Preferably, the synchronization node corresponding to the source database extracts the digital fingerprint of the batch and sends it to the synchronization node corresponding to the target database. After the subtask thread of the synchronization node corresponding to the target database completes the batch transmission, it verifies the digital fingerprint. If the verification matches, it reports that the batch synchronization was successful; otherwise, if the verification does not match, it reports that the batch synchronization failed.

[0014] Preferably, when the management node selects the sharding field for each table in the database of the data source, it performs the following steps: If the table in the database is a single primary key table, then the splitting field of the table is selected as the primary key; If the database table has a composite primary key, then the field with the highest distinctness is selected as the splitting field of the table. The method for calculating the distinctness of a field is: distinctness = count(distinct field name) / count(field name), where count(distinct field name) refers to the number of entries in the table after removing duplicate field values, and count(field name) refers to the number of entries in the table.

[0015] Preferably, when the management node divides the table data into several batches based on the splitting field, it performs the following steps: Set the number of batches to n, and read the minimum value change step m of the segmentation field; Calculate the partitioning step size: step = max(count(segmentation field) / n, m); The values ​​of the splitting field are divided into several intervals according to the step size, and the entries of the table corresponding to each interval are treated as a batch.

[0016] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements a database full-database synchronous breakpoint resume method as described above.

[0017] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a database full-synchronization breakpoint resume method as described above.

[0018] The beneficial technical effects of this invention include: the data volume of each batch of data obtained during full synchronization is approximately equal; it can be segmented according to character fields and is not affected by the character set; it can reasonably control the granularity of retransmission, achieving a balance between the amount of retransmitted data and the concurrency; it only requires storing the data segmentation and a small amount of status information in the external storage system; it is fast in breaking point recovery and easy to implement.

[0019] Other features and advantages of the present invention will be disclosed in detail in the following detailed description and accompanying drawings. Attached Figure Description

[0020] The invention will be further described below with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the synchronous breakpoint resume method according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram illustrating the method for dividing table data into several batches according to an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of the synchronous breakpoint resume transmission system according to an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of the computer system structure according to an embodiment of the present invention.

[0024] Among them: 10, management node cluster; 20, coordinator cluster; 30, synchronization node cluster; 40, computer system; 41, memory; 42, computer program; 43, processor. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of the present invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of the present invention.

[0026] In the following description, terms such as “inner,” “outer,” “upper,” “lower,” “left,” and “right” are used only to indicate orientation or positional relationship for the convenience of describing the embodiments and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0027] A method for resuming interrupted full-database synchronization is provided in the appendix. Figure 1 This includes the following steps: Step A01) Select the splitting field for each table in the database, split the table data into several batches based on the splitting field, and sort the batches; Step A02) Record the start and end values ​​of the corresponding splitting field for each batch, and persist the batch sorting number and the corresponding table name, start value and end value to external storage; Step A03) Establish a subtask thread pool and a sending window that matches the number of subtask threads. Synchronize the batch submission subtask threads within the sending window. After the batch synchronization is completed, update the position of the sending window. Step A04) When a breakpoint occurs and recovery is needed, read the start and end values ​​of the corresponding batch's splitting field from external storage; Step A05) Construct query conditions: Retrieve all data from the corresponding table in the database that meets the start and end values ​​of the splitting field, and use this as a recovery batch. Submit the recovery batch to the subtask thread for synchronization. Construct a batch of start and end interval information and persist it to an external system. Each interval represents a concurrently executable synchronization subtask. This achieves controllable granularity for breakpoint resumption and is suitable for multi-threaded parallel execution of data synchronization methods. Each thread can be responsible for synchronizing multiple subtasks; the subtasks can be sorted in time.

[0028] Methods for selecting the splitting field include: if the database table is a single primary key table, then select the splitting field of the table as the primary key; if the database table is a composite primary key, then select the field with the highest distinguishability as the splitting field of the table.

[0029] The method for calculating the distinctness of a field is: Distinctiveness = count(distinct field_name) / count(field_name), where count(distinct field_name) refers to the number of entries in the table after removing duplicate field values, and count(field_name) refers to the number of entries in the table.

[0030] Please see the appendix Figure 2 Methods for splitting table data into several batches based on the splitting field include: Step B01) Set the number of batches to n, and read the minimum value change step m of the splitting field; Step B02) Calculate the partitioning step size step, step = max(count(segmentation field) / n, m); Step B03) Divide the values ​​of the splitting field into several intervals according to the splitting step size step, and the entries of the table corresponding to each interval are treated as a batch.

[0031] Taking the splitting of numeric fields as an example: Taking 100 records as an example, the values ​​of this field are 1, 2, 3...99, 5000, with a minimum value of 1 and a maximum value of 5000. The goal is to divide the data into 5 parts. Traditional or other patented methods calculate the maximum and minimum values ​​of the field and construct an arithmetic sequence to divide the data, resulting in [1, 1000], [1001, 2000], [2001, 3000], [3001, 4000], [4001, 5000]. The data segmentation method in this embodiment results in [1, 20], [21, 40], [41, 60], [61, 80], [81, 5000].

[0032] It is evident that when table data is unevenly distributed, traditional or other patented methods of data segmentation have significant errors, leading to a large accumulation of data in one batch while other batches contain very little data, making it impossible to achieve a relatively balanced distribution of data. The data segmentation method in this embodiment can solve the data skew problem.

[0033] Taking the splitting of character fields as an example: Taking 1000 randomly generated 6-digit strings containing only 0-9 and az as an example, the minimum value of this field in lexicographical order is 001222, and the maximum value is ffe732. The goal is to split the string into 10 parts. Traditional methods cannot split the string. The splitting method of the open-source database full synchronization software dataX: performs string mapping. Specifically, it iterates through the string from left to right. The string length is k. The product of the ACSI value of the i-th string and 128^(ki) constructs a real number, which is then accumulated and converted into a numeric field for splitting. After splitting, it is mapped back to a string. This embodiment of the method: utilizes the natural ordering of the index to perform fast splitting by scanning the index without needing to look up the table.

[0034] As can be seen from the segmentation results, the segmentation method used by dataX involves a mapping between strings and numbers. After the obtained intervals are mapped back to strings, the sorting of the intervals under different character sets is not necessarily smaller than the substring of the previous interval. That is, it is affected by the character set, and there are problems of interval repetition and data skew. The segmentation results of the method in this embodiment are independent of the character set and are well visualized.

[0035] This data partitioning method fully utilizes database indexes, eliminating the need for table lookups when only the index is scanned, resulting in extremely high speed. Compared to methods that calculate the maximum and minimum values ​​of fields and construct arithmetic sequences to partition data, this embodiment's data partitioning method avoids data skew issues in scenarios with uneven table data distribution, ensuring that the amount of data in each batch is approximately equal. Compared to traditional methods, it solves the problem of not being able to partition based on string type fields. Compared to methods that map strings to numbers before partitioning, this embodiment leverages the inherently ordered nature of indexes, resolving the inaccuracy issues caused by character set limitations.

[0036] The method for establishing a sending window in this embodiment includes: setting the size of the sending window equal to the number of subtask threads; setting the starting position of the sending window in the first batch; establishing an ack pointer pointing to the starting position of the sending window; and persistently storing the ack pointer in external storage.

[0037] The external storage also stores the synchronization status of batches. The initial value of the synchronization status is "waiting for scheduling". After the subtask thread pool is established, the synchronization status of the batches submitted to the subtask thread is set to "sending". After the batch synchronization is completed, the synchronization status of the corresponding batch is updated to "success". If the batch synchronization is interrupted, the synchronization is immediately retried until the synchronization is completed. The method to update the position of the sending window is as follows: when the batch at the beginning position of the sending window is successfully synchronized, the ack pointer is incremented by 1 until the end position of the sending window reaches the last batch.

[0038] During synchronization, a sending window of equal size is constructed based on the set subtask thread pool size. Subtasks with corresponding numbers within the sending window are concurrently submitted to the thread pool. If a subtask fails or times out, it is re-added to the task thread pool for retry. Whenever a subtask completes, the flag in the number array within the sending window is updated, and an attempt is made to update the ack pointer and slide the sending window forward. The current and previous positions of the ack pointer represent the completed task numbers. After each update of the ack pointer, its position is persisted to the storage system.

[0039] When a breakpoint is recovered, the position of the sending window is restored according to the ack pointer. The batch that needs to be recovered is determined based on the restored position of the sending window, and the recovered batch is submitted to the subtask thread for synchronization.

[0040] The beneficial technical effects of this embodiment include: the data volume of each batch of data obtained during full synchronization is approximately equal; it can be segmented according to character fields and is not affected by the character set; it can reasonably control the granularity of retransmission, achieving a balance between the amount of retransmitted data and the amount of concurrency; it only requires storing the data segmentation and a small amount of status information in the external storage system; it is fast when recovering from breakpoints and is easy to implement.

[0041] A database full-database synchronous breakpoint resume system is provided for executing the aforementioned database full-database synchronous breakpoint resume method. Please refer to the appendix. Figure 3 It includes a management node cluster 10, a coordinator cluster 20, and a synchronization node cluster 30. The management node cluster 10 includes at least one management node, the coordinator cluster 20 includes at least one coordinator, and the synchronization node cluster 30 includes at least two synchronization nodes. The management node selects the sharding field for each table in the database of the data source, shards the table data into several batches based on the sharding field, sorts the batches, records the start and end values ​​of the corresponding sharding field for each batch, and persists the batch sorting number, the corresponding table name, start value, and end value to external storage. Two synchronization nodes run on the host containing the source database and the target database, respectively. The synchronization node for the source database establishes a subtask thread pool and a sending window matching the number of subtask threads. Batches within the sending window are submitted to the subtask threads for synchronization. After batch synchronization is complete, the synchronization node for the source database updates the position of the sending window. When a breakpoint occurs and recovery is needed, the synchronization node for the source database reads the start and end values ​​of the segmentation field for the corresponding batch from external storage. It then constructs query conditions and retrieves all data from the corresponding database table that satisfies the start and end values ​​of the segmentation field. This retrieved data is used as the batch to be recovered, and the synchronization node for the source database submits this recovered batch to the subtask threads for synchronization. The coordinator establishes network connections between the management node and several synchronization nodes.

[0042] The synchronization node corresponding to the source database extracts the digital fingerprint of the batch and sends it to the synchronization node corresponding to the target database. After the subtask thread of the synchronization node corresponding to the target database completes the batch transmission, it verifies the digital fingerprint. If the verification matches, it reports that the batch synchronization was successful; otherwise, if the verification does not match, it reports that the batch synchronization failed.

[0043] When the management node selects the sharding field for each table in the database of the data source, it performs the following steps: If the database table is a single primary key table, then select the table's splitting field as the primary key; If the database table has a composite primary key, then the field with the highest distinctness is selected as the table's splitting field. The method for calculating the distinctness of a field is: distinctness = count(distinct field_name) / count(field_name), where count(distinct field_name) refers to the number of entries in the table after removing duplicate field values, and count(field_name) refers to the number of entries in the table.

[0044] When the management node splits table data into several batches based on the splitting field, it performs the following steps: Set the number of batches to n, and read the minimum value change step m of the segmentation field; Calculate the partitioning step size: step = max(count(segmentation field) / n, m); The values ​​of the splitting field are divided into several intervals according to the step size, and the entries of the table corresponding to each interval are treated as a batch.

[0045] On the other hand, this embodiment provides a verification example of the database full-synchronization breakpoint resume method.

[0046] The source database is a MySQL database, and the target database business system is a PostgreSQL database. The table creation statement for the test.table table in the demo database is as follows: CREATE TABLE `test_table` ( `sid` varchar (6) NOT NULL, `name` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `type` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL, `city` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; 1000 randomly generated 6-digit strings containing only 0-9 and az are used as the values ​​for the sid field. Other fields contain arbitrary random values. The sid field value is lexicographically ordered, with a minimum of 001222 and a maximum of ffe732. The number of pre-segmentation batches is set to 10, the minimum step size to 10000, and the synchronization subtask thread pool size to 5. A full synchronization operation is then performed on this table.

[0047] Implementation steps: 1. Using this patented system to synchronize table data, simply fill in the corresponding source database information and target server information, and the synchronization mapping relationship can be automatically generated with one click; 2. Select the splitting field. Since this table is a single primary key table, the sid field is selected as the splitting field.

[0048] 3. Execute the character segmentation algorithm. The first query is `select sid from test_table wheresid > '001222' limit 100,1;`, which returns the result 1b52a7. The second query is `select sid from test_table where sid > '1b52a7' limit 100,1;`, and so on, iterating until the result set is empty. Collect the results of each query and construct the synchronization interval [001222,1b52a7), [1b52a7,332ab6), [332ab6,50799a), [50799a,69a3d2), [69a3d2,84bcfb), [84bcfb,a1a75d), [a1a75d,b8398c), [b8398c,d35be2), [d35be2,ecae00), [ecae00,ffe732]. Numbered 1, 2, ..., 10, the interval information is persisted to an external storage system. Each synchronization interval represents a subtask that can be executed concurrently, and each subtask is independently responsible for data synchronization within its own interval.

[0049] 4. Initialize the ack pointer to position 0, construct a sending window of size 5, and concurrently submit 5 subtasks to the thread pool for full synchronization. If a subtask fails or times out, it is re-added to the task thread pool for retry. Whenever a subtask completes, update the flag in the number array within the sending window, attempt to update the ack pointer, and slide the sending window forward. The current and previous positions of the ack pointer represent the completed task numbers. After each update of the ack pointer, persist its position to the storage system. After t1 seconds, subtask number 3 completes synchronization. At this time, the ack pointer does not meet the update condition and still points to position 0. After t2 seconds, subtask number 2 fails, and it is re-added to the task thread pool for retry. After t3 seconds, subtask number 2 completes synchronization. At this time, the ack pointer does not meet the update condition and still points to position 0. After t4 seconds, subtask number 1 completes synchronization. At this time, the ack pointer meets the update condition, moves the pointer to position 3, and persists the pointer position to external storage. Slide the sending window forward 3 units.

[0050] 5. After t5 seconds, a power outage event occurs on the machine where the synchronization system is located, and the synchronization is interrupted.

[0051] 6. After the machine restarts, a full synchronization is initiated again. The interval information and the position of the ack pointer are read from the external storage system, and the running status array is reconstructed. The starting position of the reconstructed sending window is located at number 4, and the position of the ack pointer is located at number 3. Five subtasks numbered 4-8 are submitted concurrently to the subtask thread pool, and then the "synchronization and status marking" process is repeated until all batches of data are synchronized.

[0052] On the other hand, this application provides a computer system, please refer to the appendix. Figure 4 The computer system 40 includes a memory 41, a processor 43, and a computer program 42 stored in the memory 41 and executable on the processor 43. When the computer program 42 is executed by the processor 43, it implements a database full-database synchronous breakpoint resume method as described above.

[0053] The computer system can be a general-purpose computer system or a special-purpose computer system. In specific implementations, the computer system can be a server cluster including multiple servers, such as a blockchain system including multiple nodes. Those skilled in the art will understand that... Figure 4 This is merely an example of a computer system and does not constitute a limitation on the computer system. It may include more or fewer components than shown in the figure, or combinations of certain components, or different components, such as input / output devices, network access devices, etc.

[0054] Processor 43 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processor 43 can be a microprocessor or any conventional processor.

[0055] In some embodiments, memory 41 may be an internal storage unit of a computer system, such as a hard disk or RAM. In other embodiments, memory 41 may be an external storage device of a computer system, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Furthermore, memory 41 may include both internal and external storage units of the computer system. Memory 41 is used to store the operating system, applications, boot loader, data, and other programs. Memory 41 can also be used to temporarily store data that has been output or will be output.

[0056] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program 42, which, when executed by a processor 43, implements a database full-synchronization breakpoint resume method as described above.

[0057] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A method for resuming interrupted full-database synchronization, characterized in that, Includes the following steps: Select the splitting field for each table in the database, split the table data into several batches according to the splitting field, and sort the batches; Record the start and end values ​​of the corresponding splitting field for each batch, and persistently store the batch sorting number, the corresponding table name, start value, and end value to external storage. Establish a subtask thread pool and a sending window matching the number of subtask threads, and set the size of the sending window equal to the number of subtask threads; set the starting position of the sending window to the first batch, establish an ack pointer pointing to the starting position of the sending window, and persistently store the ack pointer in external storage; The batches within the sending window are submitted to the subtask thread for synchronization. The external storage also stores the synchronization status of the batches. The synchronization status of the batches submitted to the subtask thread is set to sending. After the batch synchronization is completed, the synchronization status of the corresponding batch is updated to successful. The method for updating the position of the sending window is as follows: after the batch synchronization at the starting position of the sending window is successful, the ack pointer is incremented by 1 until the ending position of the sending window reaches the last batch; When a breakpoint is recovered, the position of the sending window is restored according to the ack pointer, the batch that needs to be recovered is determined according to the restored position of the sending window, and the start and end values ​​of the segmentation field of the corresponding batch are read from external storage. Construct query conditions, retrieve all data from the corresponding table in the database that satisfy the start and end values ​​of the segmentation field, and use them as a batch for recovery. Submit the recovered batch to the subtask thread for synchronization.

2. The database full-scale synchronization breakpoint resumption method according to claim 1, characterized in that, Methods for selecting the splitting field include: If the table in the database is a single primary key table, then the splitting field of the table is selected as the primary key; If the database table has a composite primary key, then the field with the highest distinguishability is selected as the splitting field of the table.

3. The method for resuming interrupted full-database synchronization data transmission according to claim 2, characterized in that, The method for calculating the distinctness of a field is: Distinctiveness = count(distinct field_name) / count(field_name), where count(distinct field_name) refers to the number of entries in the table after removing duplicate field values, and count(field_name) refers to the number of entries in the table.

4. A method for resuming interrupted full-database synchronization data transmission according to any one of claims 1 to 3, characterized in that, Methods for splitting table data into several batches based on the splitting field include: Set the number of batches to n, and read the minimum value change step m of the segmentation field; Calculate the partitioning step size, step = max(count(segmentation field) / n, m); The values ​​of the splitting field are divided into several intervals according to the step size, and the entries of the table corresponding to each interval are treated as a batch.

5. A database full-synchronization breakpoint resume system, used to execute the database full-synchronization breakpoint resume method as described in any one of claims 1 to 4, characterized in that, The system includes a management node cluster, a coordinator cluster, and a synchronization node cluster. The management node cluster includes at least one management node, the coordinator cluster includes at least one coordinator, and the synchronization node cluster includes at least two synchronization nodes. Each management node selects a partitioning field for each table in the source database, partitions the table data into several batches based on the partitioning field, sorts the batches, records the start and end values ​​of the partitioning field for each batch, and persistently stores the batch sorting number, corresponding table name, start and end values ​​in external storage. The two synchronization nodes run on the host where the source database and target database reside, respectively. The synchronization node corresponding to the source database establishes a subtask thread pool and a sending window matching the number of subtask threads, setting the size of the sending window equal to the number of subtask threads. The starting position of the sending window is set to the first batch, and an ack pointer is established pointing to the starting position of the sending window. The ack pointer is persistently stored in external storage. Batches within the sending window are submitted to subtask threads for synchronization. The external storage also stores the synchronization status of the batches. The synchronization status of the batch submitted to the subtask thread is set to "sending". After the batch synchronization is completed, the synchronization status of the corresponding batch is updated to "success". The method for updating the position of the sending window is as follows: when the batch at the starting position of the sending window is successfully synchronized, the ack pointer is incremented by 1 until the end position of the sending window reaches the last batch. After the batch synchronization is completed, the synchronization node corresponding to the source database updates the position of the sending window. When a breakpoint recovery occurs, the synchronization node corresponding to the source database restores the position of the sending window according to the ack pointer. Based on the restored position of the sending window, the current batch that needs to be restored is determined. The start value and end value of the segmentation field of the corresponding batch are read from external storage. The synchronization node corresponding to the source database constructs query conditions and queries all data that satisfy the start value and end value of the segmentation field from the corresponding table in the database. These are used as the batch to be restored. The synchronization node corresponding to the source database submits the restored batch to the subtask thread for synchronization. The coordinator establishes a network connection between the management node and several synchronization nodes.

6. A database full-synchronization breakpoint resume system according to claim 5, characterized in that, The synchronization node corresponding to the source database extracts the digital fingerprint of the batch and sends it to the synchronization node corresponding to the target database. After the subtask thread of the synchronization node corresponding to the target database completes the batch transmission, it verifies the digital fingerprint. If the verification matches, it reports that the batch synchronization was successful; otherwise, if the verification does not match, it reports that the batch synchronization failed.

7. A database full-database synchronization breakpoint resume system according to claim 5 or 6, characterized in that, When the management node selects the sharding field for each table in the database of the data source, it performs the following steps: If the table in the database is a single primary key table, then the splitting field of the table is selected as the primary key; If the database table has a composite primary key, then the field with the highest distinctness is selected as the splitting field of the table. The method for calculating the distinctness of a field is: distinctness = count(distinct field name) / count(field name), where count(distinct field name) refers to the number of entries in the table after removing duplicate field values, and count(field name) refers to the number of entries in the table.

8. A database full-synchronization breakpoint resume system according to claim 7, characterized in that, When the management node divides the table data into several batches based on the splitting field, it performs the following steps: Set the number of batches to n, and read the minimum value change step m of the segmentation field; Calculate the partitioning step size, step = max(count(segmentation field) / n, m); The values ​​of the splitting field are divided into several intervals according to the step size, and the entries of the table corresponding to each interval are treated as a batch.

9. A computer system, characterized in that, The computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements a database full-database synchronous breakpoint resume method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements a database full-database synchronous breakpoint resume method as described in any one of claims 1 to 4.

Images