Data synchronization method, database system, equipment and storage medium
By transmitting large-scale data in segments and processing in parallel in database asynchronous replication, the problems of low data replication efficiency and slow speed in database asynchronous replication are solved, and efficient and stable data transmission and processing are achieved.
Patent Information
- Application Number
- CN202410936698.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-08
AI Technical Summary
In the asynchronous replication process of databases, the existing technology has problems such as low data replication efficiency and low completion speed when processing large-scale data, and lacks an effective concurrency processing mechanism, resulting in high transmission failure rate and low availability.
By transmitting large-scale data in segments and processing tasks to be synchronized in parallel based on execution group information in the backup cluster, managing tasks with different processing priorities, avoiding mutual blockage of cross-database task execution, and realizing parallelization of data transmission processes and resources Reasonable distribution.
It significantly improves the efficiency and speed of asynchronous data replication, reduces the pressure of single transmission, avoids waiting delays for task execution, and improves the reliability and stability of data transmission.
Smart Images

Figure CN118916155A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data synchronization method, a database system, a device, and a storage medium. Background Art
[0002] With the explosive growth of data volume and the increasing demand for high availability, asynchronous database replication technology has become an important means to ensure data consistency, achieve data backup and disaster recovery. Asynchronous replication allows data to be replicated between databases in different geographical locations without waiting for real-time responses, which greatly improves system availability and performance.
[0003] In the prior art, when a database system performs asynchronous replication, the primary cluster uses data stream transmission to send tasks to be synchronized to the backup cluster. After receiving the tasks to be synchronized, the backup cluster uses serial processing to execute them to achieve data synchronization between the primary and backup clusters.
[0004] However, when using existing technologies to process large-scale data, there are problems such as low data replication efficiency and low completion speed. Summary of the invention
[0005] The purpose of this application is to provide a data synchronization method, database system, device and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problems of low data replication efficiency and low completion speed when processing large-scale data in the prior art.
[0006] To achieve the above objectives, the technical solutions adopted in this application are as follows:
[0007] In a first aspect, the present application provides a data synchronization method, which is applied to a standby cluster in a database system, wherein the standby cluster includes: a control node and multiple data nodes, and the method includes:
[0008] The data node sends a task data request message to the main cluster based on the task information and execution group information sent by the main cluster, the execution group information includes: a database and a processing priority of the database, the task information is used to indicate a plurality of tasks to be synchronized, and the execution group information is the execution group information of the tasks to be synchronized;
[0009] The data node receives a plurality of segmented data from the main cluster, where the segmented data is a data segment of a task to be synchronized;
[0010] The data node assembles each of the segmented data according to the segment identifier of each of the segmented data to obtain a task to be synchronized, wherein the segment identifier is used to indicate whether the segmented data is the last segmented data of the multiple segmented data;
[0011] The data node determines the execution group to which the task to be synchronized belongs according to the execution group information of the task to be synchronized, sends the task to be synchronized and the execution group information of the task to be synchronized to the thread corresponding to the execution group, and each of the threads executes the task to be synchronized in parallel.
[0012] Optionally, the data node assembles each of the segmented data according to the segment identifier of each of the segmented data to obtain the task to be synchronized, including:
[0013] The data node determines the last segment data among the plurality of segment data according to the segment identifier of each segment data;
[0014] The data node assembles and verifies each segmented data according to the offset of each segmented data to obtain a task to be synchronized.
[0015] Optionally, the data node sends each of the tasks to be synchronized to a thread corresponding to the execution group according to the execution group information of each of the tasks to be synchronized, including:
[0016] The data node determines at least one candidate thread corresponding to the execution group according to the execution group information of the task to be synchronized;
[0017] The data node determines the target thread with the smallest load according to the number of tasks and / or execution time information of each of the threads to be selected, and sends each of the tasks to be synchronized to the corresponding target thread.
[0018] Optionally, before the data node sends each of the tasks to be synchronized to a thread corresponding to the execution group according to the execution group information of each of the tasks to be synchronized, the method further includes:
[0019] The data node determines multiple tasks to be synchronized that meet preset conditions and merges the tasks to be synchronized into one task to be synchronized. The preset conditions include: the target databases and target data tables of the tasks to be synchronized are the same, and the tasks to be synchronized are write tasks.
[0020] Optionally, the data node sends a task data request message to the main cluster based on the task information and execution group information sent by the main cluster, including:
[0021] The control node receives the task information set, execution group information and conflict detection result sent by the main cluster, and sends the task information and execution group information in the task information set to each of the data nodes respectively according to the conflict detection result;
[0022] The data node generates a task data request message according to the received task information, and sends the task data request message to the main cluster.
[0023] In a second aspect, the present application provides a data synchronization method, which is applied to a main cluster in a time series database, wherein the main cluster includes: a data node, and the method includes:
[0024] The data node of the main cluster receives the task data request message from the standby cluster, and determines a plurality of tasks to be synchronized according to the task data request message;
[0025] The data node performs segment processing on the multiple tasks to be synchronized based on a preset segment size and adds segment identifiers to obtain multiple segment data;
[0026] The data node sends the multiple segmented data to the standby cluster.
[0027] Optionally, the main cluster further includes a control node, and the method further includes:
[0028] The control node performs conflict detection on multiple tasks to be sent, and determines the conflict detection results of each task to be sent;
[0029] The control node determines the multiple tasks to be synchronized according to the conflict detection result, and sends the task information set, execution group information and the conflict detection result of the multiple tasks to be synchronized to the control node of the standby cluster.
[0030] In a third aspect, the present application provides a database system, comprising: a primary cluster and multiple backup clusters as described in the first aspect and the second aspect, wherein the primary cluster and the backup clusters are used to execute the steps of the data synchronization method in the first aspect and the second aspect.
[0031] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of a data synchronization method as described in any one of the first aspect and the second aspect.
[0032] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a data synchronization method as described in any one of the first and second aspects are executed.
[0033] The beneficial effects of the present application are: by transmitting tasks with large amounts of data in segments, the pressure of single transmission can be effectively reduced, and tasks with small amounts of data can be packaged and sent at one time. By transmitting tasks with large amounts of data in segments, the parallelization of the data transmission process can also be achieved, thereby significantly improving the efficiency of data transmission. By sending the tasks to be synchronized to the corresponding threads for synchronization based on the execution group information, the tasks to be synchronized with different processing priorities can be managed and executed, and the execution processes of the tasks to be synchronized in each execution group can be made relatively independent and will not block each other. For example, there is a group of more urgent small tasks in the system, which can be assigned to a separate high-priority execution group and separated from the remaining large number of general tasks. Even if the tasks in a certain execution group are executed slowly, it will not affect the execution of more urgent tasks in other execution groups, thereby effectively avoiding the mutual blocking of cross-database task execution and reducing the delay of task execution waiting.
[0034] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0036] Figure 1 A schematic diagram of the architecture of a database system provided in an embodiment of the present application is shown;
[0037] Figure 2 A flow chart of a data synchronization method on a standby cluster side provided by an embodiment of the present application is shown;
[0038] Figure 3 A flow chart of assembling segmented data provided by an embodiment of the present application is shown;
[0039] Figure 4 A flowchart of sending a task to be synchronized to a corresponding thread provided by an embodiment of the present application is shown;
[0040] Figure 5 A flow chart showing a data node sending a task data request message to a main cluster provided by an embodiment of the present application is shown;
[0041] Figure 6 A flow chart of a data synchronization method on a master cluster side provided in an embodiment of the present application is shown;
[0042] Figure 7 A flow chart of performing conflict detection provided by an embodiment of the present application is shown;
[0043] Figure 8 An overall flow chart of a data synchronization method provided by an embodiment of the present application is shown;
[0044] Fig. 9 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0045] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0046] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0047] Asynchronous replication means that the data modification operations of the primary cluster and the standby cluster in the database system are asynchronous. That is, a data modification operation needs to be completed on the primary cluster before being asynchronously sent to the standby cluster. During this period, there may be data inconsistencies between the primary and standby clusters, but if there is no new data modification in the primary cluster, the data of the primary and standby clusters will eventually be consistent after the slave cluster executes all previous data modification operations. Asynchronous replication technology allows data to be replicated between databases in different geographical locations without waiting for a response in real time.
[0048] In the prior art, the main cluster generally transmits the tasks that need to be synchronized to the standby cluster using a data stream transmission mode, and the standby cluster then processes the data using a serial processing method.
[0049] However, with the increasing complexity of application scenarios, when synchronizing large-scale data, if a network interruption occurs during the data transmission phase, the entire data set needs to be retransmitted, which not only increases the data transmission time, but also may cause inconsistency in data synchronization. Therefore, the existing data transmission method will increase the transmission failure rate and reduce availability. In addition, the existing technology lacks an effective concurrent processing mechanism and cannot handle the replication tasks of multiple databases at the same time. When multiple database instances need to be synchronized, the processing efficiency is low.
[0050] In addition, the existing technology does not have sufficient management of task priorities and resource allocation, which may result in important replication tasks not being executed in time because resources are occupied by other tasks. For example, assuming that there are a large number of ordinary tasks and a small number of urgent tasks in the database system, since there is no priority in the existing asynchronous replication technology, the urgent tasks that should be executed first may be blocked by a large number of ordinary priority tasks.
[0051] In response to the above problems, the present application proposes a data synchronization method that transmits large-scale data in segments, which can not only effectively reduce the pressure of a single transmission, but also realize the parallelization of the data transmission process and the data processing process, and realize the reasonable allocation of processing resources for tasks based on priority, thereby significantly improving the efficiency and speed of asynchronous data replication.
[0052] Figure 1 This is a schematic diagram of the architecture of a database system given in this application. Figure 1 The database system includes a primary cluster and at least one standby cluster, and the standby cluster includes a control node and multiple data nodes. The primary cluster can be the cluster that actually performs data modification operations, and the standby cluster can be a standby cluster used for disaster recovery. The standby cluster can receive data modification operations sent by the primary cluster and make the same modifications on the standby cluster. When asynchronous replication is in progress, the standby cluster is generally only used for data reading and does not modify data.
[0053] The data synchronization method of the present application can be applied to Figure 1 In the database system shown in FIG. 1 , the data synchronization method of the present application is described from both the main cluster and the backup cluster. Figure 2 The following is a step-by-step description of the data synchronization method on the standby cluster side. Figure 2 , the method comprising:
[0054] Step 201, the data node sends a task data request message to the main cluster based on the task information and execution group information sent by the main cluster. The execution group information includes: a database and a processing priority of the database. The task information is used to indicate multiple tasks to be synchronized, and the execution group information is the execution group information of the tasks to be synchronized.
[0055] In the database system of the present application, a data modification operation can be regarded as an asynchronous replication task. The task is bound to the transaction. Each task has a globally unique self-incrementing task ID, group ID, transaction ID corresponding to the task, data partition involved in the task, and actual modified data and other attributes.
[0056] Reference Figure 1 The standby cluster includes a control node and multiple data nodes. The control node of the standby cluster can receive the task information sent by the main cluster and distribute the received task information to each data node. After receiving the task information, the data node can send a task data request message to the main cluster based on the task information. The task data request message includes the task offset requested by the standby cluster data node. The task offset can be the task ID of the currently completed task in the data node. After receiving the task data request message sent by the standby cluster data node, the main cluster can determine the task that needs to be copied to the standby cluster according to the offset in the task data request message.
[0057] Optionally, an execution group can be an execution unit with independent priority and execution resources. One execution group corresponds to multiple database groups. Each database group includes multiple tasks to be synchronized for the same database. The tasks in each database group are arranged in the order of task ID to ensure the order and correctness of task execution. Among them, the databases corresponding to the same execution group have the same processing priority. The execution resources used by tasks in different execution groups are different, so tasks in different execution groups will not preempt and block each other. Execution groups with higher priorities have more priority execution resources to achieve reasonable scheduling of tasks with different urgency levels.
[0058] The execution group information includes the database and the priority corresponding to the database. Specifically, the execution group information may indicate the execution group to which all databases related to the task to be synchronized are located, and the processing priority of each database. The processing priority of the database may be the priority of the execution group to which the database is located, and the database related to the task to be synchronized may be the database to which the task to be synchronized writes data or reads data.
[0059] Step 202: The data node receives a plurality of segmented data from the main cluster, where the segmented data is a data segment of a task to be synchronized.
[0060] Optionally, the main cluster may determine whether segmented transmission is required for the task to be synchronized based on a preset segment size.
[0061] In the first implementation method, if the total amount of data of the tasks to be synchronized is less than the preset segment size, all the tasks to be synchronized can be packaged and sent to the control node of the standby cluster through a single RPC (Remote Procedure Call) request, thereby ensuring that when there are multiple tasks to be synchronized with small data amounts, network resources can be used efficiently and data transmission can be completed with the least RPC calls.
[0062] In the second implementation method, if the total amount of data of the tasks to be synchronized is greater than the preset segment size, and there is at least one task among the tasks to be synchronized whose data amount is greater than the preset segment size, then the tasks to be synchronized with smaller data amount can be processed first, and all the tasks to be synchronized with a total amount of data less than the preset segment size can be packaged and sent to the control node first, and then the large data volume tasks with data amount greater than the preset segment size can be segmented to obtain multiple segmented data of the large data volume tasks.
[0063] In the third implementation, if the total amount of data of the tasks to be synchronized is greater than the preset segment size, and the data amount of each task is smaller than the preset segment size, the tasks to be transmitted can be packaged into multiple data packets, and each data packet is sent to the control node of the standby cluster.
[0064] It should be noted that the problem solved by the present application is the synchronous processing of large-scale data, so the segmented data mentioned below all represent data segmentation for large-scale data tasks.
[0065] The segmented data is the data segment of the task to be synchronized. The primary cluster can package small data volume tasks and segment large data volume tasks according to the data volume of the task requested by the standby cluster, divide the tasks to be sent to the standby cluster for synchronization into multiple segmented data, and send each segmented data to the data node corresponding to the task data request message.
[0066] Step 203: The data node assembles each segmented data according to the segment identifier of each segmented data to obtain a task to be synchronized. The segment identifier is used to indicate whether the segmented data is the last segmented data of multiple segmented data.
[0067] The segmented data may include a segmentation identifier, which is used to indicate that the data segment is segmented data of a large data volume task and to indicate whether the segmented data is the last segmented data among multiple segmented data.
[0068] After receiving the segmented data, the data node can temporarily store the segmented data in memory or on disk, and determine whether the last segmented data has been received based on the segment identifier of the segmented data. If the last segmented data is received, it means that all segmented data related to the task to be synchronized have been transmitted.
[0069] After all segmented data are transmitted, the data node can assemble and deserialize these segmented data, and verify the integrity and consistency of the data. When the verification is successful, the assembled data will be used as the task to be synchronized.
[0070] Step 204: The data node determines the execution group to which the task to be synchronized belongs according to the execution group information of each task to be synchronized, and sends each task to be synchronized to the thread corresponding to the execution group, so that each thread executes the task to be synchronized in parallel.
[0071] It should be noted that there may be multiple tasks with data volumes greater than the preset segment size in the tasks requested by the standby cluster. In this case, the above steps 202 to 205 can be executed for each task respectively. When the task to be synchronized is successfully transmitted, the data node performs data synchronization again.
[0072] In this application, the execution group to which the task to be synchronized belongs can be obtained according to the target database corresponding to the task to be synchronized, and the database groups in all execution groups can be regularly checked through the background sending thread inside the data node, and the task to be synchronized that needs to be executed can be sent to the thread corresponding to the execution group. Among them, the execution group to which the task to be synchronized belongs can be the execution group to which the database related to the reading and writing of the task to be synchronized belongs.
[0073] Each execution group corresponds to a thread group, which includes multiple threads. The threads in the thread group are used to process the tasks to be synchronized in each database group of the execution group. As a possible implementation method, each thread in the thread group can be used to process the tasks to be synchronized in a database group, further realizing the independent execution of tasks in each database group.
[0074] The data node can assign the tasks to be synchronized to the thread groups of the corresponding execution groups according to the execution group information of the tasks to be synchronized, so that the thread groups between the execution groups are independent of each other, thereby preventing unnecessary blocking and delay of other tasks to be synchronized due to the slow execution of the tasks to be synchronized in a certain execution group.
[0075] In an embodiment of the present application, the data node in the standby cluster sends a task data request message to the main cluster based on the task information and execution group information sent by the main cluster, the execution group information includes: a database and a processing priority of the database, the data node receives multiple segmented data from the main cluster, the segmented data is a data segment of the task to be synchronized, the data node assembles each segmented data according to the segment identifier of each segmented data to obtain the task to be synchronized, the segment identifier is used to indicate whether the segmented data is the last segmented data of multiple segmented data, the data node determines the execution group to which the task to be synchronized belongs according to the execution group information of the task to be synchronized, and sends the task to be synchronized and the execution group information of the task to be synchronized to the thread corresponding to the execution group, and each thread executes the task to be synchronized in parallel.
[0076] By transmitting tasks with large amounts of data in segments, the pressure of single transmission can be effectively reduced, and tasks with small amounts of data can be packaged and sent at one time. By transmitting tasks with large amounts of data in segments, the data transmission process can be parallelized, thereby significantly improving the efficiency of data transmission. By sending the tasks to be synchronized to the corresponding threads for synchronization based on the execution group information, tasks to be synchronized with different processing priorities can be managed and executed, and the execution processes of the tasks to be synchronized in each execution group can be relatively independent and will not block each other. For example, if there is a group of more urgent tasks in the system, they can be assigned to a separate high-priority execution group and separated from the remaining large number of general tasks. Even if the tasks in a certain execution group execute slowly, it will not affect the execution of tasks in other execution groups, thereby effectively avoiding the mutual blocking of cross-database task execution and reducing the waiting time for task execution.
[0077] The following is a further description of assembling each segment of data according to the segment identifier of each segment of data to obtain multiple tasks to be synchronized, such as Figure 3 As shown, the above step 203 includes:
[0078] Step 301: A data node determines the last segment data among a plurality of segment data according to the segment identifiers of each segment data.
[0079] As a possible implementation, if the segmentation identifier is a preset value, the current segmentation data is the last segmentation data. Exemplarily, if an "EOF" mark is added to the segmentation data, the segmentation data can be considered to be the last segmentation data.
[0080] As another possible implementation, the segment identifier may also indicate the segment number of the task to be synchronized that the current segment data belongs to, so that the data node may assemble the segment data in sequence based on the segment identifier.
[0081] Step 302: The data node assembles and verifies each segmented data according to the offset of each segmented data to obtain the task to be synchronized.
[0082] The offset of the segmented data can indicate the order of the segmented data. As a possible implementation method, the data node can request data in batches based on the current offset. After receiving each segmented data, the segmented data is temporarily stored in the memory or on the disk, and then the offset of the next required data segment is calculated and requested. Therefore, the order of each segmented data can be determined based on the offset of each segmented data.
[0083] After the data node assembles the segmented data, it can perform deserialization and verify the integrity and consistency of the data. After the verification is successful, the task to be synchronized is obtained.
[0084] Furthermore, the data node sends each task to be synchronized to the thread corresponding to the execution group according to the execution group information of each task to be synchronized, such as Figure 4 As shown, including:
[0085] Step 401: The data node determines at least one candidate thread corresponding to the execution group according to the execution group information of the task to be synchronized.
[0086] In this application, one execution group corresponds to one thread group, and the selected thread can be a thread in the thread group corresponding to the execution group. All tasks to be synchronized in the execution group are completed by the threads in the thread group, and the data node can send the tasks to be synchronized to the thread group corresponding to the execution group according to the execution group to which the tasks to be synchronized belong.
[0087] Step 402: The data node determines the target thread with the smallest load according to the number of tasks and / or execution time information of each thread to be selected, and sends each task to be synchronized to the corresponding target thread.
[0088] The tasks in the threads to be selected are arranged in the form of queues, and each thread is used to execute tasks in one queue. The target thread can be the thread with the smallest load. When determining the target thread, in the first implementation method, the data node can determine the number of tasks in the queues of all the threads to be selected through the monitoring function, and determine the target thread for processing the tasks to be synchronized based on the current number of tasks. The data node can schedule the tasks to be synchronized based on the number of tasks in the queue, determine the thread with the least number of currently unexecuted tasks or the least total number of tasks as the target thread, and give priority to allocating the tasks to be synchronized to the target thread with the least number of currently unexecuted tasks or the least total number of tasks, thereby balancing the task load of each thread queue and reducing the overload of a specific queue.
[0089] The execution time information may include at least one of the following: the total execution time or average execution time of the executed tasks, or the estimated execution time obtained by estimating the execution time of the remaining unexecuted tasks.
[0090] In the second implementation, the data node can determine the thread with the longest total execution time or the shortest estimated execution time as the target thread based on the total execution time of the executed tasks in the queue and the estimated execution time of the remaining unexecuted tasks, and send the task to be synchronized to the target thread.
[0091] In the third implementation method, different weights can be assigned to the execution time and the number of tasks, and a weighted calculation is performed based on the number of tasks in the task queue corresponding to the thread and the execution time to obtain the score of each queue, and the thread corresponding to the queue with the highest score is used as the target thread.
[0092] In an embodiment of the present application, by balancing the load of the queues in each thread, the waiting time of tasks in the queue is reduced, the processing speed of tasks is accelerated, and it can be ensured that all execution resources can be effectively utilized, avoiding the situation where some resources are idle and some resources are overloaded.
[0093] As a preferred implementation, for a large number of small-scale write operations, the present application can also merge tasks. Specifically, before the data node sends each task to be synchronized to the thread corresponding to the execution group according to the execution group information of each task to be synchronized, it also includes:
[0094] The data node determines multiple tasks to be synchronized that meet preset conditions and merges the tasks to be synchronized into one task to be synchronized. The preset conditions include: the target databases and target data tables of the tasks to be synchronized are the same, and the tasks to be synchronized are write tasks.
[0095] If there are multiple tasks to be synchronized that are write operations, and the database and data table corresponding to the write operations are the same, the data node can merge the multiple tasks to be synchronized into one transaction.
[0096] By merging write tasks that meet preset conditions, the total number of transactions can be reduced and the overhead of transaction submission can be reduced, thereby improving the efficiency of data writing, reducing the need for database locking and logging, reducing resource consumption, and improving the overall performance of the database system.
[0097] The following is a further description of the above data node sending a task data request message to the main cluster based on the task information and execution group information sent by the main cluster. Figure 5 As shown, the above step 201 includes:
[0098] Step 501: The control node receives a task information set, execution group information, and conflict detection results sent by the main cluster, and sends the task information and execution group information in the task information set to each data node according to the conflict detection results.
[0099] The control node of the primary cluster can push the task information set, execution group information and conflict detection results that need to be synchronized to the control node of the standby cluster. The task information set includes multiple task information that need to be synchronized. The task information can be the transaction ID of the task, or the data offset, etc. The conflict detection result can indicate which tasks can be executed in parallel, so that the control node can determine the tasks to be synchronized that can be executed in parallel based on the conflict detection result, and send the tasks to be synchronized and the execution group information of the tasks to be synchronized to each data node.
[0100] After receiving the task information set and the execution group information, the control node of the standby cluster may distribute the task information in the task information set to each data node, and send the execution group information to all data nodes.
[0101] The control node may distribute the task information in the task information set to each data node evenly, or distribute the task information to each data node according to the resource occupancy rate of the data node.
[0102] Step 502: The data node generates a task data request message according to the received task information, and sends the task data request message to the main cluster.
[0103] After receiving the task information, the data node can compare the task ID indicated by the task information with the latest task ID saved by the data node itself. In one possible implementation, the task ID can be a transaction ID, which is arranged in ascending order, and the latest completed transaction ID has a larger ID value. If the transaction ID indicated by the task information is greater than the latest transaction ID saved by the data node itself, the data node can use the latest transaction ID saved by the data node itself as the task offset and send a task data request message to the main cluster. After the main cluster receives the task data request message, it can determine the tasks to be synchronized that need to be sent to the data nodes of the standby cluster based on the task data request message.
[0104] Next, the data synchronization method of this application is described from the main cluster side. Figure 6 This is a flow chart of a data synchronization method on the main cluster side provided in this application. The execution subject of this method can be Figure 1 The data nodes of the master cluster are as follows: Figure 6 , the method comprising:
[0105] Step 601: A data node of the active cluster receives a task data request message from a standby cluster, and determines a plurality of tasks to be synchronized according to the task data request message.
[0106] The data node of the active cluster may receive the task data request message sent by the control node of the standby cluster, parse and obtain the task offset in the task data request message, and determine the tasks to be synchronized that need to be synchronized from the active cluster to the standby cluster.
[0107] Step 602: The data node segments the multiple tasks to be synchronized based on a preset segment size and adds segment identifiers to obtain multiple segment data.
[0108] The segment identifier can be used to indicate whether the segment data is the last segment data of the task to be synchronized.
[0109] In the present application, the data size of the task to be synchronized can be determined based on the preset segment size, so as to segment the task to be synchronized with a larger data size. If the data size of the task to be synchronized is larger than the preset segment size, the task to be synchronized can be segmented to obtain multiple segmented data of the task to be synchronized; if the data size of the task to be synchronized is smaller than the preset segment size, the task to be synchronized can be not segmented.
[0110] As a possible implementation manner, if the total amount of data of the tasks to be synchronized is smaller than the preset segment size, all the tasks to be synchronized may be packaged and sent to the standby cluster at one time.
[0111] It should be noted that if there are multiple tasks to be synchronized that need to be transmitted in segments, the data nodes of the main cluster can segment and transmit each task to be synchronized in turn, or segment and process each task to be synchronized and add different task identifiers, and then transmit them in parallel to the data nodes of the backup cluster. The specific transmission method is not limited in this application.
[0112] For example, assuming that the preset segment size is 120MB, if the total amount of data of all tasks to be synchronized does not exceed 120MB, all tasks to be synchronized can be packaged at one time and sent through a single RPC request. If the amount of data of a single task to be synchronized is greater than 120MB, the remaining tasks with small data amounts can be packaged and sent to the standby cluster, and then the tasks to be synchronized with a data amount greater than 120MB can be processed and transmitted in segments.
[0113] Step 603: The data node sends the multiple segmented data to the standby cluster.
[0114] As a possible implementation method, the data node of the active cluster can segment the synchronization task based on the segment size, add a segment identifier to each segment, obtain multiple segment data, and then transmit each segment data to the data node of the standby cluster in sequence.
[0115] As another possible implementation, when starting segmented transmission, the data node of the standby cluster can send a request offset to the data node of the primary cluster. The data node of the primary cluster determines the data segment to be sent based on the request offset, adds a segment identifier to the data segment, obtains the segmented data, and sends the segmented data to the data node of the standby cluster. This transmission method can flexibly adjust the amount of data to be transmitted based on the memory space of the standby cluster, thereby improving the flexibility of data transmission.
[0116] After receiving the segmented data, the data node of the standby cluster may temporarily store the segmented data in memory or on a disk, and after receiving all the segmented data, assemble and deserialize the segmented data to complete the segmented transmission of the task to be synchronized.
[0117] In the embodiment of the present application, the main cluster determines the synchronization tasks based on the preset segment size, packages and sends tasks with small data volumes, and transmits tasks with large data volumes in segments. This can not only efficiently utilize network resources, but also reduce transmission time, ensure the reliability and stability of data transmission, and improve the efficiency of data transmission.
[0118] As a preferred implementation solution, the main cluster of this application can also perform conflict detection on tasks to determine tasks that can be executed in parallel. Specifically, Figure 7 As shown, the conflict detection process includes:
[0119] Step 701: The control node performs conflict detection on multiple tasks to be sent, and determines the conflict detection results of each task to be sent.
[0120] Before sending the task information set and the execution group information to the control node of the standby cluster, the control node of the master cluster may perform conflict detection on the tasks to be sent. The conflict detection result may indicate whether there is a conflict between multiple tasks to be sent.
[0121] Among them, when the data partitions involved in the tasks to be sent are the same, it can be considered that there is a conflict in the tasks to be sent. When the data partitions involved in the tasks to be sent are different, it can be considered that there is no conflict in the tasks to be sent. The tasks to be sent without conflict can be executed in parallel.
[0122] Step 702: The control node determines multiple executable tasks to be synchronized according to the conflict detection result, and sends task information, execution group information and conflict detection results of the multiple tasks to be synchronized to the standby cluster.
[0123] The control node of the main cluster can determine whether there is a conflict between the tasks to be sent based on the conflict detection results, and determine the tasks to be sent without conflict as tasks to be synchronized, and package the task information of multiple tasks to be synchronized into a task information set and send it to the control node of the standby cluster, so that after receiving the task information set, the standby cluster can execute the above steps 501-502 to send a task data request message to the data node of the main cluster.
[0124] As an example, the data partitions involved in the task 1 to be sent are partition A and partition B, the task 2 to be sent operates partition C and partition D, and the task 3 to be sent operates partition B and partition D. After the control node of the main cluster performs conflict detection, it is found that there are no conflicting partitions between the task 1 to be sent and the task 2 to be sent. However, the task 3 to be sent and the task 1 to be sent both involve partition B, and there is a conflict. The control node of the main cluster records the conflict detection results of the tasks 1 to be sent, the tasks 2 to be sent, and the tasks 3 to be sent, and takes the task information of the tasks 1 to be sent, the tasks 2 to be sent, and the tasks 3 to be sent as a task information set, and sends the conflict detection results, the task information set, and the execution group information to the control node of the standby cluster. After that, the control node of the standby cluster can send a batch of tasks and the execution group information that do not conflict to the data nodes of the standby cluster each time according to the conflict detection results, and after the data nodes of the standby cluster have completed the tasks, send the next batch of non-conflicting task information to the data nodes. For example, the control node of the standby cluster can determine that the task 3 to be sent needs to be sent to the data nodes of the standby cluster after the tasks 1 to be sent and the tasks 2 to be sent are completed according to the conflict detection results.
[0125] In an embodiment of the present application, the control node of the main cluster performs conflict detection on the tasks to be sent and sends the conflict detection results to the control node of the standby cluster, so that the control node of the standby cluster can distribute the tasks to be synchronized that can be executed in parallel to each data node when distributing tasks, thereby ensuring that the tasks received by each data node must not conflict with each other. The parallelism of task execution is improved and the execution efficiency is improved without the data nodes of the standby cluster being aware of the conflict detection mechanism. In addition, the present application ensures that conflict-free tasks can be executed immediately, avoiding unnecessary waiting and reducing the total task processing time.
[0126] Next, combine Figure 8 , the overall process of the data synchronization method of this application is explained.
[0127] Reference Figure 8The control node of the main cluster first performs conflict detection on the tasks to be sent and obtains the conflict detection results. Then, the task information, conflict detection results and execution group information are sent to the control node of the standby cluster. The control node of the standby cluster sends the execution group information to each data node, and distributes the task information of a batch of tasks that do not conflict to each data node according to the conflict detection results. The data node of the standby cluster sends a task data request message to the data node of the main cluster. The data node of the main cluster determines the tasks to be synchronized according to the task data request message, and transmits the tasks to be synchronized with a large amount of data in segments based on the preset segment size. After receiving the segmented data, the data node of the standby cluster assembles and verifies the segmented data based on the offset and segment identifier of the segmented data, and sends the obtained tasks to be synchronized to the corresponding threads, and each thread executes the tasks to be synchronized in parallel.
[0128] Based on the same inventive concept, the present application also provides a database system corresponding to the data synchronization method. The structure of the database system can refer to Figure 1 The system architecture shown, since the principle of the system in the embodiment of the present application to solve the problem is similar to the above-mentioned data synchronization method in the embodiment of the present application, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be repeated.
[0129] Reference Figure 1 The database system architecture diagram shown in FIG. The database system includes a primary cluster and at least one standby cluster. The primary cluster includes a control node, and the standby cluster includes a control node and at least one data node. The data node in the standby cluster is used to:
[0130] Based on the task information and execution group information sent by the main cluster, a task data request message is sent to the main cluster. The execution group information includes: a database and a processing priority of the database. The task information is used to indicate multiple tasks to be synchronized. The execution group information is the execution group information of the tasks to be synchronized.
[0131] Receive multiple segmented data from the main cluster, where the segmented data is a data segment of the task to be synchronized;
[0132] Assemble each segmented data according to the segmentation identifier of each segmented data to obtain a task to be synchronized, wherein the segmentation identifier is used to indicate whether the segmented data is the last segmented data of multiple segmented data;
[0133] The execution group to which the task to be synchronized belongs is determined according to the execution group information of the task to be synchronized, and the task to be synchronized and the execution group information of the task to be synchronized are sent to the thread corresponding to the execution group, and each thread executes the task to be synchronized in parallel.
[0134] The data nodes in the standby cluster are also used for:
[0135] Determine the last segment data among the plurality of segment data according to the segment identifier of each segment data;
[0136] The segmented data are assembled and verified according to the offset of each segmented data to obtain the task to be synchronized.
[0137] The data nodes in the standby cluster are also used for:
[0138] Determine at least one thread to be selected corresponding to the execution group according to the execution group information of the task to be synchronized;
[0139] The target thread with the smallest load is determined according to the number of tasks and / or execution time information of each thread to be selected, and each task to be synchronized is sent to the corresponding target thread.
[0140] The data nodes in the standby cluster are also used for:
[0141] The target thread is determined according to the current number of tasks of each thread to be selected, and the tasks to be synchronized are sent to the target thread.
[0142] The data nodes in the standby cluster are also used for:
[0143] A plurality of tasks to be synchronized that meet preset conditions are determined, and the tasks to be synchronized are merged into one task to be synchronized. The preset conditions include: the target databases and target data tables of the tasks to be synchronized are the same, and the tasks to be synchronized are write tasks.
[0144] The control node of the standby cluster is used for:
[0145] Receive the task information set, execution group information and conflict detection results sent by the main cluster, and send the task information and execution group information in the task information set to each data node according to the conflict detection results;
[0146] The data nodes in the standby cluster are used to:
[0147] A task data request message is generated according to the received task information, and the task data request message is sent to the main cluster.
[0148] The data nodes in the main cluster are used to:
[0149] Receive a task data request message from the standby cluster, and determine multiple tasks to be synchronized according to the task data request message;
[0150] Based on the preset segment size, multiple tasks to be synchronized are segmented and segment identifiers are added to obtain multiple segmented data;
[0151] Send multiple segmented data to the standby cluster.
[0152] The data nodes in the main cluster are used to:
[0153] Determine multiple tasks to be sent according to the task data request message;
[0154] Perform conflict detection on multiple tasks to be sent, and determine the conflict detection results of each task to be sent;
[0155] A plurality of tasks to be synchronized are determined according to the conflict detection result.
[0156] The control node in the main cluster is also used to:
[0157] Perform conflict detection on multiple tasks to be sent, and determine the conflict detection results of each task to be sent;
[0158] According to the conflict detection result, multiple executable tasks to be synchronized are determined, and the task information set, execution group information and conflict detection result of the tasks to be synchronized are sent to the control node of the standby cluster.
[0159] Fig. 9 A structural schematic diagram of an electronic device provided in an embodiment of the present application is shown, including: a processor 901, a storage medium 902 and a bus 903, wherein the storage medium 902 stores machine-readable instructions executable by the processor 901, and when the electronic device runs the data synchronization method as in the embodiment, the processor 901 communicates with the storage medium 902 via the bus 903, and the processor 901 executes the machine-readable instructions, and the processor 901 executes the preamble of the method item to execute the steps in the above-mentioned data synchronization method.
[0160] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed when a processor is running, and the processor executes the steps in the above-mentioned data synchronization method.
[0161] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.
[0162] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0164] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0165] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), disk or optical disk, and other media that can store program codes.
[0166] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.
[0167] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A data synchronization method, characterized in that: A standby cluster is applied to a database system, wherein the standby cluster includes: a data node, and the method includes: The data node sends a task data request message to the main cluster based on the task information and execution group information sent by the main cluster, the execution group information includes: a database and a processing priority of the database, the task information is used to indicate a plurality of tasks to be synchronized, and the execution group information is the execution group information of the tasks to be synchronized; The data node receives a plurality of segmented data from the main cluster, where the segmented data is a data segment of a task to be synchronized; The data node assembles each of the segmented data according to the segment identifier of each of the segmented data to obtain a task to be synchronized, wherein the segment identifier is used to indicate whether the segmented data is the last segmented data of the multiple segmented data; The data node determines the execution group to which the task to be synchronized belongs according to the execution group information of the task to be synchronized, and sends the task to be synchronized and the execution group information of the task to be synchronized to the thread corresponding to the execution group, so that each of the threads executes the task to be synchronized in parallel.
2. The method according to claim 1, characterized in that The data node assembles each of the segmented data according to the segment identifier of each of the segmented data to obtain a task to be synchronized, including: The data node determines the last segment data among the plurality of segment data according to the segment identifier of each segment data; The data node assembles and verifies each segmented data according to the offset of each segmented data to obtain a task to be synchronized.
3. The method according to claim 1, characterized in that The data node sends each of the tasks to be synchronized to a thread corresponding to the execution group according to the execution group information of each of the tasks to be synchronized, including: The data node determines at least one candidate thread corresponding to the execution group according to the execution group information of the task to be synchronized; The data node determines the target thread with the smallest load according to the number of tasks and / or execution time information of each of the threads to be selected, and sends each of the tasks to be synchronized to the corresponding target thread.
4. The method according to claim 1, characterized in that Before the data node sends each of the tasks to be synchronized to the thread corresponding to the execution group according to the execution group information of each of the tasks to be synchronized, the method further includes: The data node determines multiple tasks to be synchronized that meet preset conditions and merges the tasks to be synchronized into one task to be synchronized. The preset conditions include: the target databases and target data tables of the tasks to be synchronized are the same, and the tasks to be synchronized are write tasks.
5. The method according to claim 1, characterized in that The standby cluster further includes: a control node; The data node sends a task data request message to the main cluster based on the task information and execution group information sent by the main cluster, including: The control node receives the task information set, execution group information and conflict detection result sent by the main cluster, and sends the task information and execution group information in the task information set to each of the data nodes respectively according to the conflict detection result; The data node generates a task data request message according to the received task information, and sends the task data request message to the main cluster.
6. A data synchronization method, characterized in that: A main cluster applied to a time series database, the main cluster comprising: a data node, the method comprising: The data node of the main cluster receives the task data request message from the standby cluster, and determines a plurality of tasks to be synchronized according to the task data request message; The data node performs segment processing on the multiple tasks to be synchronized based on a preset segment size and adds segment identifiers to obtain multiple segment data; The data node sends the multiple segmented data to the standby cluster.
7. The method according to claim 6, characterized in that The main cluster includes: a control node, and the method further includes: The control node performs conflict detection on multiple tasks to be sent, and determines the conflict detection results of each task to be sent; The control node determines the multiple tasks to be synchronized according to the conflict detection result, and sends the task information set, execution group information and the conflict detection result of the multiple tasks to be synchronized to the control node of the standby cluster.
8. A database system, characterized in that: The database system comprises: a primary cluster as claimed in any one of claims 1 to 7 and a plurality of standby clusters, wherein the primary cluster and the standby clusters are used to execute the steps of the data synchronization method as claimed in any one of claims 1 to 7.
9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the data synchronization method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data synchronization method according to any one of claims 1 to 7 are executed.
Citation Information
Cited By
Data synchronization method and device, equipment and storage medium
CN120994472A
