Data transmission method and system for distributed database

Abstract

The application discloses a kind of data sending method and system for distributed database, belong to distributed database technical field, the method is: obtaining the performance evaluation value of current node and the performance evaluation value of other nodes, obtains first sending node in combination with election strategy;If current node is first sending node, then the first data of local is sent to target system in batches, if it is successfully sent, then the current cut-off point of current batch data is generated, and the current cut-off point is sent to all nodes until data sending is completed;If current node is not first sending node, and first sending node satisfies election trigger condition, then second sending node is obtained by re-election, if current node is second sending node, continue to send data according to the second data of local and the current cut-off point received, until the sending of second data is completed, therefore, by implementing the application, the reliability and efficiency of data sending can be realized.

Description

Technical Field

[0001] This invention relates to the field of distributed database technology, and in particular to a data transmission method and system for distributed databases. Background Technology

[0002] With the widespread application of distributed database systems, enterprises need to continuously synchronize real-time changes in database data to downstream target systems such as data warehouses and message queues to support data analysis, monitoring, and alerting. Because database write loads are extremely high and the migration process cannot be interrupted, if the current sending node fails, it should automatically switch to another node and resume transmission from the point of interruption to ensure that data is not lost or duplicated.

[0003] Currently, a typical database migration breakpoint resume method (such as CN119597727A) collects data from the source database and caches the collected data and corresponding collection points locally. When an interruption occurs, the latest collection point is retrieved from the local cache, and data collection continues from this point as the starting point. The unique identifier of the data is used to filter duplicate data until the collection is completed, and then all data is sent to the target database.

[0004] However, the collected sites are only stored in the local cache of a single migration process. Once a permanent local failure occurs, the site information will be completely lost, and the collection can only be started from scratch or rely on manual intervention, resulting in low continuity and accuracy of data transmission and low migration efficiency. Summary of the Invention

[0005] This invention provides a data transmission method and system for distributed databases, which can solve the problem of how to avoid migration interruption or data loss caused by single point of failure.

[0006] This invention provides a data sending method for distributed databases, applied to replica nodes in a distributed database, the method comprising: Obtain performance evaluation values ​​and receive performance evaluation values ​​from other replica nodes; elect the first sending node based on all the performance evaluation values ​​and a preset election strategy. If the replica node is the first sending node, the first data to be migrated stored locally is sent to the target system in batches. During the sending of each batch of data, if the sending is successful, the current cutoff point corresponding to the current batch of data is generated and the current cutoff point is sent to all the replica nodes until the sending of the first data to be migrated is completed. If the replica node is not the first sending node, and the first sending node meets the preset election triggering conditions, a second sending node is re-elected. If the current replica node is the second sending node, the next batch of data is sent according to the second data to be migrated stored locally and the received current cutoff point, until the sending of the second data to be migrated is completed. The first data to be migrated and the second data to be migrated are the same.

[0007] This invention establishes a highly available distributed data migration system through performance evaluation-driven dynamic election, batch sending and truncation point tracking, and a seamless task switching mechanism in case of node failure. It enables timely cleanup of local data and synchronous broadcasting of global progress, optimizes storage utilization and state consistency, and ensures the continuity of sending tasks when nodes are abnormal. It also achieves seamless takeover of sending tasks, significantly improving the reliability, efficiency and fault tolerance of distributed database data sending.

[0008] Furthermore, the election triggering condition includes the existence of a network partition in the first sending node, and the data sending method for the distributed database further includes: If the current replica node is the first sending node and loses connection with other replica nodes due to the network partition, the first sending node continues to send batch data and records the cutoff point during the period of disconnection, and meets the election triggering condition to re-elect a second sending node; Until the first sending node re-establishes a connection with the other replica nodes, the latest local truncation point and local data to be migrated in the first sending node are determined, the first sending node is set to read-only synchronization state, and the latest global truncation point is obtained by querying the second sending node. Based on the global cutoff point and the local cutoff point, the state is aligned with the second sending node, and the first sending node after state alignment is converted into a non-sending node.

[0009] This complete process, including network partition detection, independent transmission during partitioning, state alignment after reconnection, and node role transformation, establishes a data transmission fault tolerance mechanism in network partitioning scenarios. It ensures the continuity of data migration during partitioning and the consistency of state after partition recovery, significantly improving the data migration reliability and self-healing capability of distributed databases in unstable network environments.

[0010] Furthermore, the step of aligning the state with the second sending node based on the global cutoff point and the local cutoff point specifically involves: If the local cutoff point lags behind the global cutoff point, then the batch data between the local cutoff point and the global cutoff point in the local data to be migrated is deleted, and the local cutoff point is updated to the global cutoff point to achieve alignment with the state of the second sending node. If the local cutoff point precedes the global cutoff point, the local cutoff point and the batch data between the local cutoff point and the global cutoff point are sent to the second sending node, and the verification result of the second sending node is received. If the verification result is successful, the global cutoff point is confirmed to be updated to the local cutoff point to achieve state alignment with the second sending node. The verification result is used to characterize the result of the second sending node verifying whether the batch data between the local cutoff point and the global cutoff point has been successfully sent to the target system.

[0011] This two-way state alignment mechanism, which combines lag-to-catch and advance-to-verify, establishes a precise state synchronization system. It enables intelligent coordination of the sending progress between nodes after partition recovery and reliable verification of data consistency, significantly improving the state consistency and data reliability assurance capabilities of distributed databases in complex network environments.

[0012] Furthermore, the data transmission method for distributed databases also includes: If the current replica node is the first sending node, then add the first election identifier to the batch data being sent; If the current replica node is the second sending node, then a second election identifier is added to the batch data being sent; The first election identifier and the second election identifier are monotonically increasing numerical identifiers determined according to the election order. They are used to retain the batch data corresponding to the larger election identifier and delete the batch data corresponding to the smaller election identifier when there is duplicate data in the batch data received by the target system.

[0013] By embedding election identifiers, generating monotonically increasing identifiers, and establishing an identifier-based duplicate data adjudication mechanism, a data deduplication and consistency assurance system is established. This enables automatic identification and effective processing of duplicate data in multi-node transmission scenarios, significantly improving the eventual consistency of distributed data migration and the accuracy of data in the target system.

[0014] Furthermore, the data transmission method for distributed databases also includes: When a conflict is detected between the batch data sent by the first sending node during the period of disconnection and the batch data sent by the second sending node, a forced backtracking mechanism is triggered. A safe cutoff point is determined based on the local cutoff point and the global cutoff point; In response to the forced backtracking mechanism, the local cutoff point is forcibly reset to the safe cutoff point, and other replica nodes are notified to reset synchronously.

[0015] This complete process, including conflict detection, safe cutoff point determination, forced backtracking, and global synchronization, establishes an emergency handling mechanism for data conflicts. It enables safe rollback of transmission progress and rapid recovery of cluster status, significantly improving the data consistency guarantee capability and system resilience of the distributed database under extreme abnormal scenarios.

[0016] Furthermore, the election of the first sending node based on all the performance evaluation values ​​and the preset election strategy specifically involves: The replica node corresponding to the maximum value of the performance evaluation value is determined as the first sending node. If multiple replica nodes have the same performance evaluation value, and all of them are the maximum value of the performance evaluation value, then among the multiple replica nodes with the same performance evaluation value, the replica node corresponding to the minimum node identifier value is selected as the first sending node. The performance evaluation value of the current replica node is obtained by normalizing the performance indicators of the current replica node, determining the initial score corresponding to each performance indicator, and then weighting and summing all the initial scores according to a preset weight ratio.

[0017] This complete process, involving optimal performance selection, tie-breaking determination, multi-indicator normalization, and weighted comprehensive evaluation, establishes a scientific and fair node election mechanism. It achieves standardization, fairness, and configurability in the selection of sending nodes, significantly improving the rationality of the selection of sending nodes in the distributed database and the stability of system operation.

[0018] Furthermore, the data transmission method for distributed databases also includes: If the current replica node is the first sending node, monitor the remaining processing capacity of the first sending node in real time; When the remaining processing capacity is less than a preset safety threshold, the amount of data that can be processed is calculated based on the current data flow rate and the remaining processing capacity. Based on the amount of data that can be processed and the current cutoff point, a predicted cutoff point is determined, and the predicted cutoff point and a prompt message for switching sending nodes are sent to the other replica nodes. Continue sending data until the preset cutoff point is reached, at which point the election trigger condition is met, and actively stop sending data.

[0019] This complete process, including real-time load monitoring, processing capacity prediction, switch point pre-calculation, early warning broadcasting, and orderly termination, establishes an active switching mechanism for sending nodes. This enables smooth migration of sending tasks and early warning before node performance degrades, significantly improving the smoothness, predictability, and adaptability to changes in node load in distributed database data transmission.

[0020] Furthermore, the data transmission method for distributed databases also includes: If the current replica node is not the sending node, then the local data queue of the current replica node is logically divided into an active buffer and a cleanup buffer. The active buffer is used to store real-time written data after the previous cutoff point, and the cleanup buffer is used to store local storage data that has been sent but has not received global confirmation information. The difference in data volume between the active buffer and the buffer to be cleaned is monitored in real time. When the difference in data volume is greater than a preset storage security threshold, a flow control signal is sent to the data writing node to reduce the writing rate of the corresponding data partition until the difference in data volume is less than the storage security threshold. When the current cutoff point of the current sending node is received, the memory block before the current cutoff point is marked as a reclaimable segment. During the idle window period when the input / output load is less than the preset load threshold, the sequence number and process reference of the reclaimable segment are checked to see if they meet the preset cleanup conditions. If so, the memory block corresponding to the reclaimable segment is physically released, and the released storage space is added to the free block list for use in overwriting subsequent real-time data.

[0021] This complete process, including dual buffer partitioning, buffer pressure monitoring, reverse flow control, storage marking based on cutoff points, idle period cleanup, and resource recycling, establishes a refined storage management mechanism for non-sending nodes. This enables efficient utilization of storage resources, dynamic adjustment of write traffic, and timely reclamation of storage space, significantly improving the storage efficiency, resource utilization, and system stability of the distributed database on non-sending nodes.

[0022] Another embodiment of the present invention also provides a replica node, including: an election module, a sending module, and a resume module; The election module is used to obtain performance evaluation values ​​and receive performance evaluation values ​​from other replica nodes, and elect a first sending node based on all the performance evaluation values ​​and a preset election strategy. The sending module is used to send the first batch of data to be migrated stored locally to the target system in batches if the replica node is the first sending node. During the sending process of each batch of data, if the sending is successful, the current cutoff point corresponding to the current batch of data is generated and the current cutoff point is sent to all the replica nodes until the sending of the first batch of data to be migrated is completed. The resume module is used to re-elect a second sending node if the replica node is not the first sending node and the first sending node meets the preset election triggering conditions. If the current replica node is the second sending node, it continues to send the next batch of data according to the second data to be migrated stored locally and the received current cutoff point, until the sending of the second data to be migrated is completed, wherein the first data to be migrated and the second data to be migrated are the same.

[0023] This invention constructs a highly autonomous replica node architecture through the cohesive integration of an election module, a sending module, and a resume module. The election module enables real-time evaluation of node performance and dynamic election of the globally optimal node, ensuring that the cluster always selects the node with the best performance to undertake sending tasks. When elected as the first sending node, the sending module achieves precise tracking of sending progress and consistent maintenance of cluster state through batch sending and a global synchronization mechanism based on a cutoff point. The resume module automatically triggers a re-election when the first sending node fails and supports seamless takeover of subsequent sending tasks based on a global cutoff point when the same node is elected as the second sending node. This cohesive architecture, through the close collaboration of functional modules, significantly improves the autonomous decision-making capability, fault self-healing capability, and task continuity assurance capability of a single replica node, reduces inter-module communication overhead, and achieves efficient and reliable distributed data transmission.

[0024] Another embodiment of the present invention provides another type of replica node, characterized in that it includes: a target system and multiple replica nodes; Each of the replica nodes executes the data sending method for distributed databases as described in this invention; The target system is used to receive batch data sent by the replica node. When there is duplicate data in the received batch data, it compares the size relationship of the election identifiers corresponding to the duplicate data, retains the batch data corresponding to the larger election identifier, and deletes the batch data corresponding to the smaller election identifier.

[0025] This collaborative mechanism, combining autonomous execution by replica nodes with intelligent deduplication in the target system, constructs an end-to-end data reliability assurance system. Replica nodes, through autonomous execution of data transmission methods, achieve dynamic election of sending nodes, batch transmission, truncation point synchronization, and failover, ensuring high availability and continuity of data migration. The target system, by receiving and parsing election identifiers in batch data, establishes a deduplication mechanism based on time sequence numbers, intelligently identifying and eliminating data duplication caused by network partitions or node switching, retaining the most recently sent batch data. This architecture, through the collaboration of autonomous sending and intelligent receiving, significantly improves the eventual consistency and reliability of distributed database data transmission, ensuring the accuracy and integrity of data in the target system, while reducing the coordination complexity at the sending end, achieving efficient and reliable distributed data migration. Attached Figure Description

[0026] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating a data transmission method for a distributed database provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a replica node provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a data sending system for a distributed database provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0030] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0031] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0032] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0033] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0034] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0035] See Figure 1 To address the problem of avoiding migration interruptions or data loss due to single points of failure in existing technologies, an embodiment of the present invention provides a data transmission method for distributed databases, comprising: Step 101: Obtain the performance evaluation value and receive the performance evaluation values ​​of other replica nodes. Elect the first sending node based on all the performance evaluation values ​​and the preset election strategy.

[0036] In the above steps, each replica node (e.g., N1, N2, N3) within the replica group independently and periodically (e.g., every 5 seconds) collects a series of performance metrics reflecting its ability to perform migration tasks. After collecting the raw metrics, these values ​​with different dimensions need to be normalized and mapped to a standardized scoring range (e.g., 0-100 points) for subsequent weighted calculations. Each node calculates a quantified, dynamically changing performance evaluation value that comprehensively reflects its overall capabilities based on the collected and normalized performance metrics using a preset weighted summation formula. This evaluation value represents the suitability of the replica node as a sending node. When data transmission is initiated, an election process is triggered. All healthy nodes (N1, N2, N3) within the replica group calculate their own performance evaluation value at the current moment. Each node broadcasts its performance evaluation value to all other replica nodes in the group through an internal inter-replica communication protocol. After receiving the performance evaluation values ​​from all other nodes in the group, each node determines the election result according to a unified election strategy. After the election results are generated, the winning node (e.g., N1) will broadcast a confirmation message to the other replica nodes in the replica group as the first sending node. After receiving this message, the other replica nodes (N2, N3) will enter the standby node state, stop any election-related activities, and prepare to receive the migration cutoff point update from the sending node.

[0037] Step 102: If the replica node is the first sending node, the first data to be migrated stored locally is sent to the target system in batches. During the sending of each batch of data, if the sending is successful, the current cutoff point corresponding to the current batch of data is generated and the current cutoff point is sent to all the replica nodes until the sending of the first data to be migrated is completed.

[0038] In the above steps, the same real-time data migration logic runs independently on each data replica node of the distributed database. This means that each replica node generates the data to be migrated locally. Before officially starting data extraction, the range of all data to be migrated is first determined based on the Elastic Index. Specifically, the Elastic Index is a dynamic and scalable index structure that can scan and identify the boundaries of all data to be migrated within a partition in real time (such as a range based on timestamps or transaction IDs), thereby ensuring that the migration covers a complete and unrestricted dataset. After the range is determined, data transmission begins. Only the elected first sending node can extract the locally generated data to be migrated and send it to the downstream target system via the network. Other replica nodes that are not sending nodes, although they also generate migration data locally, will temporarily cache it and not send it. During migration, data extraction is based on the sequence order of transactions. For example, in the ShuangzhaoDB distributed database, because transactions use a serial execution model, the order between transactions is consistent, thus ensuring consistency of subsequent cutoff points and overall data consistency. After successfully retrieving and sending each batch of data, the first sending node calculates a migration truncation point. This truncation point uniquely identifies the latest position of the successfully sent data (for example, it could be the ID or sequence number of the last transaction). The first sending node broadcasts this latest truncation point to all other replica nodes in the group via the database's internal inter-replica communication protocol. Upon receiving this information, each replica node updates its local migration progress record and can safely clear any cached migration data prior to that truncation point. This process is repeated to continue sending the next batch of data until all data to be migrated has been sent.

[0039] For example, suppose a distributed database cluster has three replicas of a data partition P, located on nodes N1, N2, and N3. These three nodes form a replica group, jointly providing services and fault tolerance for partition P. In an e-commerce order data migration scenario, after the cluster starts, nodes N1, N2, and N3 communicate through an internal leader election protocol. Assume the election result is that N1 becomes the sending node for the current partition P, while N2 and N3 become standby nodes. All three nodes (N1, N2, N3) process business writes in real time and generate data to be migrated locally (e.g., writing to a local "export table"). Only N1 actively reads data from its local migration queue, packages it into batches, and sends it to the downstream data warehouse. Although N2 and N3 also generate the same data, they retain it in their local queues and do not send it. Assuming N1 successfully sends a batch of data containing transactions with IDs from T100 to T120, N1 generates a migration cutoff point with the content "T120". N1 broadcasts the cutoff point information "P_migration_checkpoint = T120" to N2 and N3 via its internal log replication channel. Upon receiving this information, N2 and N3 update the migration progress of partition P in their own memory and can safely delete data up to T120 in their local queues to free up space.

[0040] Step 103: If the replica node is not the first sending node, and the first sending node meets the preset election triggering conditions, then a second sending node is re-elected. If the current replica node is the second sending node, the next batch of data is sent according to the second data to be migrated stored locally and the received current cutoff point, until the sending of the second data to be migrated is completed. The first data to be migrated and the second data to be migrated are the same.

[0041] In the above steps, when the current replica node is not the first sending node (i.e., in standby state), it continuously monitors the health status of the first sending node through a leader election algorithm. If the first sending node is detected to meet the preset election triggering conditions (e.g., failure, network partition, or proactive optimization switch), all healthy replica nodes automatically trigger a new round of election. The re-election process is the same as the election method in step 101: each replica node obtains its own and other nodes' performance evaluation values, and selects a new sending node based on a comprehensive score and a preset strategy, denoted as the second sending node. If the current replica node is elected as the second sending node, it reads the latest received current cutoff point from its local storage (this cutoff point is the last successfully sent and broadcast by the first sending node before the failure). Based on the cutoff point, the node locates the next batch of data after the cutoff point in the second batch of data to be migrated in its local storage. It should be noted that the content of the first batch of data to be migrated and the second batch of data to be migrated is the same (because the data of multiple replicas is consistent), therefore the second sending node has the exact same sequence of data to be migrated as the first sending node. Starting from this position, subsequent batches of data are sent to the target system in the original order. Each time a batch is successfully sent, a new cutoff point is generated and broadcast to all replica nodes to ensure that other nodes can continue transmitting in case of subsequent failures. This process continues until all data to be migrated has been sent.

[0042] It's important to note that during the migration process, regardless of the number of elections (i.e., multiple nodes failing or being replaced sequentially), each newly elected sending node follows the same resumption rule: it starts sending from the latest truncation point stored locally and continues broadcasting the new truncation point after successful transmission. Since each truncation point is broadcast and persistently stored locally on all healthy replica nodes, after any number of failovers, the new node can always obtain the accurate resumption position locally, thus ensuring that the migration task can be resumed indefinitely until data transmission is complete.

[0043] For example, if node N1 suddenly goes offline due to a hardware failure, N2 and N3 quickly detect N1's offline status through a heartbeat mechanism or consensus protocol. N2 and N3 immediately trigger a new round of leader election, assuming N2 is elected as the new sending node, i.e., the second sending node. After becoming the second sending node, N2 first checks its local migration cutoff point; for example, it knows that the last successfully sent transaction was T120. Therefore, N2 reads data from its local migration queue starting from transaction T121, packages it, and sends it. The entire switchover process is automatic; for the downstream target system, it's simply a brief pause in the data flow followed by its resumption, and the data is continuous and uninterrupted.

[0044] As an example of an embodiment of the present invention, the election triggering condition includes the existence of a network partition in the first sending node, and the data sending method for the distributed database further includes: If the current replica node is the first sending node and loses connection with other replica nodes due to the network partition, the first sending node continues to send batch data and records the cutoff point during the period of disconnection, and meets the election triggering condition to re-elect a second sending node; Until the first sending node re-establishes a connection with the other replica nodes, the latest local truncation point and local data to be migrated in the first sending node are determined, the first sending node is set to read-only synchronization state, and the latest global truncation point is obtained by querying the second sending node. Based on the global cutoff point and the local cutoff point, the state is aligned with the second sending node, and the first sending node after state alignment is converted into a non-sending node.

[0045] In this embodiment, in a distributed database, when a network partition occurs, the first sending node loses contact with other replica nodes due to the partition. However, this node may not be aware that it has been isolated. Therefore, the first sending node may continue to perform data sending tasks in the isolated state, sending local data to be migrated to the target system in batches according to the original order, and recording the local cutoff point corresponding to each successful batch. At the same time, other healthy replica nodes in the cluster, unable to receive the heartbeat or communication response from the first sending node, determine that the first sending node is unavailable, triggering the election trigger condition, and re-electing a second sending node to continue sending data. To ensure the consistency of the state after network recovery, the bidirectional verification and cutoff point backtracking control method of this embodiment is introduced. When the network recovers and the original first sending node re-establishes connections with other replica nodes, the node executes the following state alignment process: the node actively stops all data sending activities, only allowing the reception of query and state synchronization requests, that is, entering a read-only synchronization state to avoid sending conflicts with the current second sending node. A state query request is sent to the second sending node that is actually responsible for sending in the current cluster to obtain the latest global cutoff point maintained by the second sending node (i.e., the cluster-recognized sending progress). The node reads its local cutoff point recorded during the period of disconnection, compares it with the global cutoff point to determine whether the local cutoff point is lagging or ahead, and performs state alignment based on the comparison result to ensure that the node's data progress is consistent with the global cutoff point. At this point, the node exits the read-only synchronization state, voluntarily relinquishes its sending node status, and becomes a non-sending node (i.e., a standby node). It no longer participates in data transmission, only receiving subsequent cutoff point updates and clearing its local cache. If another election occurs later, this node can still be elected as the new sending node based on its latest locally stored cutoff point.

[0046] As an example of an embodiment of the present invention, the step of aligning the state with the second sending node based on the global truncation point and the local truncation point specifically includes: If the local cutoff point lags behind the global cutoff point, then the batch data between the local cutoff point and the global cutoff point in the local data to be migrated is deleted, and the local cutoff point is updated to the global cutoff point to achieve alignment with the state of the second sending node. If the local cutoff point precedes the global cutoff point, the local cutoff point and the batch data between the local cutoff point and the global cutoff point are sent to the second sending node, and the verification result of the second sending node is received. If the verification result is successful, the global cutoff point is confirmed to be updated to the local cutoff point to achieve state alignment with the second sending node. The verification result is used to characterize the result of the second sending node verifying whether the batch data between the local cutoff point and the global cutoff point has been successfully sent to the target system.

[0047] In this embodiment, if the local cutoff point of the first sending node is less than the global cutoff point (e.g., the local cutoff point is T120 and the global cutoff point is T150), it indicates that the node sent less data during the disconnection period than the progress confirmed by the cluster, or that some of its data was not successfully received downstream. At this time, the current replica node, acting as the first sending node, determines all batches of data in its local data to be migrated that are after the local cutoff point and before the global cutoff point (i.e., data between T121 and T150). These batches of data are logically deleted from the local data to be migrated cache or marked as overwriteable, because these data have been successfully sent and confirmed by the second sending node and do not need to be retained. The locally recorded cutoff point is directly updated to the global cutoff point (T150) to align the local progress with the cluster-recognized progress. Afterward, this node no longer sends any data, but only acts as a backup node to receive subsequent cutoff point updates. If this node is re-elected as a sending node in the future, it will start sending from T151 and will not repeatedly send data from T121 to T150.

[0048] If the local cutoff point of the original first sending node is greater than the global cutoff point (e.g., the local cutoff point is T180, and the global cutoff point is T150), it indicates that the node independently sent new data (i.e., T151 to T180) that the second sending node had not yet sent during the period of disconnection. At this time, the current replica node, acting as the first sending node, extracts all batches of data (T151 to T180) between the local cutoff point and the global cutoff point from its local data to be migrated, along with the corresponding local cutoff point value (T180). It then sends the extracted data and the local cutoff points to the second sending node for verification. Upon receiving this data, the second sending node checks the legality of these transactions, including whether the transaction IDs are consecutive and connect with the data after the global cutoff point; whether the data content is consistent with the local replica (which should normally be consistent since multiple replicas share the same data); and whether there are any conflicts with the data it has already sent. If verification passes, the second sending node updates the global cutoff point from T150 to T180 and notifies the original node that verification has passed. If verification fails (e.g., data gaps or inconsistencies), a subsequent forced backtracking mechanism is triggered (handled as an exception). Upon receiving the successful verification result, the original first sending node confirms that the global cutoff point has been updated to the local cutoff point (T180) by the second sending node. At this point, the node's local progress is consistent with the cluster's global progress. The node exits the read-only synchronization state, becomes a standby node, and stops sending data. Since the global cutoff point has advanced to T180, subsequent sending tasks are continued by the second sending node from T181.

[0049] For example, when the first sending node N1 rejoins the cluster after a network partition recovery, it first enters a read-only / suspended state, prohibiting any data extraction or sending operations until it completes state synchronization with the second sending node (such as N3). During state synchronization, N1 initiates a state query request to the current extractor N3 to obtain the current global cutoff point. If N1's local cutoff point lags behind N3's, N1 automatically discards all data in its local cache that is earlier than the global cutoff point and resynchronizes from transactions after the global cutoff point. If N1's local cutoff point is ahead of N3's (i.e., N1 sent data during the isolation period), N1 submits its locally sent but unconfirmed transaction records to N3. After verifying the legality of these transactions, N3 updates them to the new global cutoff point, thereby achieving a smooth state merge. After completing state alignment, N1 voluntarily relinquishes its status as a sending node and becomes a standby node, waiting for the next election.

[0050] As an example of an embodiment of the present invention, the data sending method for a distributed database further includes: If the current replica node is the first sending node, then add the first election identifier to the batch data being sent; If the current replica node is the second sending node, then a second election identifier is added to the batch data being sent; The first election identifier and the second election identifier are monotonically increasing numerical identifiers determined according to the election order. They are used to retain the batch data corresponding to the larger election identifier and delete the batch data corresponding to the smaller election identifier when there is duplicate data in the batch data received by the target system.

[0051] In this embodiment, each sending node elected is assigned an election identifier as a monotonically increasing number based on the election order (e.g., the first sending node receives identifier 1, the second sending node receives identifier 2, and so on). When the current replica node is elected as the first sending node, it adds the first election identifier to the metadata (such as message headers or packet prefixes) of each batch of data to be sent; similarly, if the current replica node is elected as the second sending node, a second election identifier is added to the batch of data to be sent. These batches of data carrying election identifiers are sent to the target system via the network. The target system maintains a deduplication status table, using the unique identifier of each piece of data (e.g., a combination of transaction ID and transaction sequence number) as the key, recording the largest election identifier that has been received. When the target system receives a batch of data, it extracts the unique identifier of each data item and the election identifier carried by that batch. For each data item, it queries the status table for the largest recorded election identifier corresponding to that identifier. If the election identifier of the current data is greater than the recorded value, the data is considered valid, written to the target storage, and the maximum value in the status table is updated to the current election identifier. If the election identifier of the current data is less than or equal to the recorded value, the data is considered duplicate data and is discarded without further processing. In this way, even during network partitioning or failover, if the old sending node and the new sending node may send the same batch of data, the downstream system can automatically retain data from the latest election cycle and discard duplicate data from the old cycle based on the size of the election identifier, thus achieving accurate one-time data delivery without relying on upstream coordination.

[0052] As an example of an embodiment of the present invention, the data sending method for a distributed database further includes: When a conflict is detected between the batch data sent by the first sending node during the period of disconnection and the batch data sent by the second sending node, a forced backtracking mechanism is triggered. A safe cutoff point is determined based on the local cutoff point and the global cutoff point; In response to the forced backtracking mechanism, the local cutoff point is forcibly reset to the safe cutoff point, and other replica nodes are notified to reset synchronously.

[0053] In this embodiment, when a transaction identification gap (e.g., missing intermediate transaction ID) or data content inconsistency is detected between the batch data sent by the first sending node during the disconnection period and the batch data sent by the second sending node, it is determined that normal state alignment cannot be completed, triggering a forced backtracking mechanism. First, a safe cutoff point agreed upon by all replica nodes is determined. This safe cutoff point is typically selected from the last cutoff point jointly confirmed by all nodes before the partition occurred, or the minimum value among the current local cutoff points of all replica nodes, ensuring that data before this point has been correctly received by the downstream target system without dispute. After determining the safe cutoff point, the forced backtracking mechanism is activated, forcibly resetting the local cutoff point of each replica node to this safe cutoff point. Specifically, the first sending node discards all data in its local data to be migrated that is after the safe cutoff point (this data may come from different nodes and may have conflicts or be missing), and rolls back the migration progress recorded locally to the safe cutoff point. Simultaneously, all other replica nodes are notified through the inter-replica communication protocol to synchronously perform the same reset operation, ensuring complete uniformity of the migration state within the entire replica group. After the reset is complete, the second sending node currently responsible for sending will resend all subsequent data batches starting from the position after the safe truncation point. The downstream target system will then use the idempotent deduplication mechanism of the transaction unique identifier to filter out duplicate data that may have been received before.

[0054] As an example of an embodiment of the present invention, the election of the first sending node based on all the performance evaluation values ​​and a preset election strategy specifically includes: The replica node corresponding to the maximum value of the performance evaluation value is determined as the first sending node. If multiple replica nodes have the same performance evaluation value, and all of them are the maximum value of the performance evaluation value, then among the multiple replica nodes with the same performance evaluation value, the replica node corresponding to the minimum node identifier value is selected as the first sending node. The performance evaluation value of the current replica node is obtained by normalizing the performance indicators of the current replica node, determining the initial score corresponding to each performance indicator, and then weighting and summing all the initial scores according to a preset weight ratio.

[0055] In this embodiment, each replica node periodically collects multiple dynamic performance metrics, including CPU utilization, memory utilization, disk I / O latency, network latency and packet loss rate to the target system, and the hit rate of the data to be migrated in memory. After collecting the raw metrics, the node performs normalization processing on each metric: for example, for CPU utilization, 0% is mapped to 100 points and 100% is mapped to 0 points; for network latency, the lowest latency is mapped to 100 points and the highest latency is mapped to 0 points; other metrics are mapped to a unified scoring range (e.g., 0-100 points) according to the same principle, obtaining an initial score for each metric. Subsequently, the node performs a weighted summation of all initial scores according to a preset weight ratio and a preset calculation formula to calculate the performance evaluation value of the current replica node. The calculation formula is expressed as: (H score )=W cpu S cpu +W mem S mem +W io S io +W net S net +W aff S aff Among them, S cpu S mem S io S net S aff These are the normalized scores for CPU utilization, memory usage, disk I / O latency, network latency and packet loss rate to the target system, and the hit rate of the data to be migrated in memory. cpu W mem W io W net W aff These are the weighting coefficients for each indicator, summing to 1. These weights can be configured by the database administrator based on the business scenario. For example, if the migration bottleneck is mainly in the network, W can be increased. net The weight of W can be increased if the CPU load on database nodes is generally high. cpu The weight.

[0056] All replica nodes exchange their calculated performance evaluation values ​​via an internal communication protocol. During election, each node independently compares all collected performance evaluation values ​​and determines the maximum value. If this maximum value uniquely corresponds to a replica node, that node is elected as the first sending node. If multiple replica nodes have the same performance evaluation value, all of which are the maximum, the node with the smallest node identifier (e.g., node ID) among these nodes with the same score is selected as the first sending node. This election strategy, which prioritizes the highest score and selects the node with the smallest ID in case of a tie, ensures that the election result is unique and deterministic under any circumstances, avoiding inconsistencies in election results (split-brain) caused by identical scores or network latency. It also allows the system to dynamically allocate migration tasks to the node with the best overall capabilities at present.

[0057] As an example of an embodiment of the present invention, the data sending method for a distributed database further includes: If the current replica node is the first sending node, monitor the remaining processing capacity of the first sending node in real time; When the remaining processing capacity is less than a preset safety threshold, the amount of data that can be processed is calculated based on the current data flow rate and the remaining processing capacity. Based on the amount of data that can be processed and the current cutoff point, a predicted cutoff point is determined, and the predicted cutoff point and a prompt message for switching sending nodes are sent to the other replica nodes. Continue sending data until the preset cutoff point is reached, at which point the election trigger condition is met, and actively stop sending data.

[0058] In this embodiment, after the current replica node is elected as the first sending node, a lightweight load monitoring module runs on that node to continuously monitor its remaining processing capacity in real time. This module collects the following metrics in real time: CPU / memory utilization (the node's current computing resource usage); migration queue depth (the amount of locally cached, yet-to-be-sent data to be migrated); and data throughput rate (the amount of data processed and sent by the node per unit time). Based on these metrics, the node's remaining processing capacity C is calculated. A pre-set safety threshold is then established. When the first sending node detects that its remaining processing capacity C is below this safety threshold, it determines that the current node is about to enter a high-load state and triggers a predictive switchover process.

[0059] Specifically, the node calculates Q=V×T based on the current data flow rate V (e.g., the amount of data successfully sent per unit time) and the remaining processing capacity C to obtain the amount of data that can be stably processed in the future, i.e., the amount of data that can be processed. Using the current cutoff point that has been sent and acknowledged as a benchmark, and adding this amount of processable data, a predicted cutoff point P is calculated. target(For example, if the current cutoff point is T120 and the data volume can be processed in 30 batches, then the predicted cutoff point is T150). The first sending node will predict the cutoff point P. target The notification message indicating the switching of the sending node (such as a "Prepare to Switch" flag) is broadcast to all other replica nodes in the group via the internal replica communication protocol, informing them that the current node is about to exit the sending role and asking other nodes to prepare to take over. After this, the first sending node continues to execute the data sending task, sending the remaining batches of data in the original order until its sending progress reaches the predicted cutoff point P. target (i.e., T150). When the predicted cutoff point is reached, the node actively stops sending data and meets the preset election trigger conditions (i.e., proactive optimization switching), triggering a new round of election. Other healthy replica nodes in the cluster, having received the predicted cutoff point and switching notification information in advance, can preload the data stream after the cutoff point and enter hot standby mode. Thus, after the election is completed, the newly elected sending node will start from the next position P after the predicted cutoff point. target+1 (T151) Initiates seamless transmission, achieving a smooth and uninterrupted load balancing switchover. Through the aforementioned predictive switching methods, the node can proactively relinquish transmission rights before the node load becomes too high, preventing the node from experiencing performance degradation or failure due to overload, while ensuring that no data is lost or duplicated during the switchover process.

[0060] As an example of an embodiment of the present invention, the data sending method for a distributed database further includes: If the current replica node is not the sending node, then the local data queue of the current replica node is logically divided into an active buffer and a cleanup buffer. The active buffer is used to store real-time written data after the previous cutoff point, and the cleanup buffer is used to store local storage data that has been sent but has not received global confirmation information. The difference in data volume between the active buffer and the buffer to be cleaned is monitored in real time. When the difference in data volume is greater than a preset storage security threshold, a flow control signal is sent to the data writing node to reduce the writing rate of the corresponding data partition until the difference in data volume is less than the storage security threshold. When the current cutoff point of the current sending node is received, the memory block before the current cutoff point is marked as a reclaimable segment. During the idle window period when the input / output load is less than the preset load threshold, the sequence number and process reference of the reclaimable segment are checked to see if they meet the preset cleanup conditions. If so, the memory block corresponding to the reclaimable segment is physically released, and the released storage space is added to the free block list for use in subsequent real-time data overwriting.

[0061] It should be noted that in real-time migration scenarios, database writes are continuous, while the migration process is phased. If standby nodes (N2, N3) passively receive cutoff point information while waiting to become standby nodes, they will face the following problems: storage bloat, i.e., standby nodes need to cache all data from the previous cutoff point to the latest write; if the migration link is blocked or there is network jitter, the local queue of the standby node will quickly accumulate, leading to memory overflow or disk space exhaustion; cleanup conflicts, i.e., if cleanup logic and write logic compete in the same physical storage area, it will lead to a decrease in write performance. To solve the above problems, the method of this embodiment is proposed.

[0062] In this embodiment, when the current replica node is not elected as the sending node (i.e., it is in standby state), the node logically divides its local data queue to be migrated into two areas: an active buffer and a cleanup buffer. The active buffer is used to store real-time written data newly generated since the previous cutoff point, i.e., data that has not yet been confirmed for migration; the cleanup buffer is used to store local data that has been sent but has not yet received global confirmation information (cutoff point broadcast).

[0063] To prevent excessive storage pressure on the standby node, a backpressure mechanism is introduced. This mechanism continuously monitors the difference in data volume between the active buffer and the buffer to be cleaned (i.e., the amount of unmigrated data). When this difference exceeds a preset storage safety threshold, it indicates excessive local data backlog and a risk of storage exhaustion. In this case, the standby node sends a flow control signal to the data writing node (e.g., the primary replica) handling the write operation of that data partition via an internal communication protocol. Upon receiving the flow control signal, the data writing node reduces the write rate of the corresponding data partition based on the overall cluster load (e.g., adjusting the transaction commit confirmation frequency) until the data volume difference monitored by the standby node falls back below the storage safety threshold, thereby preventing the standby node from crashing due to data accumulation.

[0064] Meanwhile, to avoid impacting real-time write performance on cleanup operations, a "segmented marking-asynchronous reclamation" strategy is adopted. When a standby node receives the current truncation point broadcast by the current sending node, it does not immediately perform physical deletion. Instead, it marks the memory blocks before the truncation point as reclaimable segments. A low-priority background cleanup thread is started, which only performs cleanup operations during idle windows when the input / output load is below a preset load threshold. Before cleanup, the thread performs double checks: first, it compares whether the transaction sequence number of the data blocks in the reclaimable segment is strictly less than the confirmed migration position; second, it checks whether the data block is referenced by other internal processes (such as backup tasks) (i.e., whether the reference count is 0). Only when both checks pass is the memory block corresponding to the reclaimable segment physically released, and the released storage space is added to the free block linked list for subsequent new real-time write data to directly overwrite. This achieves the recycling of storage space, avoids the overhead of frequent memory allocation and release, and ensures that cleanup operations do not affect real-time write performance.

[0065] In summary, the migration capability of this invention is distributed across all replica nodes. When any node fails, other healthy nodes can automatically take over, ensuring uninterrupted migration service and significantly improving system availability. Through truncation point broadcasting and fault resumption mechanisms, data is ensured to be successfully sent at least once. Combined with idempotent deduplication in the downstream system, end-to-end precise one-time data migration can be achieved, avoiding duplication or omissions. Based on dynamic health election and truncation point state synchronization, fault switching is completed automatically, with the new sending node precisely resuming transmission from its local truncation point without manual intervention; the switching process is transparent to the downstream. State synchronization and election are fully implemented using the database kernel's replica communication protocol, eliminating the need for external coordination components such as ZooKeeper, reducing architectural complexity and operational costs, and preventing external components from becoming new performance bottlenecks or failure points. The migration flow for each data partition is managed independently, and the migration throughput can scale linearly with the number of cluster nodes, meeting the needs of large-scale real-time data synchronization.

[0066] like Figure 2 As shown, based on the above method embodiments, an embodiment of the present invention provides a replica node 200, including: an election module 201, a sending module 202, and a resuming module 203; The election module 201 is used to obtain performance evaluation values ​​and receive performance evaluation values ​​from other replica nodes, and elect a first sending node based on all the performance evaluation values ​​and a preset election strategy. The sending module 202 is used to send the first data to be migrated stored locally to the target system in batches if the replica node is the first sending node. During the sending process of each batch of data, if the sending is successful, the current cutoff point corresponding to the current batch of data is generated and the current cutoff point is sent to all the replica nodes until the sending of the first data to be migrated is completed. The resume transmission module 203 is used to re-elect a second sending node if the replica node is not the first sending node and the first sending node meets the preset election triggering conditions; if the current replica node is the second sending node, it continues to send the next batch of data according to the second data to be migrated stored locally and the received current cutoff point, until the sending of the second data to be migrated is completed, wherein the first data to be migrated and the second data to be migrated are the same.

[0067] like Figure 3 As shown, based on the above method embodiments, an embodiment of the present invention further provides a data sending system 300 for a distributed database, including: a target system 301 and multiple replica nodes 302; Each replica node 302 executes the data sending method for distributed databases as described in this invention; The target system 303 is used to receive batch data sent by the replica node. When there is duplicate data in the received batch data, it compares the size relationship of the election identifiers corresponding to the duplicate data, retains the batch data corresponding to the larger election identifier, and deletes the batch data corresponding to the smaller election identifier.

[0068] It is understood that the above system item embodiments correspond to the method item embodiments of the present invention, and can implement the data sending method for distributed databases provided by any of the above method item embodiments of the present invention.

[0069] It should be noted that the system embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0070] For ease of description and brevity, the system embodiments of the present invention include all the implementation methods described in the above embodiments of the data transmission method for distributed databases, and will not be repeated here.

[0071] Based on the above embodiments of the data transmission method for distributed databases, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the data transmission method for distributed databases of any embodiment of the present invention.

[0072] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0073] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0074] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0075] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the data transmission method for a distributed database described in any of the above-described method embodiments of the present invention.

[0076] Based on the above-described method embodiments, this invention also provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of any of the above-described method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0077] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0078] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data transmission method for distributed databases, characterized in that, The method, applied to replica nodes in a distributed database, includes: Obtain performance evaluation values ​​and receive performance evaluation values ​​from other replica nodes; elect the first sending node based on all the performance evaluation values ​​and a preset election strategy. If the replica node is the first sending node, the first data to be migrated stored locally is sent to the target system in batches. During the sending of each batch of data, if the sending is successful, the current cutoff point corresponding to the current batch of data is generated and the current cutoff point is sent to all the replica nodes until the sending of the first data to be migrated is completed. If the replica node is not the first sending node, and the first sending node meets the preset election triggering conditions, a second sending node is re-elected. If the current replica node is the second sending node, the next batch of data is sent according to the second data to be migrated stored locally and the received current cutoff point, until the sending of the second data to be migrated is completed. The first data to be migrated and the second data to be migrated are the same.

2. The data transmission method for a distributed database as described in claim 1, characterized in that, The election triggering condition includes the existence of a network partition in the first sending node, and the data sending method for the distributed database further includes: If the current replica node is the first sending node and loses connection with other replica nodes due to the network partition, the first sending node continues to send batch data and records the cutoff point during the period of disconnection, and meets the election triggering condition to re-elect a second sending node; Until the first sending node re-establishes a connection with the other replica nodes, the latest local truncation point and local data to be migrated in the first sending node are determined, the first sending node is set to read-only synchronization state, and the latest global truncation point is obtained by querying the second sending node. Based on the global cutoff point and the local cutoff point, the state is aligned with the second sending node, and the first sending node after state alignment is converted into a non-sending node.

3. The data transmission method for distributed databases as described in claim 2, characterized in that, The step of aligning the state with the second sending node based on the global truncation point and the local truncation point specifically involves: If the local cutoff point lags behind the global cutoff point, then the batch data between the local cutoff point and the global cutoff point in the local data to be migrated is deleted, and the local cutoff point is updated to the global cutoff point to achieve alignment with the state of the second sending node. If the local cutoff point precedes the global cutoff point, the local cutoff point and the batch data between the local cutoff point and the global cutoff point are sent to the second sending node, and the verification result of the second sending node is received. If the verification result is successful, the global cutoff point is confirmed to be updated to the local cutoff point to achieve state alignment with the second sending node. The verification result is used to characterize the result of the second sending node verifying whether the batch data between the local cutoff point and the global cutoff point has been successfully sent to the target system.

4. The data transmission method for distributed databases as described in claim 1, characterized in that, The data transmission method for distributed databases also includes: If the current replica node is the first sending node, then add the first election identifier to the batch data being sent; If the current replica node is the second sending node, then a second election identifier is added to the batch data being sent; The first election identifier and the second election identifier are monotonically increasing numerical identifiers determined according to the election order. They are used to retain the batch data corresponding to the larger election identifier and delete the batch data corresponding to the smaller election identifier when there is duplicate data in the batch data received by the target system.

5. The data transmission method for a distributed database as described in claim 2, characterized in that, The data transmission method for distributed databases also includes: When a conflict is detected between the batch data sent by the first sending node during the period of disconnection and the batch data sent by the second sending node, a forced backtracking mechanism is triggered. A safe cutoff point is determined based on the local cutoff point and the global cutoff point; In response to the forced backtracking mechanism, the local cutoff point is forcibly reset to the safe cutoff point, and other replica nodes are notified to reset synchronously.

6. The data transmission method for a distributed database as described in claim 1, characterized in that, The process of electing the first sending node based on all the performance evaluation values ​​and a preset election strategy is as follows: The replica node corresponding to the maximum value of the performance evaluation value is determined as the first sending node. If multiple replica nodes have the same performance evaluation value, and all of them are the maximum value of the performance evaluation value, then among the multiple replica nodes with the same performance evaluation value, the replica node corresponding to the minimum node identifier value is selected as the first sending node. The performance evaluation value of the current replica node is obtained by normalizing the performance indicators of the current replica node, determining the initial score corresponding to each performance indicator, and then weighting and summing all the initial scores according to a preset weight ratio.

7. The data transmission method for a distributed database as described in claim 1, characterized in that, The data transmission method for distributed databases also includes: If the current replica node is the first sending node, monitor the remaining processing capacity of the first sending node in real time; When the remaining processing capacity is less than a preset safety threshold, the amount of data that can be processed is calculated based on the current data flow rate and the remaining processing capacity. Based on the amount of data that can be processed and the current cutoff point, a predicted cutoff point is determined, and the predicted cutoff point and a prompt message for switching sending nodes are sent to the other replica nodes. Continue sending data until the preset cutoff point is reached, at which point the election trigger condition is met, and actively stop sending data.

8. The data transmission method for a distributed database as described in claim 1, characterized in that, The data transmission method for distributed databases also includes: If the current replica node is not the sending node, then the local data queue of the current replica node is logically divided into an active buffer and a cleanup buffer. The active buffer is used to store real-time written data after the previous cutoff point, and the cleanup buffer is used to store local storage data that has been sent but has not received global confirmation information. The difference in data volume between the active buffer and the buffer to be cleaned is monitored in real time. When the difference in data volume is greater than a preset storage security threshold, a flow control signal is sent to the data writing node to reduce the writing rate of the corresponding data partition until the difference in data volume is less than the storage security threshold. When the current cutoff point of the current sending node is received, the memory block before the current cutoff point is marked as a reclaimable segment. During the idle window period when the input / output load is less than the preset load threshold, the sequence number and process reference of the reclaimable segment are checked to see if they meet the preset cleanup conditions. If so, the memory block corresponding to the reclaimable segment is physically released, and the released storage space is added to the free block list for use in subsequent real-time data overwriting.

9. A replica node, characterized in that, include: Election module, sending module, and resume module; The election module is used to obtain performance evaluation values ​​and receive performance evaluation values ​​from other replica nodes, and elect a first sending node based on all the performance evaluation values ​​and a preset election strategy. The sending module is used to send the first batch of data to be migrated stored locally to the target system in batches if the replica node is the first sending node. During the sending process of each batch of data, if the sending is successful, the current cutoff point corresponding to the current batch of data is generated and the current cutoff point is sent to all the replica nodes until the sending of the first batch of data to be migrated is completed. The resume module is used to re-elect a second sending node if the replica node is not the first sending node and the first sending node meets the preset election triggering conditions. If the current replica node is the second sending node, it continues to send the next batch of data according to the second data to be migrated stored locally and the received current cutoff point, until the sending of the second data to be migrated is completed, wherein the first data to be migrated and the second data to be migrated are the same.

10. A data transmission system for distributed databases, characterized in that, include: The target system and multiple replica nodes; Each of the replica nodes executes the data sending method for a distributed database as described in any one of claims 1-8; The target system is used to receive batch data sent by the replica node. When there is duplicate data in the received batch data, it compares the size relationship of the election identifiers corresponding to the duplicate data, retains the batch data corresponding to the larger election identifier, and deletes the batch data corresponding to the smaller election identifier.

Images