Data redistribution method and device for distributed memory database, equipment and medium

Abstract

The invention discloses a data redistribution method, device and equipment for a distributed memory database and a medium, and belongs to the field of data migration. The method comprises the steps that expansion nodes are added to the distributed memory database, and data balance processing is conducted on the distributed memory database; screening target migration data from the to-be-migrated data according to the transmission time consumption calculated in real time; and performing dynamic migration processing according to the target migration data. According to the method, the data volume of each migration is determined through the predetermined acceptable service pause duration, the data can be divided into smaller data which can be flexibly split for transmission, and the granularity of the data is refined; dynamic adjustment is carried out according to the real-time migration state, resource competition with foreground transactions is reduced, resource waste is avoided, and the resource utilization rate is increased.

Description

Technical Field

[0001] This invention relates to the technical field of distributed data storage, and in particular to a method, apparatus, device, and medium for redistributing data in a distributed in-memory database. Background Technology

[0002] Relational databases are database systems that organize data based on the relational model. They use a two-dimensional table structure to store data, maintain data integrity through primary and foreign keys, support data manipulation using Structured Query Language (SQL), and adhere to ACID transaction principles and normalization theory to achieve data optimization. Their logical data structure is separated from physical storage, table relationships are established through common columns, and they support stored procedures and concurrency control technologies, making them widely used in enterprise-level data management.

[0003] Because relational in-memory databases store data in memory and distribute it across multiple nodes, they require database expansion to meet the demands of high concurrency and low latency when high-concurrency data is input. One common data processing method is to add new nodes to the database through "manual expansion," "downtime expansion," or "semi-automatic expansion." This requires maintenance personnel to manually configure partition mappings and then migrate data to different nodes using the fixed partition rules set in the mappings to achieve data migration and storage.

[0004] However, the above methods have the following technical problems: whether it is manual, shutdown or semi-automatic expansion, data is migrated through fixed partitioning rules, which not only makes the partitioning granularity coarse, but also consumes a lot of CPU and bandwidth resources. Once a large amount of data is migrated, it will occupy the bandwidth and resources of business processing, causing the current business to be interrupted, which increases the processing latency and affects the business storage processing efficiency. Summary of the Invention

[0005] This invention provides a data redistribution method, apparatus, device, and medium for a distributed in-memory database, which can solve the technical problems of low processing efficiency and potential business interruption when migrating data using fixed rules in existing technologies.

[0006] A first aspect of this invention provides a data redistribution method for a distributed in-memory database, the method comprising: Add expansion nodes to the distributed memory database and perform data balancing on the distributed memory database. The data balancing process is the process of determining the node partition to which the data to be migrated in the distributed memory database belongs. The target migration data is selected from the data to be migrated based on the real-time calculated transmission time, wherein the transmission time is determined based on the preset service pause duration to determine the duration required to transmit the data; Dynamic migration processing is performed based on the target migration data, wherein the dynamic migration processing is the process of adjusting the migration frequency of the target migration data during the migration of the data to be migrated from the original node to the expansion node.

[0007] In conjunction with the first aspect, in one implementation, adding expansion nodes to the distributed in-memory database and performing data balancing processing on the node cluster within the distributed in-memory database includes: Determine the corresponding number of expansion nodes based on business needs, and add the expansion nodes to the distributed memory database; A node hash ring is generated based on the node cluster of the distributed memory database, and the data to be migrated is determined within the node hash ring. The node cluster includes the expansion node and the original node of the distributed memory database. The data to be migrated is hashed based on the partition key under the node hash ring, and the partition of the node to which the data to be migrated belongs is determined based on the hash calculation result.

[0008] In conjunction with the first aspect, in one implementation, the step of filtering target migration data from the data to be migrated based on the real-time calculated transmission time includes: Generate an index table of the data to be migrated and determine the preset service pause duration and target throughput, wherein the preset service pause duration is the upper limit of the delay time of the business transaction set by the user in advance, and the target throughput is the capacity of data to be migrated per unit time. The amount of processing per transaction is determined based on the preset service pause duration and the target throughput. Based on the single transaction volume, data of corresponding capacity is filtered from the data to be migrated to obtain the target migration data.

[0009] In conjunction with the first aspect, in one implementation, the dynamic migration processing based on the target migration data includes: The migration frequency value is determined based on the target throughput, and the target migration data is migrated according to the migration frequency value and the index table; Obtain the real-time migration throughput of the target migration data, and determine the deviation between the real-time migration throughput and the target throughput; If the deviation value does not meet the preset deviation threshold, the migration frequency value is adjusted, and the real-time migration throughput is reacquired until the data to be migrated is migrated.

[0010] In conjunction with the first aspect, in one implementation, migrating the target migration data based on the target throughput and the index table includes: Based on the index table, accurate requests are sent simultaneously to partitions of two different nodes; After receiving feedback from the two partitions, the target migration data is read from the partition of the first node in a single thread based on the target throughput, and the target migration data is written to the partition of the second node in a single thread, while blocking transactions in the partitions of the first and second nodes.

[0011] In conjunction with the first aspect, in one implementation, the blocking of transactions in the partitions of the first node and the second node includes: When reading the target migration data from the partition of the first node, a change log is generated; After the target migration data is written to the partition of the second node, the change log is applied to the partition of the second node, and transactions of the partitions of the first node and the partition of the second node are paused during the application process.

[0012] In conjunction with the first aspect, in one implementation, the method further includes: After the distributed in-memory database restarts due to a crash and the stored metadata is determined, snapshot data is obtained. The metadata includes the node cluster and the storage information of each node. The snapshot data is a complete data copy with the shortest interval between the time of the distributed in-memory database crash. The historical hash ring is obtained from the snapshot data, and the metadata is distributed according to the historical hash ring.

[0013] A second aspect of the present invention provides a data redistribution apparatus for a distributed in-memory database, the apparatus comprising: The balancing module is used to add expansion nodes to the distributed memory database and perform data balancing processing on the distributed memory database. The data balancing processing is the process of determining the node partition to which the data that needs to be migrated in the distributed memory database belongs. The filtering module is used to filter target migration data from the data to be migrated based on the real-time calculated transmission time, wherein the transmission time is the duration required to transmit the data determined based on the preset service pause duration; The migration module is used to perform dynamic migration processing based on the target migration data. The dynamic migration processing is the process of adjusting the migration frequency of the target migration data during the migration of the data to be migrated from the original node to the expansion node.

[0014] Compared to existing technologies, the data redistribution method, apparatus, device, and medium for a distributed in-memory database provided in this invention offer the following advantages: This invention can add expansion nodes to the distributed in-memory database and perform data balancing; it can filter target migration data from the data to be migrated based on real-time calculated transmission time; and it performs dynamic migration processing based on the target migration data. This invention determines the amount of data to be migrated each time by pre-determined acceptable service pause duration, allowing data to be divided into smaller, flexibly splittable data for transmission, refining the granularity of the data; and it dynamically adjusts based on the real-time migration status, reducing resource competition with foreground transactions, avoiding resource waste, and improving resource utilization. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a data redistribution method for a distributed memory database according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the framework of a newly added expansion node provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a new hash ring and a long hash ring provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the range of the data to be migrated within the new hash ring, provided by an embodiment of the present invention. Figure 5 This is a schematic diagram illustrating the specific range of data to be migrated according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the index structure of an index table provided in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating how data sub-ranges to be migrated from the original partition 3 to the new partition 6, provided by an embodiment of the present invention. Figure 8 This is a schematic diagram of the dynamic migration processing provided in an embodiment of the present invention; Figure 9 This is a flowchart of a partitioning operation that maintains a serializable isolation level, provided by an embodiment of the present invention. Figure 10 This is a schematic diagram of a process for rerouting in-transit transactions with routing errors, provided by an embodiment of the present invention. Figure 11 This is a schematic diagram of a database data recovery process provided in an embodiment of the present invention; Figure 12 This is an operation flowchart of a data redistribution method for a distributed in-memory database provided in an embodiment of the present invention; Figure 13 This is a schematic diagram of the connection structure between the rebalancing coordinator and each node provided in an embodiment of the present invention; Figure 14 This is a schematic diagram of the structure of a data redistribution device for a distributed memory database provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Relational databases are database systems that organize data based on the relational model. They use a two-dimensional table structure to store data, maintain data integrity through primary and foreign keys, support data manipulation using Structured Query Language (SQL), and adhere to ACID transaction principles and normalization theory to achieve data optimization. Their logical data structure is separated from physical storage, table relationships are established through common columns, and they support stored procedures and concurrency control technologies, making them widely used in enterprise-level data management.

[0018] Because relational in-memory databases store data in memory and distribute it across multiple nodes, they require database expansion to meet the demands of high concurrency and low latency when high-concurrency data is input. One common data processing method is to add new nodes to the database through "manual expansion," "downtime expansion," or "semi-automatic expansion." This requires maintenance personnel to manually configure partition mappings and then migrate data to different nodes using the fixed partition rules set in the mappings to achieve data migration and storage.

[0019] However, the above methods have the following technical problems: whether it is manual, shutdown or semi-automatic expansion, data is migrated through fixed partitioning rules, which not only makes the partitioning granularity coarse, but also consumes a lot of CPU and bandwidth resources. Once a large amount of data is migrated, it will occupy the bandwidth and resources of business processing, causing the current business to be interrupted, which increases the processing latency and affects the business storage processing efficiency.

[0020] To address the aforementioned issues, the following detailed embodiments will be used to describe and explain a data redistribution method, apparatus, device, and medium for a distributed in-memory database provided in this application.

[0021] To address the technical issues of low processing efficiency and potential business interruption when migrating data using fixed rules in existing technologies, referencing Figure 1 The diagram shows a flowchart of a data redistribution method for a distributed memory database according to an embodiment of the present invention.

[0022] As an example, the data redistribution method for the distributed in-memory database may include: S11. Add expansion nodes to the distributed memory database and perform data balancing processing on the distributed memory database, wherein the data balancing processing is the process of determining the node partition to which the data to be migrated in the distributed memory database belongs.

[0023] The distributed in-memory database can be a ShuangzhaoDB database cluster. One or more nodes can be added to the distributed in-memory database for scaling. After adding scaling nodes, data balancing can be performed on the distributed in-memory database. This data balancing process determines the node partitions to which the data to be migrated belongs. Through data balancing, the data to be migrated and the partitions of the new nodes to which the migrated data belongs can be determined, so that data transfer can be performed according to the partitions of the new nodes.

[0024] Reference Figure 2 The diagram shows a schematic of the framework of a newly added expansion node provided in an embodiment of the present invention.

[0025] As an example, the step of adding expansion nodes to the distributed in-memory database and performing data balancing on the node cluster within the distributed in-memory database may include the following sub-steps: S111. Determine the corresponding number of expansion nodes according to business needs, and add the expansion nodes to the distributed memory database.

[0026] S112. Generate a node hash ring based on the node cluster of the distributed memory database, and determine the data to be migrated within the node hash ring. The node cluster includes the expansion node and the original node of the distributed memory database.

[0027] S113. Perform hash calculation on the data to be migrated according to the partition key under the node hash ring, and determine the partition of the node to which the data to be migrated belongs based on the result of the hash calculation.

[0028] Distributed in-memory database clusters can be scaled up using a "transactional approach," allowing the addition of one or more new nodes. Optionally, the addition of new nodes is typically detected by the ZooKeeper cluster monitoring. For clarity, the following explanation will focus on adding one new node for scaling.

[0029] A new expansion node can be added to the database cluster based on business application needs. The target number of nodes to add is assessed based on the actual business load, data volume, and hardware resources, such as CPU and memory resources. Each new node will automatically generate its own logical partition (e.g., a new node contains logical partitions P5 and P6), but at this time, all new logical partitions are in an "empty data" state, storing no business data. These newly generated logical partitions are not added to the cluster's "consistent hash ring," which is the core basis for cluster data routing. Only the original partitions are retained at this time to ensure that existing transactions are not temporarily affected by the addition of the new node. They are not added to the hash ring yet, as follows: Figure 2 As shown.

[0030] Reference Figure 3-4 The diagrams show a new hash ring and a long hash ring provided by an embodiment of the present invention, and a diagram showing the range of the data to be migrated within the new hash ring provided by an embodiment of the present invention.

[0031] After a new expansion node is added, a "rebalancing coordinator" is automatically elected. This coordinator acts as the core driver of the entire expansion process, responsible for coordinating the planning and execution of data migration. Based on the overall situation of the cluster's current nodes (including the new node) and logical partitions (including the new partition), the coordinator calculates and generates a new consistent hash ring, resulting in a node hash ring. This new hash ring contains all the logical partitions of the new nodes, and the "position" of the new partition on the hash ring corresponds to the data range previously handled by the original partition, such as... Figure 3 As shown. Next, the new node hash ring can be compared with the original hash ring to determine the data to be migrated, referring to... Figure 4 This demonstrates the calculated range of data to be migrated. The coordinator election and the generation of a new hash ring clearly define the data migration target.

[0032] Reference Figure 5 The diagram illustrates the specific range of data to be migrated according to an embodiment of the present invention.

[0033] The coordinator further clarifies the migration details of specific partitions. For example, if a certain existing partition (e.g., P4) needs to migrate some data to a specific new partition (e.g., P5), the coordinator determines the specific range of this data. Figure 5 As shown.

[0034] It should be noted that since scaling is done on a per-node basis, rather than a partition, when migration is required, data will only be migrated to partitions on different nodes. There is no possibility of migrating data between different partitions within the same node.

[0035] When a new node joins the cluster (for example, from 2 nodes and 4 partitions to 3 nodes and 6 partitions), the newly added node (node ​​3) cannot immediately begin operation. The cluster enters a data balancing phase. During this phase, the cluster operates in a data rebalancing state. Only after the data between partitions is balanced can the cluster enter a fully ready state with 3 nodes. Therefore, the complete switch from a 2-node, 4-partition working state to a 3-node, 6-partition working state requires a balancing phase. To avoid impacting business performance, data balancing is performed asynchronously in small batches, and the entire process takes some time to complete. This process is also imperceptible to the business logic, and the same applies to scaling down.

[0036] For example, if a table originally contained 1 million rows of data, hashed according to the partition field, it would be distributed across 4 partitions on 2 nodes, with each partition containing approximately 250,000 rows. When a new node joins, the cluster becomes 6 partitions, with the 1 million rows distributed across these 6 partitions (approximately 167,000 rows per partition). This involves migrating some data from the original 4 partitions to partitions 5 and 6 of the new node. The specific implementation involves using a new hash ring under the 6-partition topology to hash the data based on the partition key to determine the partition to which the data belongs, thus filtering out the data that needs to be migrated. This mechanism enables elastic scaling without downtime.

[0037] S12. Select target migration data from the data to be migrated based on the real-time calculated transmission time, wherein the transmission time is the duration required to transmit the data determined based on the preset service pause duration.

[0038] Since the amount of data to be migrated may be large, the target migration data required for each migration can be selected from the data to be migrated based on the real-time calculated transmission time, and the target migration data can be migrated each time.

[0039] Since migration requires resources, a preset service pause duration can be set. This is the time during which business transactions can be paused. The time for each migration data transmission is then calculated based on this preset pause duration, and the required target migration data is determined according to the time. This method ensures that business transactions are not interrupted for extended periods while improving transmission flexibility and efficiency.

[0040] In an optional embodiment, the step of filtering target migration data from the data to be migrated based on the real-time calculated transmission time may include the following sub-steps: S121. Generate an index table of the data to be migrated and determine the preset service pause duration and target throughput, wherein the preset service pause duration is the upper limit of the delay time of the service transaction set by the user in advance, and the target throughput is the capacity of data to be migrated per unit time.

[0041] S122. Determine the amount of processing per transaction based on the preset service pause duration and the target throughput.

[0042] S123. Based on the single transaction processing volume, filter data of corresponding capacity from the data to be migrated to obtain the target migration data.

[0043] Reference Figure 6 The diagram shows a schematic representation of the index structure of an index table provided in an embodiment of the present invention.

[0044] In one embodiment, a rebalancing coordinator can be initiated, followed by a scanning process that generates a dedicated index for each table in the distributed in-memory database cluster. The core function of this index is to mark all rows that need to be migrated across the rebalancing range (i.e., to identify which data in each table falls within the migration scope). This index is automatically updated as part of regular transaction execution, ensuring that transactions running normally during the scaling process do not cause the index to become invalid. Simultaneously, the background scanning process runs entirely in the background, not consuming core transaction processing resources, and therefore does not block normal transaction execution. Specifically, as follows... Figure 6 As shown, Figure 6 An index structure created for the data to be migrated.

[0045] After the index is generated, the coordinator selects a portion of the data from the determined range to be migrated as the objects for a single migration. The selection is based on a preset business pause duration, which is the expected migration time. For example, if the user sets the maximum pause time to 100ms, the coordinator will estimate the amount of data that can be successfully migrated within the specified pause time based on the current system performance (such as network latency and transaction processing speed). The migration time for this amount of data is the expected migration time, which is usually less than the maximum pause time.

[0046] This ensures that the migration process of this data is controlled within the user-specified "pause time". The "pause time" setting gives users control over the impact of scaling: users can set a clear upper limit for "transaction delays caused by rebalancing work" by setting this time, avoiding uncontrollable impacts on business response speed caused by large-scale data migration.

[0047] Reference Figure 7 This diagram illustrates how, according to an embodiment of the present invention, data is used to determine the data subrange to be migrated from the original partition 3 to the new partition 6.

[0048] Distributed transactions only involve the source and target partitions, so the rebalancing process does not interrupt transaction processing on logical partitions that are not involved in the rebalancing. The execution frequency of transactions used for data migration is determined by the target throughput requested by the user (in rows or megabytes), and this frequency also depends on the amount of data that can be migrated within the requested pause time.

[0049] The coordinator estimates the maximum amount of data a single balancing transaction can migrate (e.g., moving a maximum of 1000 rows or 2MB of data per transaction) based on the user-defined "pause time," ensuring latency remains within acceptable limits. This determines the processing capacity of a single transaction. Then, based on this single transaction processing capacity, the coordinator filters data of the corresponding capacity from the data to be migrated each time, obtaining the target migration data. Filtering the target migration data of the corresponding capacity for transmission each time closely reflects the actual situation and ensures that data migration occurs within the user-defined "pause time," minimizing the impact of migration on business operations.

[0050] S13. Perform dynamic migration processing based on the target migration data, wherein the dynamic migration processing is the process of adjusting the migration frequency of the target migration data during the migration of the data to be migrated from the original node to the expansion node.

[0051] Transactions used for data migration are executed at a frequency determined by the target throughput (in rows or megabytes) requested by the user, which in turn depends on the amount of data that can be migrated within the requested pause time.

[0052] Therefore, the amount of data that can be migrated during the pause time is adjusted to control the frequency of migration transactions, thereby enabling fully dynamic migration of the target data. For example, if the capacity of each transfer is small, the frequency of migration transactions is increased, increasing the number of transactions that migrate the target data; conversely, if the capacity of each transfer is large, the frequency of migration transactions is reduced, decreasing the number of transactions that migrate the target data, thus releasing the computer resources occupied by the migration, ensuring that the migration can be completed within the user-defined "pause time," and improving resource utilization.

[0053] In one embodiment, the dynamic migration processing based on the target migration data may include the following sub-steps: S131. Determine the migration frequency value based on the target throughput, and migrate the target migration data according to the migration frequency value and the index table.

[0054] S132. Obtain the real-time migration throughput of the target migration data, and determine the deviation between the real-time migration throughput and the target throughput.

[0055] S133. If the deviation value does not meet the preset deviation threshold, the migration frequency value is adjusted, and the real-time migration throughput is reacquired until the data to be migrated is migrated.

[0056] In one embodiment, the frequency of executing migration transactions is not fixed, but dynamically calculated based on the target throughput. For example, if the user sets a target throughput (e.g., 10,000 rows / second or 20MB / second), the coordinator performs a simple calculation: theoretical request frequency = target throughput / data volume of a single transaction (e.g., target: 10,000 rows / second, single transaction: 1,000 rows, theoretical throughput: 10,000 / 1,000 = 10 transactions / second). This means that the coordinator needs to initiate a balancing transaction approximately every 100 milliseconds to achieve the target throughput.

[0057] Reference Figure 8 This diagram illustrates a flow chart of dynamic migration processing according to an embodiment of the present invention. The actual throughput of the system is affected by factors such as current database load, network conditions, and data size. Therefore, the coordinator continuously monitors the actual data migration speed and dynamically adjusts the frequency to maintain the target. Figure 8 This demonstrates a closed-loop feedback loop completed by the monitoring and adjustment process.

[0058] In one operation mode, a migration frequency value (e.g., 10 transactions / second in the example above) can be determined based on the target throughput, and a migration transaction can be initiated according to the migration frequency value, and the target migration data can be migrated to the corresponding partition according to the index table.

[0059] During this process, the real-time migration throughput of the target migration data can be obtained, and the difference between the real-time migration throughput and the target throughput can be determined to obtain the deviation value. If the deviation value does not meet the preset deviation threshold, the migration frequency value is adjusted and the migration is performed again. At the same time, the real-time migration throughput can be obtained again, and the difference between the real-time migration throughput and the target throughput can be repeatedly adjusted until the data to be migrated is completed.

[0060] Specifically, if the real-time migration throughput is greater than the target throughput, and the deviation between the real-time migration throughput and the target throughput is greater than a threshold, the migration frequency can be reduced to decrease the number of migration transactions initiated, for example, from 10 transactions / second to 9 transactions / second.

[0061] If the real-time migration throughput is less than the target throughput, and the deviation between the real-time migration throughput and the target throughput is greater than the threshold, the migration frequency can be increased by increasing the number of migration transactions initiated, for example, from 10 transactions / second to 11 transactions / second.

[0062] If the real-time migration throughput is greater than the target throughput and the deviation between the real-time migration throughput and the target throughput is less than a threshold, or if the real-time migration throughput is less than the target throughput and the deviation between the real-time migration throughput and the target throughput is less than a threshold, the migration frequency value can remain unchanged. For specific operations, please refer to... Figure 8 .

[0063] Because distributed transactions transfer data between logical partitions and maintain a serializable isolation level by blocking pending transactions until the transfer is complete, this transfer is atomic from the perspective of data partitioning. This is because the index allows the source partition to accurately locate and extract all relevant rows within the hash range.

[0064] To maintain the serializable isolation level, in one embodiment, migrating the target migration data based on the target throughput and the index table may include the following sub-steps: S1311. Send accurate requests to the partitions of two different nodes simultaneously according to the index table.

[0065] S1312. After receiving feedback information from the two partitions, the target migration data is read from the partition of the first node in a single thread based on the target throughput, and the target migration data is written to the partition of the second node in a single thread, while blocking the transactions of the partitions of the first node and the second node.

[0066] Reference Figure 9 The diagram illustrates a partitioning operation flowchart for maintaining a serializable isolation level according to an embodiment of the present invention.

[0067] Figure 9 This diagram illustrates how the serializable isolation level is maintained during transaction execution. In the diagram, partitions 3 and 6 run in single-threaded mode, trading optimal performance and data consistency for sequential execution. Serializability is the highest level of transaction isolation (traditional databases simulate this serialization effect using complex mechanisms like locking or multi-version concurrency control, but this incurs significant overhead: lock contention, deadlock detection, context switching, etc.). Migration transactions, involving multiple partitions, are upgraded to distributed transactions, which use a two-phase commit protocol to guarantee atomicity. During this process, partitions 3 and 6 work together. The transaction coordinator sends "ready" requests to partitions 3 and 6, which are queued and executed in their respective partition queues. Only after both partitions 3 and 6 return ready do the coordinator send commit requests, executing the commit again in their respective queues. Within the migration transaction, partitions 3 and 6 remain internally single-threaded and serializable. The specific steps are described in detail below: The first step is to enqueue data migration transactions: After the data migration transaction (e.g., transaction to move rows P3->P6) is received by the rebalancing coordinator, it is inserted into the serialization queues of the source partition P3 and the target partition P6, respectively.

[0068] The second step is to ensure the atomicity and consistency of the migration transactions: When a migration transaction is executed in the P3 queue, a batch of data that needs to be migrated is read. During this process, other transactions in the P3 queue are blocked.

[0069] Similarly, when the data is executed in the P6 queue, it will be written to P6, during which other transactions in the P6 queue will be blocked.

[0070] Because the two partitions are guaranteed by the transaction coordinator to either "succeed all" or "fail all," and each partition is executed serially, this ensures that during the migration process, no other transaction will see the data "in transit," and no other task will be able to concurrently modify the data being migrated, thus avoiding data inconsistency.

[0071] The third step is to control transaction delays: As mentioned earlier, migration transactions are broken down into small batches, which only briefly block transactions in their respective queues when executed. Because the batches are small and execution is fast, this blocking time (pause time) is predictable and controllable. Once execution is complete, normal transactions in the queue can continue to be processed.

[0072] Handling client error routing transactions: During the migration process, if a transaction is routed to the old partition (P3), but the data that the transaction needs to operate on has already been migrated to P6, this transaction will not be executed in P3. This is because after the migration transaction is executed, the hash ring version is updated. If a transaction waiting in the queue is found to have data that is no longer in the current partition, it will be marked as misrouted, removed from the queue, and then sent to the loopback layer to be rerouted to the correct new partition (P6) for queued execution.

[0073] During the migration process, the source and target partitions will perform undo log operations; if any related node fails, the transaction will be rolled back and restarted, thus ensuring the ACID semantics of the relational database.

[0074] When a distributed transaction first arrives at each relevant partition, it may not execute immediately if other transactions with higher priority exist in the partition's queue. Transactions en route to migrate hash ranges may arrive before or during the rebalancing transaction. These en route transactions are sent back to the routing grid and rerouted with an updated hash ring containing the latest changes. This design allows hash ring updates to propagate with a delay. The hash ring is versioned and serialized by the distributed transaction coordinator, so delayed updates are safe, and the system can handle situations where hash ring updates are missing or out of order.

[0075] Reference Figure 10 This diagram illustrates a process for rerouting in-transit transactions with routing errors, according to an embodiment of the present invention.

[0076] Figure 10 This demonstrates the process of rerouting in-transit transactions with routing errors. The routing adjustment of in-transit transactions can be based on the delayed propagation mechanism of a versioned hash ring.

[0077] like Figure 10 As shown, there are two transactions in the current transaction queue of partition 3. The first is the migration transaction for row A (when transaction A was added to the transaction queue, the migration transaction for row A had not yet been completed; therefore, from a routing perspective, transaction A would be routed to partition 3). The second is the transaction to delete row A. Both transactions are already ready in the transaction queue. When the first migration transaction in partition 3 is completed, and the deletion transaction for row A is executed in partition 3, row A has already been migrated. Therefore, the current deletion transaction for row A is identified as an erroneous transaction. This transaction will be resent to the routing mesh and rerouted to partition 6 for execution using a new hash ring.

[0078] Additionally, if the system uses a write-ahead log (WAL), the log application must meet the following requirement: even if some data has been migrated to a new location, all transactions can still execute on the same dataset from which they originally operated (i.e., transactions are unaware of changes in data location). Typically, transactions are written to the disk log immediately upon receipt to reduce latency. However, during rebalancing, transactions written to the disk log may ultimately not be executed on the partition where their logs belong. In one or more embodiments, the system handles these "transactions that ultimately fall into the wrong log" by logging the transactions used to migrate data along with regular transactions and replaying the data migration process during the recovery phase. If a transaction is logged in the wrong partition's log, it will be rejected during log replay and will eventually appear in the command log of the correct partition.

[0079] A transaction is routed to P3 and logged in P3's log, but the data to be operated on is subsequently moved to P6. During log recovery and replay, when P3 attempts to replay this transaction, it checks the current actual location of the data. If the data is found to be already on P6, P3 rejects the transaction. This rejected transaction is not lost. The system mechanism (routing to the correct partition based on the latest location of the data) ensures that it is forwarded to the correct partition (P6) in the recovery process and executed on P6. This guarantees that a transaction will ultimately be executed only once on the partition where its data resides.

[0080] Furthermore, in data migration transactions, the two partitions in a blocked state (source and target partitions) can have their routes handled by the transaction system through preemptive load balancing, i.e., directly returning immediate error responses to the client. This design avoids the situation where all transactions entering the system are routed to the blocked partitions, resulting in other unblocked partitions being available but without transaction processing (resource idleness). Load balancing can be implemented in two ways: one is to specify a maximum acceptable wait time for each transaction individually, and the other is to set a unified rule through global configuration. Preemptive load balancing of blocked partitions can balance resource utilization.

[0081] Because data migration requires blocking transactions in two partitions, this operation may delay the processing of existing business and reduce its efficiency. To minimize delays, in one embodiment, blocking transactions in the first node's partition and the second node's partition may include the following sub-steps: S13121. When reading the target migration data in the partition of the first node, a change log is generated.

[0082] S13122. After writing the target migration data to the partition of the second node, the change log is applied to the partition of the second node, and transactions of the partitions of the first node and the second node are paused during the application process.

[0083] To avoid blocking relevant partitions during the entire data transfer process, the following approach is taken: first, record the changes to the data that has undergone rebalancing (i.e., record the change log), and then apply the change log; and only when the last segment of the change log is applied will the transaction be blocked.

[0084] It should be noted that the change log here is a different concept from the write-ahead log (WAL) mentioned earlier.

[0085] The system asynchronously copies a batch of data from the source partition to the target partition without blocking normal transactions. This copying process does not require completely stopping the service on the source partition. After copying begins, the system starts logging all modifications to the copied data. These changes are recorded in a log.

[0086] Once the basic data replication is complete, the system enters a very brief blocking phase. During this phase, new transactions on both the source and target partitions are paused, and the accumulated change logs are quickly applied to the target partition. The hash ring is then atomically switched, formally transferring data ownership to the target partition. Finally, transaction processing on both partitions resumes.

[0087] This reduces the blocking time from the "total data migration time" to just the "last change log application time." For large-scale data migrations, this significantly reduces the impact on business operations.

[0088] In one embodiment, the database may crash due to various circumstances, such as data migration or business processing. To quickly recover the data in the database, in one embodiment, the method may further include the following steps: S14. After the distributed memory database restarts due to a crash and the stored metadata is determined, snapshot data is obtained. The metadata includes the node cluster and the storage information of each node. The snapshot data is a complete data copy with the shortest interval between the time of the distributed memory database crash.

[0089] S15. Obtain the historical hash ring from the snapshot data, and perform data distribution processing on the metadata according to the historical hash ring.

[0090] In one embodiment, after the distributed memory database restarts due to a crash, the stored metadata can be determined within the database, and snapshot data can be obtained. The metadata includes the node cluster and the storage information of each node. The snapshot data is a complete data copy with the shortest interval between the time of the distributed memory database crash and the time of the crash.

[0091] The historical hash ring is obtained from the snapshot data, and the metadata is distributed according to the historical hash ring. Specifically, data migration can be performed in the manner described in steps S11-S13 above, that is, the data partitions are determined according to the historical hash ring, and then the data migration is performed to achieve data recovery.

[0092] The ShuangzhaoDB database system maintains high availability by replicating partitions within the cluster. In this way, if a node fails, only the partition replicas on that node become inaccessible; the cluster retains the complete database dataset. This mechanism for increasing database capacity (referring to the transactional scaling mentioned earlier) also employs the same replication logic (i.e., replicating new partitions to ensure availability) when handling "new partitions added to the hash ring."

[0093] If database persistence is enabled, the database must ensure that a consistent system state is rebuilt at all times, regardless of whether nodes are added / removed or data is being rebalanced. In one or more embodiments, the system achieves this by using a write-ahead log and truncating the log through periodic snapshots of the dataset (i.e., deleting log content that has been persisted before the snapshot, preventing the log from growing indefinitely). These snapshots, the write-ahead log, and their metadata contain enough information to determine which files come from the same cluster and whether there is enough information to load the dataset and re-execute all transactions.

[0094] When the entire distributed database cluster crashes and restarts, the database system needs to reintegrate the data files and organizational information scattered across different nodes and restore them to a consistent state before the crash, especially during the special period when the data location is dynamically changing due to data rebalancing.

[0095] Reference Figure 11 This diagram illustrates a data recovery process for a database according to an embodiment of the present invention. Metadata records the cluster structure and state of the database. Metadata is structural information about the database; it does not contain user data but describes the structure and state of the entire database cluster. It includes valid cluster members to prevent the incorrect addition of nodes or files from other unrelated clusters during recovery.

[0096] Logical structure information, including partition scheme information, table structure information, and version history information of hash functions and hash rings (this is key information for rebalancing and recovery because it records the changes in data distribution rules over time).

[0097] Persistence mechanism: These contingency plans are also persisted, stored together with snapshots and write-ahead logs through a distributed consensus mechanism to ensure their consistency.

[0098] Snapshot data is a dataset at a specific point in time; it's a complete copy of the data at that point. When a cluster crashes, the recovery process doesn't replay all the logs from scratch; instead, it first loads the latest successful snapshot. Snapshots contain crucial metadata, especially the hash ring layout and hash function.

[0099] Write-Ahead Log (WAL): Enables "replaying all transactions". While snapshots provide the basic state at a certain point in time, WAL provides all data changes from the snapshot point to the crash point.

[0100] The WAL itself also contains metadata, such as log sequence numbers. The transactions involved in data rebalancing are also recorded in the WAL.

[0101] The WAL playback and recovery process is as follows: The first step is for the system to restore data from the snapshot.

[0102] The second step is to replay the WAL logs.

[0103] The third step involves re-executing the transaction when a log entry for a "rebalancing transaction" is encountered, moving a batch of data from the source partition to the target partition.

[0104] Fourthly, ordinary user transactions are also replayed. The system uses the old hash function saved in the snapshot to correctly route these transactions. If a transaction is misrouted, the system will forward it to the correct partition for execution.

[0105] The fifth step involves sequentially replaying all transactions (including rebalancing transactions) to accurately reproduce all data state changes and data location changes from the snapshot point to the crash point.

[0106] Specifically, data can be migrated to its corresponding partition using the methods described above, thereby achieving data recovery.

[0107] Additionally, when a new node joins the cluster, the system records the change in the number of logical partitions to disk through distributed transactions. This allows the database to know upon restart that additional log and snapshot data are needed to rebuild the database. The distributed transactions used for rebalancing are also logged, so the data migration process is re-executed during the log replay phase. The data required by the transactions in the log will always be in the correct position because it is either loaded from the snapshot used to truncate the log or moved to the correct position through re-execution of the data migration process.

[0108] Snapshots must contain the hash function used when the snapshot was generated, because the hash function used during log replay may have changed; and if log replay is performed after a snapshot is restored, the replay operation will assume that the data is still in its original storage location (i.e., the location when the snapshot was generated). The consistency of the hash function is the key to ensuring accurate data location.

[0109] The rebalancing process is highly available. If a node fails, all rebalancing transactions in transit will be handled by the distributed transaction mechanism, and these transactions will be automatically rolled back and restarted. If the elected coordinator responsible for driving the rebalancing process fails, the system will immediately replace the coordinator, and the new coordinator will continue to perform the rebalancing operation until the target state of the current cluster topology is reached.

[0110] If the entire cluster fails during the rebalancing process, the rebalancing process will resume once the command logs are replayed. The system will then re-elect a coordinator and initiate all remaining necessary rebalancing transactions until the target state is reached.

[0111] In addition to scaling up (adding capacity) an existing cluster, the same mechanism described above can also be used to rebalance data among existing partitions (and thus existing nodes) within a cluster (e.g., migrating some of the partition data of a node to other nodes with lower load when the node is overloaded).

[0112] At this point, there is no need to add a new partition to the hash ring. Instead, this step is skipped, and the existing partition is shrunk and rebalanced to other partitions.

[0113] The processes of the ShuangzhaoDB database system described above can be implemented through software, hardware, firmware, or any combination thereof. These processes are preferably implemented in one or more computer programs running on a rebalancing coordinator node, which includes components such as a processor, processor-readable storage media (including, for example, volatile and non-volatile memory and / or storage elements), and input / output devices. Each computer program can be represented as a set of instructions (program code) in a code module, residing in the node's random access memory. Before the node needs to invoke this set of instructions, the instructions can be stored in other computer memory (e.g., a hard disk drive, or removable storage such as an optical disc, external hard drive, memory card, flash drive, etc.), or stored on another computer system, and can be downloaded via the Internet or other networks.

[0114] This invention can guarantee data consistency only during the data migration process because when the migration transaction is executed, the source / target partition synchronously records the rollback log. If a node fails or there is network jitter, the transaction will automatically roll back and restart. Moreover, the sequence number can be copied, generating a unique sequence number for the transaction and attaching it to the change log. Log synchronization ensures data consistency between the old and new nodes. It can also provide high availability guarantees. If a single node fails during migration, the migration transactions on that node are rolled back and restarted through a distributed transaction mechanism, without affecting the overall migration process. If the rebalancing coordinator fails, the system immediately elects a new coordinator, which continues to advance the remaining migration tasks based on metadata. If the cluster crashes during migration, the system reproduces the migration process by "snapshot loading + WAL log replay". During replay, the old hash function in the snapshot is used to correctly route transactions, ensuring that the migration can resume from the point of interruption.

[0115] Reference Figure 12 The diagram illustrates an operation flowchart of a data redistribution method for a distributed memory database according to an embodiment of the present invention.

[0116] Specifically, the data redistribution method for the distributed in-memory database can include the following three stages: Phase 1: Initialization phase of rebalancing: Triggering condition: A new node joins the cluster; The operating steps include: Initiate the rebalancing process to clarify the need to redistribute existing data; select a rebalancing control node (i.e., coordinator) to coordinate subsequent migration tasks.

[0117] The second stage, the dynamic rebalancing stage (the core execution phase of this invention): Core objective: To complete the migration of data between the old and new nodes and ensure consistency; The operating steps include: Construct a new hash ring: Combine the logical partitions of the new nodes to generate a consistent hash ring that contains new data distribution rules; Determine the data to be migrated: Compare the old and new hash rings to calculate the data content that needs to be moved across partitions / nodes; Create a migration index: Create a dedicated index for the data to be migrated to accurately mark the migration range; Split migration tasks: Based on network conditions and the business's tolerable latency (such as pause time), the migration task is split into multiple smaller tasks; Execute migration transactions: Distribute small transactions to the corresponding partitions for execution, and ensure the ACID properties of transactions through "single partition serial execution + distributed two-phase commit".

[0118] The third stage, dynamic rebalancing optimization process (finishing and compatibility handling): The operating steps include: Handling WAL (Write-Ahead Log): Ensuring transaction log compatibility during migration (e.g., forwarding misrouted logs). Dynamic balancing across multiple replicas: Synchronously adjusts the distribution of data replicas to ensure load balancing across the multi-replica cluster.

[0119] When scaling up the ShuangzhaoDB database (adding new nodes or utilizing unsaturated nodes), data and related transactions need to be migrated to the new target node. This process must satisfy three core constraints: If a portion of data is in-flight, all transactions that depend on that data must have access to the "complete subset of data"; each row in that subset is "displayed only once" for any given transaction.

[0120] Modifications made to data by committed transactions must be visible to all subsequent transactions.

[0121] During the migration process, transactions need to be automatically routed to the current location of the data (which may be the original node or the new node that has been migrated to); if a routing error occurs (such as the data has been migrated but the transaction still points to the original node), the system needs to reroute in a transparent manner (the whole process does not require user / application intervention and is completely unknown to the outside world).

[0122] The ShuangzhaoDB database system provides execution isolation guarantees for transactions, meaning that when a transaction accesses data through its specified partition key, it ensures that "all necessary data required by the transaction" is visible. This means that even if data from multiple tables and arbitrary row sets may be temporarily stored across multiple partitions during certain periods (such as during expansion and migration), the executing transaction does not need to be aware of this cross-partition situation.

[0123] By employing the technical solution of this invention, data can be divided into smaller, flexibly split / mergeable units (such as sub-partitions based on high cardinality primary keys), enabling automatic adjustment of the partition structure according to workload changes and avoiding the concentration of hot data.

[0124] When dynamically splitting, merging partitions, or migrating data, the impact on foreground transaction processing is minimized by optimizing data transmission strategies and reducing lock contention, thereby ensuring system throughput and low latency and reducing the performance overhead of partition adjustments.

[0125] Optionally, a workload awareness model can be built to monitor metrics such as data volume, access frequency, and response time of each partition in real time, and automatically trigger partition adjustment operations (such as splitting hot partitions and merging idle partitions) without manual intervention, so as to achieve adaptive decision-making for partitions.

[0126] Furthermore, this invention designs a dynamic node access mechanism based on consistent hashing, which automatically detects and participates in data distribution when a new node joins without interrupting transaction processing; during scaling down, it securely migrates the data of nodes to be removed, ensuring that no data is lost and the business is unaware of the changes, thus achieving uninterrupted elastic scaling up / down.

[0127] The invention also introduces a sequence number replication mechanism to generate a unique sequence number for each transaction and attach it to the change log. Through log synchronization, it enables consistent data migration between old and new nodes. It also builds a real-time consistency detection model to monitor data integrity during the rebalancing process, avoid inconsistencies between replicas, and ensure data consistency and ACID properties of transactions during rebalancing.

[0128] Treating rebalancing as a background task reduces resource contention with foreground transactions through dynamic resource allocation (such as limiting migration bandwidth and off-peak processing); designing an incremental data migration strategy that migrates only changed data reduces rebalancing overhead and optimizes rebalancing resource scheduling.

[0129] In addition, it ensures that data rebalancing is automatically triggered and load is allocated on demand during expansion, and that data is efficiently integrated during shrinkage, avoiding resource waste and realizing full automation of the "expansion-rebalancing-shrinkage" process, reducing operation and maintenance costs, and enabling synergy between rebalancing and expansion / shrinkage.

[0130] Reference Figure 13 The diagram shows a schematic diagram of the connection structure between the rebalancing coordinator and each node provided in an embodiment of the present invention.

[0131] In the ShuangzhaoDB database system, data maintenance relies on multiple nodes within a distributed database cluster. Each node contains multiple logical partitions and possesses independent processing, memory, and disk resources. Nodes connect and interact with each other via a computer network. To support system expansion and data rebalancing operations, a rebalancing coordinator and nodes are configured. The rebalancing coordinator executes the data redistribution method for the distributed in-memory database of this invention.

[0132] In this embodiment, the present invention provides a data redistribution method for a distributed in-memory database. Its advantages include: adding expansion nodes to the distributed in-memory database and performing data balancing; selecting target migration data from the data to be migrated based on real-time calculated transmission time; and performing dynamic migration processing based on the target migration data. The present invention determines the amount of data to be migrated each time by pre-determined acceptable service pause duration, allowing data to be divided into smaller, flexibly splittable data for transmission, refining the granularity of the data; and then dynamically adjusting based on the real-time migration status, reducing resource competition with foreground transactions, avoiding resource waste, and improving resource utilization.

[0133] This invention also provides a data redistribution device for a distributed in-memory database, see [link to relevant documentation]. Figure 14 The diagram shows a schematic of the structure of a data redistribution device for a distributed memory database according to an embodiment of the present invention.

[0134] As an example, the data redistribution device for the distributed in-memory database may include: The balancing module 201 is used to add expansion nodes to the distributed memory database and perform data balancing processing on the distributed memory database. The data balancing processing is the process of determining the node partition to which the data to be migrated in the distributed memory database belongs. The filtering module 202 is used to filter target migration data from the data to be migrated based on the real-time calculated transmission time, wherein the transmission time is the duration required to transmit data determined based on a preset service pause duration; The migration module 203 is used to perform dynamic migration processing based on the target migration data. The dynamic migration processing is the process of adjusting the migration frequency of the target migration data during the migration of the data to be migrated from the original node to the expansion node.

[0135] Optionally, the step of adding expansion nodes to the distributed in-memory database and performing data balancing processing on the node cluster within the distributed in-memory database includes: Determine the corresponding number of expansion nodes based on business needs, and add the expansion nodes to the distributed memory database; A node hash ring is generated based on the node cluster of the distributed memory database, and the data to be migrated is determined within the node hash ring. The node cluster includes the expansion node and the original node of the distributed memory database. The data to be migrated is hashed based on the partition key under the node hash ring, and the partition of the node to which the data to be migrated belongs is determined based on the hash calculation result.

[0136] Optionally, the step of filtering target migration data from the data to be migrated based on the real-time calculated transmission time includes: Generate an index table of the data to be migrated and determine the preset service pause duration and target throughput, wherein the preset service pause duration is the upper limit of the delay time of the business transaction set by the user in advance, and the target throughput is the capacity of data to be migrated per unit time. The amount of processing per transaction is determined based on the preset service pause duration and the target throughput. Based on the single transaction volume, data of corresponding capacity is filtered from the data to be migrated to obtain the target migration data.

[0137] Optionally, the dynamic migration processing based on the target migration data includes: The migration frequency value is determined based on the target throughput, and the target migration data is migrated according to the migration frequency value and the index table; Obtain the real-time migration throughput of the target migration data, and determine the deviation between the real-time migration throughput and the target throughput; If the deviation value does not meet the preset deviation threshold, the migration frequency value is adjusted, and the real-time migration throughput is reacquired until the data to be migrated is migrated.

[0138] Optionally, migrating the target migration data based on the target throughput and the index table includes: Based on the index table, accurate requests are sent simultaneously to partitions of two different nodes; After receiving feedback from the two partitions, the target migration data is read from the partition of the first node in a single thread based on the target throughput, and the target migration data is written to the partition of the second node in a single thread, while blocking transactions in the partitions of the first and second nodes.

[0139] Optionally, the blocking of transactions in the partitions of the first node and the second node includes: When reading the target migration data from the partition of the first node, a change log is generated; After the target migration data is written to the partition of the second node, the change log is applied to the partition of the second node, and transactions of the partitions of the first node and the partition of the second node are paused during the application process.

[0140] Optionally, the device further includes: The acquisition module is used to acquire snapshot data after the distributed memory database restarts due to a crash and the stored metadata is determined. The metadata includes the node cluster and the storage information of each node. The snapshot data is a complete data copy with the shortest interval between the time of the distributed memory database crash. The distribution module is used to obtain the historical hash ring from the snapshot data and perform data distribution processing on the metadata according to the historical hash ring.

[0141] Those skilled in the art will understand that, for ease of description and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0142] Furthermore, this application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data redistribution method of the distributed memory database as described in the above embodiments.

[0143] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer-executable program, the computer-executable program being used to cause a computer to execute the data redistribution method for a distributed memory database as described in the above embodiments.

[0144] In the description of the embodiments of the present invention, it should be noted that the terms "above," "below," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. When an element such as a layer, region, or substrate is referred to as being "above" or "on top of" another element, it may be directly on the other element, or there may be an intermediate element. Conversely, when an element is referred to as being "directly on" or "above" another element, there is no intermediate element. It should also be understood that when an element is referred to as being "below" or "under" another element, it may be directly below or under the other element, or there may be an intermediate element. Conversely, when an element is referred to as being "directly below" or "under" another element, there is no intermediate element. Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0145] Those skilled in the art will understand that embodiments of this application may also include computer program products. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), devices, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data redistribution method for a distributed in-memory database, characterized in that, The method includes: Add expansion nodes to the distributed memory database and perform data balancing on the distributed memory database. The data balancing process is the process of determining the node partition to which the data to be migrated in the distributed memory database belongs. The target migration data is selected from the data to be migrated based on the real-time calculated transmission time, wherein the transmission time is determined based on the preset service pause duration to determine the duration required to transmit the data; Dynamic migration processing is performed based on the target migration data, wherein the dynamic migration processing is the process of adjusting the migration frequency of the target migration data during the migration of the data to be migrated from the original node to the expansion node.

2. The data redistribution method for a distributed in-memory database according to claim 1, characterized in that, The process of adding expansion nodes to the distributed in-memory database and performing data balancing on the node cluster within the distributed in-memory database includes: Determine the corresponding number of expansion nodes based on business needs, and add the expansion nodes to the distributed memory database; A node hash ring is generated based on the node cluster of the distributed memory database, and the data to be migrated is determined within the node hash ring. The node cluster includes the expansion node and the original node of the distributed memory database. The data to be migrated is hashed based on the partition key under the node hash ring, and the partition of the node to which the data to be migrated belongs is determined based on the hash calculation result.

3. The data redistribution method for a distributed in-memory database according to claim 1, characterized in that, The step of filtering target migration data from the data to be migrated based on the real-time calculated transmission time includes: Generate an index table of the data to be migrated and determine the preset service pause duration and target throughput, wherein the preset service pause duration is the upper limit of the delay time of the business transaction set by the user in advance, and the target throughput is the capacity of data to be migrated per unit time. The amount of processing per transaction is determined based on the preset service pause duration and the target throughput. Based on the single transaction volume, data of corresponding capacity is filtered from the data to be migrated to obtain the target migration data.

4. The data redistribution method for a distributed in-memory database according to claim 3, characterized in that, The dynamic migration processing based on the target migration data includes: The migration frequency value is determined based on the target throughput, and the target migration data is migrated according to the migration frequency value and the index table; Obtain the real-time migration throughput of the target migration data, and determine the deviation between the real-time migration throughput and the target throughput; If the deviation value does not meet the preset deviation threshold, the migration frequency value is adjusted, and the real-time migration throughput is reacquired until the data to be migrated is migrated.

5. The data redistribution method for a distributed in-memory database according to claim 4, characterized in that, The migration of the target migration data based on the target throughput and the index table includes: Based on the index table, accurate requests are sent simultaneously to partitions of two different nodes; After receiving feedback from the two partitions, the target migration data is read from the partition of the first node in a single thread based on the target throughput, and the target migration data is written to the partition of the second node in a single thread, while blocking transactions in the partitions of the first and second nodes.

6. The data redistribution method for a distributed in-memory database according to claim 5, characterized in that, The transactions that block the partitions of the first node and the partitions of the second node include: When reading the target migration data from the partition of the first node, a change log is generated; After the target migration data is written to the partition of the second node, the change log is applied to the partition of the second node, and transactions of the partitions of the first node and the partition of the second node are paused during the application process.

7. The data redistribution method for a distributed in-memory database according to any one of claims 1-6, characterized in that, The method further includes: After the distributed in-memory database restarts due to a crash and the stored metadata is determined, snapshot data is obtained. The metadata includes the node cluster and the storage information of each node. The snapshot data is a complete data copy with the shortest interval between the time of the distributed in-memory database crash. The historical hash ring is obtained from the snapshot data, and the metadata is distributed according to the historical hash ring.

8. A data redistribution device for a distributed in-memory database, characterized in that, The device includes: The balancing module is used to add expansion nodes to the distributed memory database and perform data balancing processing on the distributed memory database. The data balancing processing is the process of determining the node partition to which the data that needs to be migrated in the distributed memory database belongs. The filtering module is used to filter target migration data from the data to be migrated based on the real-time calculated transmission time, wherein the transmission time is the duration required to transmit the data determined based on the preset service pause duration; The migration module is used to perform dynamic migration processing based on the target migration data. The dynamic migration processing is the process of adjusting the migration frequency of the target migration data during the migration of the data to be migrated from the original node to the expansion node.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the program, it implements the data redistribution method for a distributed memory database as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the data redistribution method for a distributed memory database as described in any one of claims 1-7.

Images