Distributed database processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides a distributed database processing method and device, a storage medium and a program product, and relates to the field of distribution. The method comprises the following steps: determining a query request for a target storage node in N storage nodes; the query request comprises a fragment table name of a target storage node, and any storage node corresponds to one fragment table; replacing a fragment table name in the query request with a main partition name of a target storage node to obtain a target query request; and performing data query based on the target query request. According to the method provided by the invention, the accuracy of the query route is improved, and the data query accuracy and processing efficiency in complex scenes such as dynamic updating of the fragmentation key are improved.

Description

Technical Field

[0001] This application relates to the field of distributed systems, and more particularly to a distributed database processing method, device, storage medium, and program product. Background Technology

[0002] Against the backdrop of the rapid development of big data and cloud computing, distributed databases, due to their high concurrency and high availability characteristics, have become the core support for data-intensive scenarios such as e-commerce and the Internet of Things. Taking e-commerce platforms as an example, the volume of order data often grows exponentially, and single-machine databases face bottlenecks in terms of storage capacity and computing power, making it difficult to meet the real-time processing needs of massive amounts of data.

[0003] In current implementations, distributed databases can address data distribution issues by introducing sharding keys. During table design or initialization, the sharding key (e.g., order ID) is statically determined. The system maps data records to fixed database shards based on rules such as the hash value of this sharding key. Query requests based on this sharding key are then routed to the corresponding target node for execution.

[0004] However, the above implementation method is difficult to effectively achieve dynamic updates of the sharding key. Summary of the Invention

[0005] This application provides a distributed database processing method, device, storage medium, and program product to solve the technical problem of dynamic updating of shard keys.

[0006] Firstly, this application provides a distributed database processing method applied to a distributed database system. The distributed database system includes N storage nodes, each storage node includes N partitions, and the N partitions include one primary partition and N-1 secondary partitions. The primary partition is used to store valid data, and the secondary partitions are used for temporary data migration or synchronization. The primary partition of any storage node subscribes to the binary logs of the partitions with the same name in other storage nodes of the N storage nodes, where N is an integer greater than 1. The method includes:

[0007] Determine the query request for the target storage node among N storage nodes; the query request includes the shard table name of the target storage node, and each storage node corresponds to one shard table;

[0008] Replace the shard table name in the query request with the primary partition name of the target storage node to obtain the target query request;

[0009] Data is retrieved based on the target query request.

[0010] In this embodiment, the database middleware rewrites the table names of SQL requests at the query routing level, ensuring that all query operations are accurately directed to the primary partition within the storage node that stores the currently valid data. This mechanism avoids the complexity of multiple partitions (especially auxiliary partitions used for temporary operations) within the storage node, ensuring that query operations always access the latest and valid version of the data. This improves the accuracy and efficiency of query results in complex distributed data management scenarios (such as when supporting data migration or shard key updates).

[0011] In one possible implementation, the sharded table is obtained by horizontally splitting the global data table according to the sharding key and distributing it to N storage nodes, with each storage node carrying one sharded table.

[0012] A partitioned table is a sub-table obtained by splitting a sharded table into sub-tables according to predetermined rules.

[0013] In this implementation, when data records need to have their sharding key values ​​changed due to business requirements (such as modifying order numbers), the corresponding physical storage location adjustment can be transformed from traditional cross-server migration to migration between different partition tables within the same server. This reduces the impact of data migration, lowers network transmission overhead and operational complexity, and provides a foundation for flexible data management while maintaining high availability in the database.

[0014] In one possible implementation, the sharded table is obtained by splitting the global data table by taking the sharding key modulo a first modulo, where the first modulo is the total number of storage nodes N.

[0015] A partitioned table is obtained by splitting the data records in the partitioned table by taking the partition key modulo the first modulo.

[0016] The primary partition is determined by the partition corresponding to the result of taking the sharding key of the data record modulo the first modulo.

[0017] In this implementation, through the aforementioned two-level explicit modulo operation, a deterministic mapping of data from the global table to shards and then to partitions is achieved, and the primary partition to which each piece of data belongs within its node, serving as the effective data carrier, is clearly defined. This provides a precise computational foundation for subsequent partition-based data migration and management.

[0018] In one possible implementation, the method further includes:

[0019] If the shard key is updated, a data deletion log and a data insertion log will be generated;

[0020] By parsing the data deletion and data insertion logs, data is migrated from the original partition to the target partition;

[0021] Based on the primary partition of the storage node where the data resides, subscribe to the binary logs of the same-named partitions in other storage nodes of N storage nodes, and migrate and synchronize the data to the primary partitions with the same name in other storage nodes.

[0022] In this implementation, the complex data reorganization implied by a single shard key update is transformed into a log stream that can be processed by existing replication mechanisms through the aforementioned processes of generating logs, migrating data, and subscribing to synchronization. This achieves dynamic adjustment of data location while ensuring eventual consistency of data in a distributed environment.

[0023] In one possible implementation, migrating data from the original partition to the target partition includes:

[0024] The original partition is determined based on the value of the sharding key before the update. The original partition is the primary partition before the data migration.

[0025] The target partition is determined based on the updated value of the sharding key, and the target partition and the original partition are located in the same storage node.

[0026] Mark the data to be deleted in the original partition;

[0027] Insert data with updated shard key values ​​into the target partition;

[0028] The deletion and insertion operations generate deletion logs and insertion logs, respectively.

[0029] In this implementation, through the steps described above, the update of the shard key is precisely transformed into a log-driven data reorganization operation within the node, providing an accurate and reliable basis for subsequent log-based synchronization mechanisms.

[0030] In one possible implementation, the method further includes:

[0031] After the partitioning rules of a distributed database system are dynamically adjusted, the mapping relationship between the sharding key and the primary partition name is redefined according to the adjusted partitioning rules.

[0032] Based on the redefined mapping relationship, subsequent received query requests will be processed by replacing the shard table name with the primary partition name.

[0033] This implementation allows the upper-layer query service to perform adaptive routing seamlessly and accurately when the underlying database storage architecture changes dynamically, thus providing elastic scalability while ensuring the continuity and correctness of query services.

[0034] In one possible implementation, the method further includes:

[0035] After the partitioning rules are dynamically adjusted, a new data synchronization group is determined based on the adjusted partitioning structure; the new data synchronization group consists of partitions that have a logical correspondence after the adjustment.

[0036] Update the log subscription relationships between partitions based on the new data synchronization group.

[0037] In this implementation, the system flexibly adjusts the data distribution architecture while simultaneously maintaining the underlying communication links that ensure data consistency, so that the entire distributed database can still provide services as a logically consistent whole during dynamic changes.

[0038] In a second aspect, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0039] The memory stores instructions that the computer executes;

[0040] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0041] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0042] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1 This is a schematic diagram of the distributed database sharding architecture provided in an embodiment of this application;

[0045] Figure 2 A flowchart illustrating a distributed database processing method provided in an embodiment of this application;

[0046] Figure 3 A schematic diagram of a sharded and partitioned two-tier storage architecture provided in an embodiment of this application;

[0047] Figure 4 A schematic diagram of a table based on sharding rules of sharding keys in a distributed database provided in an embodiment of this application;

[0048] Figure 5 A table diagram illustrating the partitioning rules and data distribution logic provided in the embodiments of this application.

[0049] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0051] It should be noted that the distributed database processing method, device, storage medium, and program product provided in this application can be used in the distributed field, or in any field other than distributed. This application does not limit the application field of the distributed database processing method, device, storage medium, and program product.

[0052] The specific application scenarios of this application cover business areas that require dynamic adjustment of data sharding logic, including but not limited to the following areas:

[0053] E-commerce transaction systems: In the core databases of e-commerce platforms, such as orders and users, business identifiers (e.g., order IDs, user IDs) are often set as sharding keys. When business rules change (e.g., order merging, user account system upgrades), the system can directly update these identifiers online without complex data migration or system downtime, ensuring business continuity and data accuracy in high-concurrency scenarios such as promotions and after-sales service.

[0054] IoT Data Platform: In IoT scenarios, massive amounts of data are stored using device IDs as the sharding key. When devices are replaced, maintained, or reorganized in batches, the system supports efficient online replacement of device IDs and adjustment of data ownership while maintaining the integrity of historical data and the continuity of queries.

[0055] Financial and Risk Control Systems: In payment, account, and risk control systems, key business identifiers such as customer numbers and transaction serial numbers also serve as sharding keys. When customer information is merged, transaction traceability is corrected, or regulatory compliance changes occur, the system can securely and consistently update the key values, ensuring the consistency of financial data and uninterrupted real-time risk monitoring.

[0056] For the above application scenarios, distributed databases currently commonly adopt methods such as... Figure 1The diagram shows a three-tier sharding architecture: "client—database middleware (compute node)—storage node". The following section will combine this with... Figure 1 This paper introduces how distributed databases implement the above application scenarios and the logic of this architecture. Figure 1 This is a schematic diagram of the distributed database sharding architecture provided in an embodiment of this application. Figure 1 As shown, the workflow of this architecture is as follows:

[0057] First, the client, acting as the request initiator, sends an SQL operation request to the database middleware (or compute node).

[0058] The database middleware is the core routing component of this architecture, and it has pre-defined sharding rules (such as hash modulo based on order ID). When it receives an SQL operation request, the middleware extracts the sharding key value from the SQL and calculates which specific storage node the data should be located on according to the pre-defined rules.

[0059] Storage nodes are the actual units that carry data. For example... Figure 1 As shown, a logical data table is horizontally partitioned into multiple shards (shard 1 and shard 2 as shown in the figure), and each shard is allocated and stored on an independent storage node. For example, shard 1 is stored on storage node 1, and shard 2 is stored on storage node 2.

[0060] Specifically, if the user ID is used as the sharding key and the hash rule "%4" is adopted, then the data with user ID 1001 (1001%4=1) will be stored on storage node 1. Thereafter, all queries for user ID=1001 will be routed by the middleware to storage node 1 for execution.

[0061] However, existing distributed database architectures based on shard keys have inherent technical limitations when facing business requirements for dynamically updating shard keys, specifically:

[0062] First, shard keys themselves cannot be directly updated. In the current architecture, the value of the shard key determines the physical storage location of the record when data is written. If the shard key is updated directly, the theoretical storage location (target shard) of the record under the new key value will change. Due to the lack of a built-in atomic processing mechanism, such an update will disrupt the original data distribution logic, and the system cannot automatically and safely relocate the data, which may lead to data query failures or the risk of data inconsistency.

[0063] Secondly, implementing updates relies on costly indirect data migration. To address these issues, existing methods involve migrating data from the original shard to the new shard using external means (such as manual operation or customized scripts). This process is not only complex and time-consuming, requiring meticulous process control, but also often impacts the availability of relevant data during migration, disrupting business continuity.

[0064] Furthermore, the migration process itself introduces performance and consistency issues. Cross-node data migration operations consume significant network and I / O resources. In a distributed environment, atomicity is difficult to guarantee during migration. If a network failure or node malfunction occurs during migration, the system is prone to data inconsistency, and there is a lack of effective automatic rollback or compensation mechanisms to repair such inconsistencies.

[0065] Finally, query routing faces challenges during updates. Even after the migration is complete, the database middleware may fail to immediately route query requests for new key values ​​correctly to the new target shard due to static routing rules or caching. This can lead to query failures or be forced to degrade to inefficient full table scans, reducing query performance and impacting user experience.

[0066] The distributed database processing method provided in this application aims to solve the aforementioned technical problems of existing technologies. It achieves dynamic updates of shard keys through a collaborative mechanism combining intra-shard partitioning and log subscription. Specifically, each shard is divided into a primary partition and a secondary partition, forming a two-layer management structure. When the shard key of a piece of data needs to be updated, the system first modifies the key value in the secondary partition of the original shard, then determines the target shard based on the new key value, and synchronizes the updated record to the target location by subscribing to the binary log of the primary partition of the target shard. Simultaneously, the database middleware dynamically rewrites the query SQL to ensure that all requests are accurately routed to the primary partition where the currently valid data resides, thereby achieving atomic updates of shard keys and ensuring data consistency without the need for cross-node data migration.

[0067] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0068] First, combine Figure 2 The distributed database processing method provided in the embodiments of this application will be described. Figure 2 This is a flowchart illustrating a distributed database processing method provided in an embodiment of this application. The execution entity in this embodiment can be a computing node or query middleware in a distributed database system. Figure 2 As shown, the method includes:

[0069] S201. Determine the query request for the target storage node among the N storage nodes.

[0070] Specifically, client applications (such as order query services on e-commerce platforms) initiate data queries, generating Structured Query Language (SQL) requests. These SQL requests typically include logical table names (i.e., sharded table names) and query conditions. Database middleware or compute nodes receive these requests and, based on preset routing rules and the sharding key values ​​in the SQL request, determine the target storage node corresponding to the request. N storage nodes constitute the storage layer of the distributed database, where N is an integer greater than 1, and each storage node is responsible for hosting a shard of the global data table.

[0071] S202. Replace the shard table name in the query request with the primary partition name of the target storage node to obtain the target query request.

[0072] After identifying the target storage node, the middleware further rewrites the original SQL request according to the partition mapping rules within that storage node. Each storage node further divides its sharded table into N partitions. These N partitions include one primary partition and N-1 secondary partitions. The primary partition is used for normal storage and providing effective data services, while the secondary partitions are mainly used for temporary operations such as data migration and synchronization.

[0073] The core step in this process is to replace the logical shard table name referenced in the original SQL request with the physical name of the specific primary partition within the target storage node. For example, if the original request is `SELECT * FROM t_order WHERE order_id=1001`, and the routing determines the target node to be node 1 with the corresponding primary partition physical table name `node1_primary_partition_0`, then the rewritten target query request would be `SELECT * FROM node1_primary_partition_0 WHERE order_id=1001`.

[0074] S203. Perform data query based on the target query request.

[0075] The middleware sends the rewritten target query request to the target storage node. Upon receiving the request, the target storage node, because it directly points to a specific primary partition, does not require additional internal routing or partition selection and can directly execute the query operation on the specified primary partition, quickly returning the query results. After receiving the results, the middleware can return them to the client application.

[0076] The distributed database processing method provided in this embodiment uses database middleware to rewrite table names in SQL requests at the query routing level, ensuring that all query operations are accurately directed to the primary partition within the storage node that stores the currently valid data. This mechanism avoids the complexity of multiple partitions (especially auxiliary partitions used for temporary operations) within the storage node, ensuring that query operations always access the latest and valid version of the data. This improves the accuracy and efficiency of query results in complex distributed data management scenarios (such as when supporting data migration or shard key updates).

[0077] The following combination Figure 3 The distributed database system architecture on which the embodiments of this application are based will be described. Figure 3 This is a schematic diagram of a sharded and partitioned two-layer storage architecture provided in an embodiment of this application.

[0078] like Figure 3 As shown, the distributed database system of this application, based on the traditional sharding architecture, introduces the concept of partitioning, forming a two-layer data management structure to achieve more refined data operation and management capabilities. This architecture specifically includes the following layers:

[0079] Client and Database Middleware Layer: Client applications access the database through the database middleware (or compute nodes). The middleware receives SQL requests from clients and undertakes the core responsibilities of routing and distribution.

[0080] Storage nodes and sharding layer: Data is physically distributed across multiple independent storage nodes, such as... Figure 3 The diagram shows storage node 1 and storage node 2. Each storage node carries a subset of data from the global logical table; this subset is called a shard. For example, shard 1 and shard 2. This layer of partitioning solves the bottlenecks of single-machine databases in terms of storage space, computing power, and memory resources, enabling horizontal scaling of the system.

[0081] Within a shard, the shards are further divided into multiple partitions. For example, shard 1 is split into partition 1 and partition 2.

[0082] A partition is a sub-table within a shard, and it is the smallest logical unit for data management and operation. Partitioning can be based on various rules, such as modulo of the shard key, numerical range, or time range. Each shard contains one primary partition and at least one secondary partition. The primary partition is the effective data carrier providing data read and write services under normal circumstances; the secondary partition is used to perform temporary or background tasks such as data migration, synchronization, and backup, achieving isolation between data operations and online services.

[0083] This two-tier architecture transforms complex data operations such as shard key updates from traditional, high-cost cross-node data migration to more efficient intra-node partition-level data reorganization and synchronization. This effectively reduces operational complexity and system overhead, providing a feasible architectural foundation for dynamic shard key updates while ensuring data consistency and service availability.

[0084] Optionally, this embodiment describes a data organization method for a distributed database, the core of which lies in using a two-layer logic of sharding and partitioning to split and manage data, the specific logic of which is as follows.

[0085] 1. Data sharding (first-level splitting)

[0086] For a large logical table in a database (e.g., the order table t_order), a sharding key (e.g., order number order_id) is first selected. Based on pre-configured sharding rules (e.g., hashing the sharding key and then taking the modulo), the large logical table is horizontally split into multiple data subsets. Each data subset is called a shard. Each shard is deployed to a separate physical database server, called a storage node. The sharded data hosted on each storage node is logically represented as a sharded table. For example, using the rule order_id%4, the t_order table can be split into 4 shards, stored on 4 servers, forming sharded tables such as server1.t_order_shard, server2.t_order_shard, etc.

[0087] 2. Data partitioning (second-level splitting)

[0088] Within each storage node, the sharded table it hosts is further refined and split. This splitting is based on partitioning rules. For example, a second modulo operation might be performed on the same `order_id`, or the table might be divided into ranges based on the month the order was created. This rule splits a sharded table into multiple smaller data units, each called a partition. Each partition physically corresponds to an independent sub-table, called a partitioned table. For example, on server1, the `t_order_shard` table can be further split into four partitioned tables using the `order_id%4` rule: `t_order_shard_p0`, `t_order_shard_p1`, ..., `t_order_shard_p3`.

[0089] Through this two-level splitting, data is finely distributed across multiple layers. Within each shard (i.e., each storage node), the partition table becomes the smallest unit of operation for data management and migration. The system dynamically designates one partition table as the primary partition to serve currently online data read and write requests; the remaining partition tables serve as secondary partitions, primarily used for background data reorganization, migration, or backup operations.

[0090] Using the above implementation method, when data records need to change their sharding key values ​​due to business requirements (such as modifying order numbers), the corresponding physical storage location adjustment can be transformed from traditional cross-server migration to migration between different partition tables within the same server. This reduces the scope of data migration, lowers network transmission overhead and operational complexity, and provides a foundation for flexible data management while maintaining high availability in the database.

[0091] The following uses the table tbsharding in a distributed database as an example, combined with... Figure 4 and Figure 5 The sharding and partitioning two-layer architecture and data operation logic described in the above embodiments will be explained in detail. Figure 4 This is a schematic diagram of a table based on sharding rules in a distributed database, provided in an embodiment of this application. Figure 5 A table diagram illustrating the partitioning rules and data distribution logic provided in the embodiments of this application.

[0092] like Figure 4 The image shows the sharding rules for the table tbsharding based on the id field. Figure 4 The core logic is to allocate data shards based on the modulo operation of the ID. The system routes data records to four different storage nodes, db1 to db4 as shown in the diagram, based on the result of taking the ID field modulo 4. For example, a record with ID=12 will be stored in the db1.tbsharding shard because 12%4=0. This sharding layer solves the bottleneck of single-machine storage and distributes data evenly across different storage nodes.

[0093] like Figure 5 As shown, based on the sharding, the partitioning rules and data distribution details within each shard are further displayed. Figure 5 The following details and operational logic of the two-tier architecture are presented:

[0094] From a partitioning perspective, each shard (e.g., db1.tbsharding) is further divided into four partition tables, with the suffixes _0 to _3 corresponding to the results of id%4. For example, db1.tbsharding_0 stores data where id%4=0.

[0095] When no sharding key update occurs, each shard contains only one partition table that actually stores data; this partition is called the primary partition. For example, for db1, its primary partition is db1.tbsharding_0, which only stores data where id%4=0; db1.tbsharding_1 to db1.tbsharding_3 within the same shard are empty secondary partitions.

[0096] When the shard key `id` is updated, data is migrated between different partition tables within the same shard, rather than across shards. For example, if `id=12` (originally in `db1.tbsharding_0`) is updated to `id=10` (10%4=2), the data will be migrated from the primary partition `db1.tbsharding_0` to the secondary partition `db1.tbsharding_2`. This process is accomplished by generating DELETE logs (from the original partition) and INSERT logs (to the target partition), rather than a single UPDATE log.

[0097] Furthermore, to ensure data consistency, the primary partition subscribes to the binary logs of all other partition tables with the same name in all other shards. For example, db1.tbsharding_0 will subscribe to the logs of db2.tbsharding_0, db3.tbsharding_0, and db4.tbsharding_0. When there are data changes in the _0 partition of any node, the change log will be synchronized to all subscribers.

[0098] Furthermore, during the SQL query rewriting process, the database middleware (compute node) dynamically rewrites the table names in the logical SQL requests issued by the application to the physical table names of the primary partitions on the node where the target data resides. For example, for the query `SELECT * FROM tbsharding WHERE id=12`, the middleware calculates that `id=12` is located on the `db1` node and rewrites it as `SELECT * FROM db1.tbsharding_0 WHERE id=12`, ensuring that the query directly accesses the primary partition where the valid data resides.

[0099] The above examples fully demonstrate how a two-tier architecture of sharding and partitioning can transform shard key update operations that may trigger cross-node migration into data reorganization within a node's partition range. By combining log subscription and SQL rewriting techniques, dynamic data management can be achieved while ensuring system consistency, availability, and query efficiency.

[0100] Optionally, the predefined rules for sharding and partitioning can be implemented using modulo operations. The following will explain the method for splitting and determining primary partitions based on modulo rules.

[0101] 1. Rules for generating sharded tables

[0102] The system takes the sharding key of the global data table and performs a modulo operation on a first modulo (this value could be, for example, the total number of storage nodes N). Based on the result, the data records are distributed to the corresponding storage nodes, thus generating a sharded table. For example, when N=4, the records with remainders of 0, 1, 2, and 3 when the sharding key is modulo 4 are stored in nodes 1 to 4 respectively, forming four sharded tables.

[0103] 2. Rules for generating partition tables

[0104] Within each storage node, for data records already allocated to that shard table, the shard key is taken again, and a modulo operation is performed on the first modulo N. Based on the result of this operation, the data records are mapped to the corresponding partitions within that node, thus logically dividing the shard table into N partition tables. For example, when N=4, records with the same remainder when the shard key is modulo 4 are assigned to the same partition table.

[0105] 3. Rules for determining the primary partition

[0106] For any given data record, its primary partition is uniquely determined by the following rule:

[0107] The partition corresponding to the sharding key of the data record modulo the first modulus N is the primary partition of the data record in the current storage node.

[0108] For example, a data record with a shard key value of 25 is stored on the node determined by taking the modulo of its shard key with respect to N. Within that node, 25%N is calculated. If N=4, the result is 1. Therefore, partition number 1 within that node is the primary partition for this data record, used for normal storage and providing access services to the data.

[0109] Through the aforementioned two-level explicit modulo operation, a deterministic mapping of data from the global table to shards and then to partitions is achieved, and the primary partition to which each piece of data belongs within its node, serving as the effective data carrier, is clearly defined. This provides a precise computational foundation for subsequent data migration and management based on partition granularity.

[0110] Optionally, when the distributed database system detects an update to the shard key value of a data record, it may trigger the following processing flow, for example:

[0111] 1. Generate change log

[0112] The system will not directly perform an in-place update (UPDATE), but will instead break down this key-value change into two independent operations:

[0113] Generate a data deletion log entry for the data record in the original partition where the data is currently located.

[0114] In the target partition where the data will be migrated, generate a data insertion log containing the new shard key value.

[0115] 2. Perform data migration

[0116] The data migration component (which can be located in the database kernel or middleware) parses the deletion and insertion logs generated above. Based on the information in the logs, it performs physical data movement operations, removing the target data record from the corresponding original partition and writing it to the target partition.

[0117] 3. Achieve cross-node data synchronization

[0118] After the data migration operation is complete, the system needs to ensure that this change is synchronized to other storage nodes to maintain global data consistency. This synchronization process relies on the following log subscription architecture:

[0119] The primary partition of the storage node where the data migration operation is performed has been pre-configured to subscribe to the binary logs of the same partitions in the other N-1 storage nodes.

[0120] The data insertion logs generated during this migration will be automatically synchronized to the corresponding primary partitions with the same name on all other storage nodes through this subscription link.

[0121] Similarly, data deletion logs generated from the original partition will also be synchronized through the subscription relationship network to which the original partition belongs.

[0122] For example, refer to Figure 5 For example, if id=12 (originally in db1.tbsharding_0) is updated to id=10, the system will generate a deletion log in db1.tbsharding_0 and an insertion log in db1.tbsharding_2, completing the data migration. Subsequently, the insertion log generated in db1.tbsharding_2 (as the new primary partition) will be synchronized to db2.tbsharding_2, db3.tbsharding_2, and db4.tbsharding_2 through their subscription relationships; while the deletion log generated in db1.tbsharding_0 will be synchronized to the _0 partition of other nodes.

[0123] Through the above process of generating logs, migrating data, and subscribing to synchronization, the complex data reorganization implied by a single shard key update is transformed into a log stream that can be processed by existing replication mechanisms. This ensures eventual consistency of data in a distributed environment while enabling dynamic adjustment of data location.

[0124] Optionally, once the shard key update request is confirmed, the system may perform data migration within the storage node, for example, by following these steps:

[0125] 1. Determine the source and destination of the migration.

[0126] The system first calculates which partition the data should currently reside in based on the value of the shard key before the update, combined with established partitioning rules (such as modulo algorithm). This partition is the original partition for this migration and also the primary partition where the data was located before the update.

[0127] Subsequently, based on the updated shard key value and the same partitioning rules, the system calculates the partition to which the data should belong under the new key value. This partition is the target partition. Crucially, this calculation is performed within the same storage node, ensuring that the target partition and the original partition reside on the same physical node, thus limiting the migration scope to within that node.

[0128] 2. Perform atomic data migration operations

[0129] After identifying the source and target, the system executes two consecutive operations under the protection of database transactions:

[0130] First, the system performs a deletion operation in the original partition, removing the data record. This operation generates a corresponding deletion log, recording key information such as which data was removed from which partition.

[0131] Next, the system inserts the complete data record carrying the new shard key value into the target partition. This operation generates a corresponding insert log, recording key information about which data was inserted into which partition.

[0132] For example, combining Figure 5 For example, when the id is updated from 12 to 10:

[0133] Based on the old value 12 (12%4=0), the original partition (i.e. the original primary partition) is determined to be db1.tbsharding_0.

[0134] Based on the new value 10 (10%4=2), the target partition is determined to be db1.tbsharding_2 (which belongs to the same db1 node as the original partition).

[0135] Delete the record with id=12 in db1.tbsharding_0 and generate a deletion log.

[0136] Insert a record with id=10 (content has been updated) into db1.tbsharding_2 and generate an insertion log.

[0137] Through the steps described in this embodiment, the update of the shard key is precisely transformed into an intra-node, log-driven data reorganization operation, providing an accurate and reliable basis for subsequent log-based synchronization mechanisms.

[0138] Optionally, the distributed database system supports online dynamic adjustment of partitioning rules, such as increasing or decreasing the number of partitions, or changing the partitioning algorithm. After such adjustments are completed, the system can ensure the correctness of query routing, for example, by following the process:

[0139] 1. Update route mappings

[0140] When partitioning rules are dynamically adjusted, the system first recalculates the correspondence between shard keys and physical storage locations for the entire dataset or the affected area, based on the new rules. The core logic is to redetermine the physical name of the primary partition corresponding to each valid shard key value, thereby establishing a new and accurate routing mapping table between shard keys and primary partition names.

[0141] This process may involve updating metadata. For example, if the number of partitions N is changed from 4 to 8, the result of modulo 8 on the sharding key of all data records needs to be recalculated and mapped to the new partition names, such as tbsharding_0 to tbsharding_7.

[0142] 2. Apply the new rules to query routing

[0143] After the routing mapping is updated, the database middleware will use the new mapping to perform the table name rewriting step for all subsequent received query requests. Specifically:

[0144] The middleware parses the query request and extracts the sharding key value.

[0145] Based on the newly established mapping relationship between shard keys and primary partition names, find the physical name of the primary partition that currently corresponds to the shard key value.

[0146] Replace the logical shard table name in the query request with the name of the new primary partition found, forming the target query request, and route it to the correct storage node for execution.

[0147] For example, suppose the original partitioning rule is id%4, corresponding to partitions _0 to _3. After dynamic adjustment, the new rule becomes id%8, corresponding to partitions _0 to _7. For a query with id=10:

[0148] Before the adjustment, based on 10%4=2, the route was routed to the primary partition tbsharding_2.

[0149] After the adjustment, the system updates the mapping relationship. The new mapping result for id=10 is 10%8=2. However, at this time, 2 represents tbsharding_2 in the new partition set, and its data range is different from the original tbsharding_2.

[0150] When a subsequent query arrives, the middleware uses the new mapping to correctly route the query id=10 to the new tbsharding_2 primary partition.

[0151] By implementing the above methods, the upper-layer query service can perform adaptive routing seamlessly and accurately when the underlying storage architecture of the database changes dynamically, thereby providing elastic scalability while ensuring the continuity and correctness of query services.

[0152] Optionally, when the partitioning rules are dynamically adjusted (e.g., increasing or decreasing the number of partitions, or changing the partitioning algorithm) resulting in changes to the physical structure of the partitions, the system must simultaneously update the log synchronization mechanism to ensure data consistency while updating the query routing mapping. The specific process is as follows:

[0153] 1. Determine the new data synchronization group

[0154] Based on the adjusted partition structure, the system redefines the logical units for data synchronization. The new data synchronization groups consist of partitions with logical correspondences across all storage nodes. These logical correspondences are typically determined based on the partitions' shared functional roles or data mapping ranges under the new rule system.

[0155] For example, suppose the partitioning rule is changed from id%4 to id%8. Before the adjustment, the partition named _0 (e.g., db1.tbsharding_0) stores all data where id%4=0. After the adjustment, the data originally stored in the _0 partition may be repartitioned into new _0 and _4 partitions (because when id%8=0 or 4, id%4 is equal to 0). Therefore, the new data synchronization group may no longer be a simple set of partitions with the same name. Instead, based on the data mapping relationship, the new _0 partition and the _4 partition need to be grouped together into a logical synchronization group to ensure that the original data range that needs to be synchronized is still covered after the adjustment.

[0156] 2. Update log subscription relationship

[0157] After the new data synchronization group composition is defined, the system updates the binary log subscription relationships between each partition. Each partition will reconfigure the list of source partitions from which it needs to subscribe to logs, and which other partitions its logs need to be subscribed to, based on its new data synchronization group.

[0158] Continuing with the example above, under the adjusted new architecture, the new partition tbsharding_0 on storage node db1 may need to simultaneously subscribe to the logs of the new tbsharding_0 and tbsharding_4 partitions on other nodes (db2, db3, db4), because they share the data range of the original _0 synchronization group. Conversely, the corresponding partitions on other nodes also need to establish subscriptions to the new _0 and _4 partitions on db1.

[0159] Through the above implementation, the system can flexibly adjust the data distribution architecture while simultaneously maintaining the underlying communication links that ensure data consistency, so that the entire distributed database can still provide services as a logically consistent whole during dynamic changes.

[0160] The electronic device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0161] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.

[0162] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.

[0163] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0164] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0167] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0168] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0169] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0170] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0171] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0172] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0173] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0174] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A distributed database processing method, characterized in that, This method is applied to a distributed database system, which includes N storage nodes, each storage node including N partitions, of which there is one primary partition and N-1 secondary partitions. The primary partition is used to store valid data, and the secondary partitions are used for temporary data migration or synchronization. The primary partition of any storage node subscribes to the binary logs of partitions with the same name on other storage nodes of the N storage nodes, where N is an integer greater than 1. The method includes: Determine the query request for the target storage node among the N storage nodes; the query request includes the shard table name of the target storage node, and each storage node corresponds to one shard table; Replace the shard table name in the query request with the primary partition name of the target storage node to obtain the target query request; Perform data query based on the target query request.

2. The method according to claim 1, characterized in that, The sharded table is obtained by horizontally splitting the global data table according to the sharding key and distributing it to the N storage nodes, with each storage node carrying one sharded table; A partitioned table is a sub-table obtained by splitting the sharded table into sub-tables according to predetermined rules.

3. The method according to claim 2, characterized in that, The sharding table is obtained by splitting the global data table by taking the sharding key modulo a first modulo, where the first modulo is the total number N of the storage nodes; The partition table is obtained by splitting the data records in the shard table by taking the shard key modulo the first modulo; The primary partition is determined by the partition corresponding to the result of taking the modulo of the first modulus from the sharding key of the data record.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: If the shard key is updated, generate a data deletion log and a data insertion log; By parsing the data deletion log and the data insertion log, the data is migrated from the original partition to the target partition; Based on the primary partition of the storage node where the data resides, subscribe to the binary logs of the same-named partitions in other storage nodes of the N storage nodes, and migrate and synchronize the data to the primary partitions with the same name in the other storage nodes.

5. The method according to claim 4, characterized in that, The process of migrating data from the original partition to the target partition includes: The original partition is determined based on the value of the sharding key before the update, and the original partition is the primary partition before the data migration. The target partition is determined based on the updated value of the sharding key, and the target partition and the original partition are located in the same storage node. The data in the original partition is marked for deletion; Insert the data with the updated shard key value into the target partition; The deletion and insertion operations generate the deletion log and the insertion log, respectively.

6. The method according to any one of claims 1-3, characterized in that, The method further includes: After the partitioning rules of the distributed database system are dynamically adjusted, the mapping relationship between the shard key and the primary partition name is re-determined according to the adjusted partitioning rules. Based on the redefined mapping relationship, the step of replacing the shard table name with the primary partition name is executed for subsequent received query requests.

7. The method according to claim 6, characterized in that, The method further includes: After the partitioning rules are dynamically adjusted, a new data synchronization group is determined based on the adjusted partitioning structure; the new data synchronization group consists of partitions that have a logical correspondence after the adjustment. Based on the new data synchronization group, update the log subscription relationships between each partition.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images