Distributed database processing method and device, equipment, storage medium and program product

Abstract

The invention provides a distributed database processing method and device, equipment, a storage medium and a program product, and relates to the technical field of distributed databases. The method comprises the steps that a capacity adjusting command from an application program is received, the capacity adjusting command is used for indicating capacity expansion or capacity reduction, and the capacity adjusting command comprises a second logic table; wherein the N storage nodes correspond to N1 first fragment tables at the moment of receiving the capacity adjustment command, and N1 is greater than or equal to N; the target data table is calculated on the basis of a second fragmentation key and a second fragmentation rule, M1 second fragmentation tables are obtained, the M1 second fragmentation tables correspond to the M storage nodes, M1 is larger than or equal to M, N storage nodes do not coincide with the M storage nodes, and N, N1, M and M1 are all integers larger than 1; and executing capacity adjustment based on the M1 second fragment tables. The method is used for achieving the effect of improving the processing flexibility of the distributed database.

Description

Technical Field

[0001] This application relates to the field of distributed database technology, and in particular to a distributed database processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] In industries such as finance and telecommunications, which have extremely high requirements for concurrency and reliability, distributed databases need to adjust the number of storage nodes to match resource demands as business scales rapidly change, requiring critical scaling capabilities. Therefore, how to handle distributed databases is of paramount importance.

[0003] Currently, existing distributed database processing methods mainly rely on preset sharding keys and sharding rules to achieve data redistribution. This method requires testers to predefine sharding keys and sharding rules during database initialization, calculate the range of data to be migrated based on the preset sharding rules, and copy the data from the original storage node to the new node, thus achieving database scaling.

[0004] However, existing distributed database processing methods rely on fixed, predefined rules, resulting in poor flexibility. Summary of the Invention

[0005] This application provides a distributed database processing method, apparatus, device, storage medium, and program product to improve the flexibility of distributed database processing.

[0006] In a first aspect, this application provides a distributed database processing method applied to computing nodes in the distributed database, wherein the distributed database further includes multiple storage nodes, and one storage node corresponds to one or more sharded tables, the method comprising:

[0007] The system receives a capacity adjustment command from an application, which indicates whether to expand or shrink the capacity. The capacity adjustment command includes a second logical table. At the moment the capacity adjustment command is received, there are N1 first shard tables corresponding to the N storage nodes. The N1 first shard tables are calculated from the first logical table based on the first shard key and the first shard rule, and N1 is greater than or equal to N.

[0008] The target data table is calculated based on the second sharding key and the second sharding rule to obtain M1 second sharding tables. The M1 second sharding tables correspond to M storage nodes. M1 is greater than or equal to M. N storage nodes do not overlap with M storage nodes. N, N1, M, and M1 are all integers greater than 1.

[0009] Capacity adjustment is performed based on the M1 second shard tables;

[0010] Before the capacity adjustment is completed, database services are provided based on the N storage nodes;

[0011] After the capacity adjustment is completed, database services are provided based on the M storage nodes.

[0012] In one possible implementation, the capacity adjustment based on the M1 second shard tables includes: writing existing data from the N storage nodes to the M storage nodes based on the second shard tables; and reading new data during the capacity adjustment process based on the first shard table, and writing the new data to the M storage nodes based on the second shard tables.

[0013] In one possible implementation, after performing capacity adjustment based on the M1 second shard tables, the method further includes: pausing write operations on the distributed database and switching the effective logical table to the second logical table; after the switching is completed, resuming write operations on the distributed database to complete the capacity adjustment.

[0014] In one possible implementation, the logic table for which the switch takes effect is the second logic table, including: determining the switch timing based on historical request logs; and the logic table for which the switch takes effect at the switch timing is the second logic table.

[0015] In one possible implementation, the method further includes: obtaining a first sharding key, a first logical table, a second sharding key, and a first data distribution feature and a second data distribution feature corresponding to the second logical table; and determining a first sharding rule and a second sharding rule based on the first data distribution feature and the second data distribution feature through a sharding rule engine.

[0016] In one possible implementation, the method further includes: collecting load indicators of the M storage nodes; if it is confirmed that the M storage nodes are overloaded based on the load indicators, automatically generating a new capacity expansion adjustment instruction, and performing capacity adjustment based on the new capacity expansion adjustment instruction.

[0017] Secondly, this application provides a database stability testing device, comprising:

[0018] A distributed database processing device is applied to computing nodes in the distributed database, which further includes multiple storage nodes, each storage node corresponding to one or more sharded tables, including:

[0019] A receiving module is configured to receive a capacity adjustment command from an application, the capacity adjustment command being used to indicate expansion or reduction, the capacity adjustment command including a second logical table; wherein, at the moment the capacity adjustment command is received, there are N1 first shard tables corresponding to the N storage nodes, the N1 first shard tables being calculated from the first logical table based on the first shard key and the first shard rule, and N1 being greater than or equal to N;

[0020] The calculation module is used to calculate the target data table based on the second sharding key and the second sharding rule to obtain M1 second sharding tables. The M1 second sharding tables correspond to M storage nodes. N storage nodes do not overlap with M storage nodes. N, N1, M and M1 are all integers greater than 1.

[0021] The adjustment module is used to perform capacity adjustment based on the M1 second shard tables;

[0022] Before the capacity adjustment is completed, database services are provided based on the N storage nodes;

[0023] After the capacity adjustment is completed, database services are provided based on the M storage nodes.

[0024] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0025] The memory stores computer-executed instructions;

[0026] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method described in any of the first aspects above.

[0027] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any of the first aspects above.

[0028] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0029] This application provides a distributed database processing method, apparatus, device, storage medium, and program product. It receives a capacity adjustment command from an application program, indicating whether to expand or shrink the database. The capacity adjustment command includes a second logical table, laying the foundation for subsequent capacity adjustment. Upon receiving the capacity adjustment command, there are N1 first shard tables corresponding to N storage nodes. These N1 first shard tables are calculated from the first logical tables based on a first shard key and a first shard rule. Based on a second shard key and a second shard rule, M1 second shard tables are obtained from the target data table, improving the dynamic expansion and contraction capability of the distributed database. These M1 second shard tables correspond to M storage nodes, and the N and M storage nodes do not overlap. Both N and M are integers greater than 1. Capacity adjustment is performed based on these M1 second shard tables, improving the flexibility of distributed database processing. Before the capacity adjustment is completed, database services are provided based on the N storage nodes; after the capacity adjustment is completed, database services are provided based on the M storage nodes. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0031] Figure 1 A flowchart illustrating a distributed database processing method provided in this application;

[0032] Figure 2 A schematic diagram of an overall architecture for scaling up and down provided in this application;

[0033] Figure 3 A flowchart illustrating a method for creating a new sharding table based on a new sharding key and sharding rules, as provided in this application;

[0034] Figure 4 A flowchart illustrating a method for redistributing existing data provided in this application;

[0035] Figure 5 A flowchart illustrating an incremental data redistribution method provided in this application;

[0036] Figure 6 A flowchart illustrating a method for switching metadata provided in this application;

[0037] Figure 7 This application provides a schematic diagram of the structure of a distributed database processing device.

[0038] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application.

[0039] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0041] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0043] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0044] First, let's explain some of the terms used in this application:

[0045] Compute node: refers to the node in a distributed database that is directly connected to the client and provides services.

[0046] Storage node: refers to the node used for data storage in a distributed database. It does not directly provide services to clients, but is scheduled by the computing node.

[0047] Remote repository: refers to a component used to store configuration data and detect data changes.

[0048] In industries such as finance and telecommunications, which have extremely high requirements for concurrency and reliability, distributed databases need to adjust the number of storage nodes to match resource demands as business scales rapidly change, requiring critical scaling capabilities. Therefore, how to handle distributed databases is of paramount importance.

[0049] Currently, existing distributed database processing methods primarily rely on predefined sharding keys and rules to redistribute data. This method requires testers to predefine sharding keys and rules during database initialization and configure them in the metadata. When the database needs to be scaled up or down, the range of data to be migrated is calculated based on the predefined sharding rules, and the data is copied from the original storage node to the new node.

[0050] However, existing distributed database processing methods rely on manually predefined fixed rules, resulting in poor flexibility. Considering these problems, this application proposes a method that eliminates the need for manually predefined fixed sharding keys and rules. Through metadata management and dynamic metadata switching, the sharding keys and rules are adjusted during the dynamic scaling of the distributed database, improving the flexibility of distributed database processing.

[0051] The executing entity of this distributed database processing method can be, for example, a processing system. Optionally, the processing system can be any existing electronic device with processing capabilities, such as a terminal or a server. In some embodiments, the processing system can also be deployed in a server cluster or a cloud environment. This application does not limit the deployment environment of the processing system.

[0052] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments.

[0053] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0054] Figure 1This is a flowchart illustrating a distributed database processing method provided in this application. Figure 1 As shown, this method is applied to the computing nodes in the distributed database, which also includes multiple storage nodes. Each storage node corresponds to one or more sharded tables. The method includes:

[0055] S101, receive a capacity adjustment command from the application, which indicates whether to expand or shrink the capacity. The capacity adjustment command includes a second logical table. At the time the capacity adjustment command is received, there are N1 first shard tables corresponding to the N storage nodes. These N1 first shard tables are calculated from the first logical table based on the first shard key and the first sharding rule.

[0056] For example, expansion can be based on capacity adjustment commands to increase the number of storage nodes or expand the capacity of existing storage nodes.

[0057] For example, scaling down can be based on capacity adjustment commands to reduce the number of storage nodes or reduce the capacity of existing storage nodes.

[0058] For example, the first logical table may be the original logical data set to be processed by sharding.

[0059] For example, the second logical table may contain logical data association information from capacity adjustment commands.

[0060] For example, the first sharding key can be a core identifier used to perform sharding calculations on the first logical table. For instance, the first sharding key could be a unique identifier (ID) or a timestamp.

[0061] For example, the first sharding rule can be a strategy for splitting data in the first logical table based on the first sharding key.

[0062] For example, the first shard table can be a data shard obtained by calculating the first logical table using the first shard key and the first shard rule. There are N1 first shard tables for N storage nodes, where N1 is greater than or equal to N, that is, the number of first shard tables is greater than or equal to the number of storage nodes.

[0063] Optionally, the processing system can split the data of the first logical table based on a preset first sharding key and first sharding rules, using hash allocation to generate N1 first shard tables. Alternatively, it can split the data of the first logical table based on a range of sharding key values, where the sharding key value range could be, for example, a time range. The association between the N1 first shard tables and N storage nodes is established through a configuration file or system mapping table.

[0064] Optionally, the processing system can receive capacity adjustment commands from the application via Remote Procedure Call (RPC), and extract parameters such as adjustment type, second logical table identifier, and target node number through the command parsing module.

[0065] S102, calculate the target data table based on the second sharding key and the second sharding rule to obtain M second sharding tables. The M1 second sharding tables correspond to the M storage nodes. M1 is greater than or equal to M. The N storage nodes do not overlap with the M storage nodes. N, N1, M and M1 are all integers greater than 1.

[0066] For example, the second sharding key could be a core field used to perform sharding calculations on the target data table.

[0067] For example, the second sharding rule can be used to generate a second sharding table that adapts to M new storage nodes.

[0068] For example, the target data table can be a data source that needs to be sharded.

[0069] Optionally, the processing system can associate the second logical table included in the capacity adjustment command with the first logical table to determine the range of the target data table and perform deduplication and format standardization on the target data table. For example, the processing system can associate the second logical table included in the capacity adjustment command with the first logical table to determine the full data of the first logical table, the unsplit incremental data, or the specific shard data to be migrated.

[0070] Optionally, the processing system can determine the second sharding key based on the data characteristics of the target data table and the first logical table. If it needs to adapt to new business scenarios, a new core field can be selected as the second sharding key. The processing system can use the same algorithm as the first sharding rule as the second sharding rule, or it can design the second sharding rule according to the characteristics of the new storage node.

[0071] Optionally, the processing system can traverse the target data table through the sharding calculation module, perform second sharding rule operations on the second sharding key of each data entry in the target data table, and determine the number of the second sharding table for each data entry. Based on the above number, M1 second sharding tables are generated in groups, and sharding metadata containing sharding range, data volume, and creation time can also be generated.

[0072] Optionally, the processing system can set the identifier prefix for storage nodes through the resource management module. For example, the N storage nodes corresponding to the N first shard tables can be prefixed with "Old", and the M1 storage nodes corresponding to the M second shard tables can be prefixed with "New".

[0073] As one possible implementation, the processing system can obtain a first sharding key, a first logical table, a second sharding key, and the corresponding first and second data distribution characteristics of the second logical table. Then, based on the first and second data distribution characteristics, a sharding rule engine determines the first and second sharding rules.

[0074] Optionally, the processing system can read configuration files or query the database metadata cache to obtain the field attributes of the first shard key and the second shard key. It can also read the mapping between the first shard key and the first logical table, and the second shard key and the second logical table, from the business configuration or metadata cache.

[0075] Optionally, the processing system can determine the uniformity characteristics by using statistical functions to count the number of different values ​​of the sharding key and the proportion of data corresponding to each value, based on the first and second logical tables mentioned above. It can also obtain the minimum, maximum, and range length of the sharding key to determine the range characteristics.

[0076] Optionally, the sharding rule engine may include hash sharding, range sharding, and list sharding, etc. For example, the processing system can, based on the sharding rule engine and preset matching rules, adapt the input data distribution characteristics to the algorithm to determine the corresponding sharding rules. For instance, if the first data distribution characteristic is "uniform sharding key distribution with no hotspots," hash sharding can be used to generate the first sharding rule; if the second data distribution characteristic is "clear sharding key value range, increasing over time," range sharding can be used to generate the second sharding rule.

[0077] It should be understood that this application does not limit the sharding algorithm included in the sharding rule engine that derives sharding rules based on data distribution characteristics. For example, the processing system can derive sharding rules based on data distribution characteristics using any existing sharding algorithm.

[0078] Optionally, the processing system can extract the second shard key based on the data record of each second logical table, and generate a hash value by executing a hash algorithm. The hash value is then modulo-operated with the number of storage nodes M to obtain the second shard table number corresponding to the data. Based on the second shard table number, the data in the second logical table is grouped to generate M1 second shard tables. Simultaneously, consecutive index values ​​can be assigned to the M storage nodes corresponding to the M1 second shard tables based on the second shard table number. The mapping relationship between the second shard table number, the storage node index value, and the physical address of the storage node is stored in the system routing table or configuration file.

[0079] By employing the above method, the first sharding key, the first logical table, the second sharding key, and the corresponding first and second data distribution characteristics of the second logical table are obtained, laying the foundation for subsequently determining the first and second sharding rules. Based on these first and second data distribution characteristics, the sharding rule engine determines the first and second sharding rules, reducing execution time and improving processing efficiency.

[0080] S103, perform capacity adjustment based on the M1 second shard tables. Before the capacity adjustment is completed, database services are provided based on the N storage nodes; after the capacity adjustment is completed, database services are provided based on the M storage nodes.

[0081] Optionally, before performing capacity adjustment, the processing system can compare the total number of records and key fields of the second shard table and the second logical table using a verification algorithm to simulate the read and write operations of the distributed database and obtain a complete and usable result of the distributed database.

[0082] Optionally, the processing system can export the data from the N1 first shard tables using a data export tool, and import it in batches into the corresponding M storage nodes' second shard tables according to the hash modulo rule of the second shard tables. Alternatively, it can use an online synchronization tool to migrate the entire dataset in real time and adjust capacity. The processing system can also monitor incremental data on the N storage nodes, convert it into standardized synchronization commands through a log parsing module, and synchronize it to the second shard tables on the M storage nodes.

[0083] Optionally, the processing system can iterate through all business primary keys and compare the key fields of the corresponding data in the storage nodes during capacity adjustment. For a full migration, the system can perform a full verification after capacity adjustment; for incremental data migration, the system can periodically perform incremental verification, for example, checking newly added data every 5 minutes. If three consecutive verifications pass, the data is considered consistent. If inconsistencies are found during verification, the system can extract the abnormal data from N storage nodes, resynchronize it, and record the anomaly in the log.

[0084] In this embodiment, a capacity adjustment command is received from the application. This command instructs whether to expand or shrink the storage. The command includes a second logical table, laying the foundation for subsequent capacity adjustment. At the moment the capacity adjustment command is received, there are N1 first shard tables corresponding to the N storage nodes. These N1 first shard tables are calculated from the first logical tables based on the first shard key and the first shard rule. Based on the second shard key and the second shard rule, M1 second shard tables are obtained from the target data table, improving the dynamic scaling capability of the distributed database. These M1 second shard tables correspond to the M storage nodes. The N and M storage nodes do not overlap, and both N and M are integers greater than 1. Capacity adjustment is performed based on these M1 second shard tables, improving the flexibility of distributed database processing. Before the capacity adjustment is completed, database services are provided based on the N storage nodes; after the capacity adjustment is completed, database services are provided based on the M storage nodes.

[0085] The following section provides a detailed explanation of how the processing system performs capacity adjustments based on these M1 second shard tables.

[0086] As one possible implementation, the processing system can write existing data from the N storage nodes to the M storage nodes based on the second sharding table, and read new data during the capacity adjustment process based on the first sharding table, and write the new data to the M storage nodes based on the second sharding table.

[0087] Optionally, the processing system can extract the second shard key for each data entry in the first shard table, calculate the second shard table number based on the second sharding rules, and form a mapping relationship between the first shard table data records, the second shard table number, and the newly added nodes. By employing a distributed data migration tool, multiple threads read the data from the first shard table of the original N storage nodes in parallel, and write it to the second shard table of the corresponding M storage nodes based on the above mapping relationship.

[0088] Optionally, the processing system can determine whether the migration was successful by comparing the first shard table and the second shard table after each batch of data is written. If the first shard table and the second shard table match, the data migration is considered successful; if they do not match, the migration is retried three times, and an exception log is recorded.

[0089] Optionally, the processing system can also obtain the write operation logs of the first shard table of the original N storage nodes through the database log parsing module. For example, the write operation logs may include core information such as operation type, data content, second shard key, and primary key.

[0090] Optionally, the processing system can also extract the second sharding key of the newly added data, calculate the second sharding table number based on the second sharding rules, and write the newly added data into the second sharding table of the corresponding storage node based on the second sharding table number. The processing system can also locate the corresponding data record in the second sharding tables of M storage nodes based on the primary key in the newly added data and the second sharding table number calculated based on the second sharding key, and perform update or delete operations. Through the above methods, existing data in N storage nodes is written to M storage nodes based on the second sharding table, and newly added data during the capacity adjustment process is read based on the first sharding table and written to the M storage nodes based on the second sharding table, ensuring data consistency and improving the stability of distributed database processing.

[0091] As one possible implementation, after the capacity adjustment is performed based on the M1 second shard tables, the processing system can pause the write operation of the distributed database and switch the effective logical table to the second logical table. Then, after the switch is completed, the write operation of the distributed database is resumed to complete the capacity adjustment.

[0092] Optionally, the processing system can call the distributed database's permission management interface to batch disable write operation permissions for the distributed database, and can also pause write operations in stages according to business priority. For example, write operations for non-core business operations can be paused first, and then write operations for core business operations can be paused. A maximum pause threshold can also be set via a timer; if the switching time exceeds the maximum pause threshold, an exception log will be recorded and a warning will be issued. For example, the maximum pause threshold could be 30 seconds.

[0093] Optionally, if write operations in the distributed database are paused, the processing system can return a standardized response through the database gateway. For example, this standardized response could be: "Capacity adjustment in progress, write operations are temporarily unavailable, estimated recovery time 30 seconds." Alternatively, write requests can be cached via a queue.

[0094] Optionally, the processing system can update the logical table identifier to the second logical table identifier, the sharding key to the second sharding key, and the sharding rule to the second sharding rule. It can also delete the mapping relationship between the first sharding table and the original N storage nodes, and add a mapping relationship between the second sharding table number and the physical addresses of M new nodes.

[0095] Optionally, the processing system can preload the data of the second logical table into the local cache of all database nodes, and then initiate a configuration commit through a distributed consistency protocol, with all nodes simultaneously switching to the data of the second logical table. If any node fails to commit, all nodes will revert to the original configuration data.

[0096] Optionally, after the switchover is complete, the processing system can batch resume write operations through the permission management interface. Write operations are automatically routed to the second shard table of M storage nodes generated based on the second logical table and the second sharding rules. A capacity adjustment completion notification is sent to the application via a callback interface. For example, this notification may include the new logical table identifier and information about the newly added nodes.

[0097] Optionally, the processing system can also monitor the write operation response latency and error rate of the newly added storage node in real time within 5 minutes of the recovery of the distributed database's write operations. If the response latency exceeds a preset response latency threshold of 100 milliseconds, or the error rate exceeds a preset error rate threshold of 0.1%, the processing system can determine that the indicator is abnormal, trigger an alarm, and provide a rollback option. For example, if the indicator is determined to be abnormal, the rollback option can be used to switch back to the first logical table and the original storage node.

[0098] Using the above method, after performing capacity adjustment based on the M1 second shard tables, the write operation of the distributed database can be paused, and the effective logical table can be switched to the second logical table, ensuring data consistency and stability. After the switch is completed, the write operation of the distributed database is resumed to complete the capacity adjustment, reducing manual intervention and improving the flexibility of distributed database processing.

[0099] In some embodiments, the processing system can determine the switching timing based on historical request logs. Then, at that switching timing, the logical table that takes effect is the second logical table.

[0100] For example, historical request logs can be business access behavior data recorded by a distributed database during operation. For instance, historical request logs include core fields such as request timestamps, request types, request volume, and response latency.

[0101] For example, the switching timing could be a time window suitable for switching the first logical table to the second logical table.

[0102] Optionally, the processing system can use a distributed log collection framework to collect historical request logs from all database nodes. These historical request logs may include fields such as request time, request type, request volume, response latency, and business identifiers. Preprocessing operations such as cleaning and standardization are then performed based on these historical request logs.

[0103] Optionally, the processing system can analyze historical request logs based on the time dimension to determine the peak and trough values ​​of the daily average load. The processing system can also use historical request logs to calculate the ratio of write operation requests to total requests in different time periods, identifying the periods with the lowest write operation percentages. Furthermore, the processing system can use historical request logs to analyze the request distribution between core and non-core business processes.

[0104] Optionally, the processing system can determine low-load time windows based on historical request logs from the past 7 days and configurable preset thresholds, using this as a switching opportunity. For example, if the request volume during a certain period is less than 30% of the daily average peak load, the write operation ratio is less than 15%, and the core business request volume is less than 5%, the processing system can determine that period as a low-load time window.

[0105] Optionally, the processing system can also input historical load features from historical request logs into a machine learning model, train the machine learning model using the handover success rate, predict the handover adaptability for each time period using the machine learning model, and output the window period with the highest adaptability as the handover timing.

[0106] Optionally, the processing system can use a timer to initiate the switchover at the confirmed switchover timing. First, write operations are paused, and then the effective logical table is switched to the second logical table. This method determines the switchover timing based on historical request logs, reducing manual intervention and lowering costs. Switching the effective logical table to the second logical table at the designated switchover timing improves switchover stability.

[0107] As one possible implementation, the processing system can collect load metrics from the M storage nodes. Then, if it is confirmed that the M storage nodes are overloaded based on these load metrics, a new capacity adjustment instruction is automatically generated, and the capacity adjustment is performed based on the new capacity adjustment instruction.

[0108] Optionally, the processing system can collect load metrics of the M storage nodes in real time based on the node's local agent and transmit them to the detection module via Hypertext Transfer Protocol (HTTP). For example, load metrics may include processor utilization, memory utilization, and network bandwidth utilization.

[0109] Optionally, the processing system can compare the collected load metrics with preset overload thresholds to determine whether the M storage nodes are overloaded. For example, the preset overload thresholds could be a processor utilization threshold of 85%, a memory utilization threshold of 90%, and a network bandwidth utilization threshold of 80%. If any of the core load metrics exceeds the preset overload threshold, the processing system can determine that the M storage nodes are overloaded. The processing system can also analyze the historical load data of the M storage nodes using a linear regression model to predict the overload threshold.

[0110] Optionally, the processing system can also generate overload alarm logs based on the overload status of the M storage nodes and trigger an automatic scaling process. For example, the overload alarm logs may include the identity of the overloaded node, the exceeded metric, and the duration. If the load of the M storage nodes is not overloaded, the metrics continue to be collected periodically.

[0111] Optionally, the processing system can automatically calculate parameters such as the target number of new nodes and new sharding rules based on the above overload determination results, combined with historical load data and node hardware specifications, and generate standardized new capacity adjustment instructions. Based on the above new capacity adjustment instructions, the second sharding key, and the hash modulo rule, the processing system can re-split the second sharding table data of the M storage nodes and perform capacity adjustment.

[0112] By using the above method, load metrics of the M storage nodes are collected, reducing operation and maintenance costs and providing data for subsequent load adaptation. If the load metrics confirm that the M storage nodes are overloaded, a new expansion adjustment command is automatically generated, and capacity adjustments are executed based on this new command, improving resource utilization and the flexibility of distributed database processing.

[0113] Figure 2 This is a schematic diagram of an overall architecture for scaling up and down provided in this application. Figure 2 As shown, the overall scaling up / down configuration includes: the application program, the original two storage nodes, and three new storage nodes.

[0114] Optionally, scaling up or down can be triggered via an application capacity adjustment command. Before scaling up, the old sharding and sharding planning are used to obtain the original sharding table. The original sharding table contains storage node 0 and storage node 1, each node carrying two corresponding data shards. After dynamic scaling up or down, the system enters the scaled-up state, where the new sharding and sharding planning are used to obtain the new sharding table. The new sharding table contains storage node 2, storage node 3, and storage node 4, which carry the corresponding data shards.

[0115] Optionally, connect to the compute node and execute commands on the compute node to specify new sharding keys and sharding rules for the tables that need to be scaled up or down. The compute node creates a new metadata storage in the remote repository according to the commands, stores the new sharding keys and sharding rules, and creates new sharded tables on the corresponding storage nodes based on the new sharding keys and rules.

[0116] At this point, two versions of metadata will exist simultaneously. The currently effective metadata stores the original sharding key and sharding rules, which is called metadata 1. The new metadata stores the sharding key and sharding rules after scaling up and down, which is called metadata 2.

[0117] Next, data redistribution is performed. Based on metadata 1, all existing data and incremental data generated during the scaling process are read, and the read data is written to the new shard table according to metadata 2. When all existing and incremental data redistribution is complete, the database briefly stops write operations, switches the currently effective metadata version to metadata 2, and resumes write operations after the switch is complete. In other words, dynamic scaling is completed based on the new sharding key and sharding rules.

[0118] Figure 3 This is a flowchart illustrating a method for creating a new sharding table based on a new sharding key and sharding rules, as provided in this application. Figure 3 As shown, the method includes:

[0119] Optionally, after the new shard table is created, the shard data of the source storage node 0 and storage node 1 are aggregated through the source logical table, and then transmitted to the target logical table after being scheduled by the shard agent. Finally, the target logical table distributes the data to the target storage nodes 2, 3, and 4, completing the deployment of the shard data corresponding to the new shard table.

[0120] For example, the core scheduling layer may include a sharding agent for connecting the source logical table and the target logical table.

[0121] For example, the source storage node may include storage node 0 and storage node 1, each of which carries data shard 0 and data shard 1. The target storage node may include storage node 2, storage node 3, and storage node 4, each of which carries data shard 0 and data shard 1.

[0122] Figure 4 This is a flowchart illustrating a method for redistributing existing data provided in this application. Figure 4 As shown, the method includes:

[0123] Optionally, source storage node 0 and storage node 1 query the database to obtain the sharded data they carry, and then write the data into the target logical table. After receiving the data, the target logical table distributes it to target storage nodes 2, 3, and 4, completing the cross-node synchronization of data shards from the source to the target storage nodes.

[0124] Figure 5 This is a flowchart illustrating an incremental data redistribution method provided in this application. Figure 5 As shown, the method includes:

[0125] Optionally, after the source storage node 0 and storage node 1 obtain their own changed data, they transmit the data to the source logical table. Through sharding agent scheduling, the changed data in the source logical table is written to the target logical table. Then, the target logical table distributes the changed data to the target storage node 2, storage node 3, and storage node 4, completing the cross-node synchronization of the changed data.

[0126] Figure 6 This is a flowchart illustrating a method for switching metadata provided in this application. Figure 6 As shown, the method includes:

[0127] Optionally, after the sharding agent triggers the switchover operation, it transfers its association with the sharded data from the source logical table to the target logical table. After the switchover is complete, the logical aggregation and access entry point of the sharded data are switched from the source logical table to the target logical table. The sharded data of the source storage node is associated with the source logical table, and the sharded data of the target storage node is associated with the target logical table.

[0128] The above are the method embodiments provided in this application. The apparatus provided in this application will be described below.

[0129] Figure 7 This application provides a schematic diagram of the structure of a distributed database processing device, as shown below. Figure 7 As shown, this device is applied to the computing nodes in the distributed database. The distributed database also includes multiple storage nodes, with each storage node corresponding to one or more sharded tables. The distributed database processing device 700 provided in this embodiment includes: a receiving module 701, a computing module 702, and an adjustment module 703. Wherein,

[0130] The receiving module 701 is used to receive a capacity adjustment command from the application, which is used to indicate expansion or reduction. The capacity adjustment command includes a second logical table. At the time the capacity adjustment command is received, there are N1 first shard tables corresponding to N storage nodes. The N1 first shard tables are calculated based on the first shard key and the first shard rule of the first logical table, and N1 is greater than or equal to N.

[0131] The calculation module 702 is used to calculate the target data table based on the second sharding key and the second sharding rule to obtain M1 second sharding tables. The M1 second sharding tables correspond to M storage nodes. M1 is greater than or equal to M. The N storage nodes do not overlap with the M storage nodes. N, N1, M and M1 are all integers greater than 1.

[0132] Adjustment module 703 is used to perform capacity adjustment based on the M1 second shard tables;

[0133] Before the capacity adjustment is completed, database services will be provided based on these N storage nodes.

[0134] After the capacity adjustment is completed, database services will be provided based on these M storage nodes.

[0135] Optionally, the calculation module 702 is further configured to obtain the first sharding key, the first logical table, the second sharding key, and the corresponding first data distribution characteristics and second data distribution characteristics of the second logical table;

[0136] Based on the first data distribution feature and the second data distribution feature, the first sharding rule and the second sharding rule are determined by the sharding rule engine.

[0137] Optionally, the adjustment module 703 is further configured to write existing data from the N storage nodes into the M storage nodes based on the second sharding table; and to read new data during the capacity adjustment process based on the first sharding table, and write the new data into the M storage nodes based on the second sharding table.

[0138] Optionally, the adjustment module 703 is also used to pause the write operation of the distributed database after performing capacity adjustment based on the M second shard tables, and to switch the effective logical table to the second logical table;

[0139] After the switch is completed, write operations to the distributed database are resumed to complete the capacity adjustment.

[0140] Optionally, the adjustment module 703 is also used to determine the switching timing based on historical request logs;

[0141] The logic table that takes effect during the switching is the second logic table.

[0142] Optionally, the adjustment module 703 is also used to collect load metrics of the M storage nodes;

[0143] If the load index confirms that the M storage nodes are overloaded, a new capacity adjustment instruction is automatically generated, and the capacity adjustment is performed based on the new capacity adjustment instruction.

[0144] This embodiment provides a distributed database processing device that can execute the methods provided in any of the above method embodiments. The implementation principle and technical effects are similar, and will not be described in detail here.

[0145] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 8As shown, the electronic device 800 provided in this embodiment includes at least one processor 801 and a memory 802. Optionally, the device 800 further includes a communication component 803. The processor 801, memory 802, and communication component 803 are connected via a bus 804.

[0146] In a specific implementation, at least one processor 801 executes computer execution instructions stored in memory 802, causing at least one processor 801 to perform the above-described method.

[0147] The specific implementation process of processor 801 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0148] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0149] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0150] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0151] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0152] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0153] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0154] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0155] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0156] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or in the form of software program modules.

[0157] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0158] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0159] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0160] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0161] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method of distributed database processing, the method comprising: The method is applied to computing nodes in the distributed database, which also includes multiple storage nodes, one of which corresponds to one or more sharded tables. The system receives a capacity adjustment command from an application, which indicates whether to expand or shrink the capacity. The capacity adjustment command includes a second logical table. At the moment the capacity adjustment command is received, there are N1 first shard tables corresponding to the N storage nodes. The N1 first shard tables are calculated from the first logical table based on the first shard key and the first shard rule, and N1 is greater than or equal to N. The target data table is calculated based on the second sharding key and the second sharding rule to obtain M1 second sharding tables. The M1 second sharding tables correspond to M storage nodes. M1 is greater than or equal to M. N storage nodes do not overlap with M storage nodes. N, N1, M, and M1 are all integers greater than 1. Capacity adjustment is performed based on the M1 second shard tables; Before the capacity adjustment is completed, database services are provided based on the N storage nodes; After the capacity adjustment is completed, database services are provided based on the M storage nodes.

2. The method of claim 1, wherein, The capacity adjustment based on the M1 second shard tables includes: Based on the second sharding table, the existing data in the N storage nodes is written into the M storage nodes; In addition, new data is read from the first shard table during the capacity adjustment process, and the new data is written to the M storage nodes based on the second shard table.

3. The method according to claim 1 or 2, characterized in that, After performing capacity adjustment based on the M1 second shard tables, the following is also included: Pause the write operations of the distributed database and switch the effective logical table to the second logical table; After the switch is completed, write operations on the distributed database are resumed to complete the capacity adjustment.

4. The method of claim 3, wherein, The logic table for which the switch takes effect is the second logic table, which includes: Determine the timing of the switchover based on historical request logs; The logic table that takes effect during the switching time is the second logic table.

5. The method according to claim 1 or 2, characterized in that, The method includes: Obtain the first sharding key, the first logical table, the second sharding key, and the corresponding first data distribution characteristics and second data distribution characteristics of the second logical table; Based on the first data distribution characteristics and the second data distribution characteristics, the first sharding rule and the second sharding rule are determined by the sharding rule engine.

6. The method of claim 1 or 2, wherein, The method further includes: Collect load metrics for M storage nodes; If the load of the M storage nodes is confirmed to be overloaded based on the load index, a new capacity expansion adjustment instruction is automatically generated, and the capacity adjustment is performed based on the new capacity expansion adjustment instruction.

7. A distributed database processing apparatus, applied to computing nodes in the distributed database, the distributed database further comprising multiple storage nodes, each storage node corresponding to one or more sharded tables, comprising: A receiving module is configured to receive a capacity adjustment command from an application, the capacity adjustment command being used to indicate expansion or reduction, the capacity adjustment command including a second logical table; wherein, at the moment the capacity adjustment command is received, there are N1 first shard tables corresponding to the N storage nodes, the N1 first shard tables being calculated from the first logical table based on the first shard key and the first shard rule, and N1 being greater than or equal to N; The calculation module is used to calculate the target data table based on the second sharding key and the second sharding rule to obtain M1 second sharding tables. The M1 second sharding tables correspond to M storage nodes. N storage nodes do not overlap with M storage nodes. N, N1, M and M1 are all integers greater than 1. The adjustment module is used to perform capacity adjustment based on the M1 second shard tables; Before the capacity adjustment is completed, database services are provided based on the N storage nodes; After the capacity adjustment is completed, database services are provided based on the M storage nodes.

8. An electronic device, comprising: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images