A dynamic sharding scheduling method and device for a distributed database and a medium

Abstract

The embodiment of the application discloses a kind of distributed database-oriented dynamic fragmentation scheduling method, equipment and medium, belong to database technical field, solve the problem that the existing distributed database scheduling mode communication delay is high, resource is not balanced, make the low problem of statistical efficiency. Including, the global network topology view corresponding to the cluster is constructed;Data volume query request is based on determining the data fragment of target table on each computing node in cluster;The computing node is divided into multiple aggregation groups, and the corresponding aggregation node is determined for each aggregation group;Based on dynamic scheduling model, determine the scheduling fitness that the scan result of data fragment is sent to the corresponding aggregation node;Based on scheduling fitness, the scan task of each data fragment is assigned to its corresponding source computing node, and the aggregation node to which the scan result should be sent is specified to carry out incremental task migration;Local summary result of all aggregation nodes is integrated, and the integration result is returned to user.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a dynamic sharding scheduling method, device and medium for distributed databases. Background Technology

[0002] With the rapid development of cloud-native database technology, large-scale distributed database clusters have become the infrastructure supporting modern data-intensive applications. These clusters typically feature cross-regional, cross-availability zone, and even cross-cloud deployments, with data volumes growing exponentially. Real-time acquisition of data volume in each table is a crucial prerequisite for scenarios such as database expansion and data migration.

[0003] Traditional data volume statistics methods, when dealing with large-scale distributed databases, typically involve the master node broadcasting statistical commands to all storage nodes. Each node independently calculates its local data and returns the results to the master node for aggregation. However, in complex topologies spanning high-latency networks, network communication latency becomes a major bottleneck. Unoptimized data transmission paths between nodes significantly increase overall statistical processing time. Another improved approach uses hash algorithms to statically map table shards to specific summing nodes for intermediate aggregation. While this reduces some of the traffic directly returned to the master node, its static sharding strategy leads to a coexistence of node overload and idleness, causing high-load nodes to reach 100% CPU utilization and statistical tasks to queue. Therefore, current methods for data volume statistics in distributed databases across data centers and clouds often suffer from high communication latency and uneven resource utilization, ultimately resulting in low statistical efficiency. Summary of the Invention

[0004] This application provides a dynamic sharding scheduling method, device, and medium for distributed databases to solve the following technical problem: When performing data volume statistics in distributed databases across data centers and cloud environments, the statistical task execution process usually suffers from high communication latency and uneven resource utilization, ultimately resulting in low statistical efficiency.

[0005] The embodiments of this application adopt the following technical solutions: This application provides a dynamic sharding scheduling method for distributed databases. The method includes: constructing a global network topology view corresponding to the cluster; wherein the global network topology view includes at least the real-time network latency between each computing node and the real-time load status of each computing node; receiving a data volume query request and determining the data shards of the target table on each computing node in the cluster based on the data volume query request; dividing the computing nodes into multiple aggregation groups according to the global network topology view, and determining corresponding aggregation nodes for each aggregation group; wherein, by dividing the aggregation groups, the network latency between nodes within the same aggregation group is lower than a preset threshold; determining the scheduling suitability for sending the scan results of the data shards to the corresponding aggregation nodes based on a dynamic scheduling model; assigning the scan tasks of each data shard to its corresponding source computing node based on the scheduling suitability, and specifying the aggregation nodes to which the scan results should be sent for incremental task migration; integrating the local summary results of all aggregation nodes, and returning the integrated results to the user.

[0006] In one implementation of this application, constructing a global network topology view corresponding to the cluster specifically includes: constructing an inter-node communication delay matrix based on the network round-trip delay between different computing nodes in the cluster; determining the available bandwidth between nodes by sending sampled data packets; collecting real-time load indicators of each computing node based on a preset time interval; constructing a global network topology view based on the inter-node communication delay matrix, the available bandwidth between nodes, and the real-time load indicators; monitoring the health status and load changes of each computing node, and updating the global network topology view when a preset trigger event occurs; wherein the trigger event includes at least one of the following: a node joining or leaving the cluster, a node load exceeding a preset threshold, a sudden increase in inter-node network delay exceeding a preset ratio, and a node losing or recovering.

[0007] In one implementation of this application, the computing nodes are divided into multiple aggregation groups based on the global network topology view, and corresponding aggregation nodes are determined for each aggregation group. Specifically, this includes: constructing a node communication graph based on the full network delay matrix contained in the global network topology view; wherein the computing nodes are the vertices of the communication graph, and the reciprocal of the network delay between any two nodes is used as the weight of the corresponding edge in the communication graph; running a community detection algorithm on the node communication graph, and identifying node communities with connections as aggregation groups; and selecting an aggregation node from the candidate aggregation nodes by using multi-objective constraints, with each node in the aggregation group as a candidate aggregation node.

[0008] In one implementation of this application, an aggregation node is selected from candidate aggregation nodes through multi-objective constraints. Specifically, this includes: obtaining the real-time load status corresponding to each candidate aggregation node and determining the average network latency from other nodes in the aggregation group to the candidate aggregation node; determining a reliability factor based on the fault domain isolation information between the candidate aggregation node and several other nodes in the group; substituting the real-time load status, average network latency, and reliability factor corresponding to each candidate aggregation node into a preset scoring function to determine the comprehensive election score corresponding to each candidate aggregation node; wherein, the preset scoring function is used to ensure that candidate aggregation nodes whose load is proportional to the average network latency and whose load is inversely proportional to the fault domain isolation degree of several nodes in the group obtain higher election scores; and determining the candidate aggregation node with the highest comprehensive election score as the aggregation node of the aggregation group.

[0009] In one implementation of this application, a dynamic scheduling model is used to determine the scheduling suitability of sending the scan results of data shards to the corresponding aggregation nodes. Specifically, this includes: determining metadata characteristics based on the number of columns and indexes in the table to which the current data shard belongs; performing hash sampling based on the data distribution key of the current data shard and determining the data distribution characteristics through the skewness of the sampling results; fusing the metadata characteristics and data distribution characteristics to determine the data complexity; obtaining the real-time load status of each candidate aggregation node; calculating the average network latency from the source compute node where the data shard resides to the aggregation group to which the aggregation node belongs for each aggregation node; determining the load balancing weight coefficient and network cost weight coefficient based on the type of the current data volume query request; and inputting the data complexity, real-time load status, average network latency, load balancing weight coefficient, and network cost weight coefficient into the dynamic scheduling model to calculate the scheduling suitability.

[0010] In one implementation of this application, the scanning task for each data shard is assigned to its corresponding source computing node, specifically including: generating a shard query plan based on scheduling suitability and distributing the shard query plan to each source computing node; wherein, the shard query plan includes at least one of the following: a list of table shards to be scanned, the address of the target aggregation node, and query execution parameters; scanning the local metadata table through the source computing node to determine the data volume of each shard and formatting the data volume into structured data records; and sending the data records to the corresponding target aggregation node when the number of data records reaches a preset sending threshold.

[0011] In one implementation of this application, incremental task migration is performed, specifically including: during the execution of the migration task, real-time monitoring of the task execution progress and load status of each source computing node; when resource overload or execution lag is detected on any source computing node, the data fragments that have not yet been scanned on the source computing node are identified as fragments to be migrated; the fragments to be migrated are rescheduled to one or more healthy computing nodes, and the aggregation node corresponding to the fragments to be migrated is notified to update the expected data reception so that subsequent results can be received on the new computing node; wherein, the aggregation node is set to an idempotent reception mechanism.

[0012] In one implementation of this application, the local summary results of all aggregation nodes are integrated, and the integrated result is returned to the user. Specifically, this includes: after the aggregation nodes complete the local summary, generating a local summary result and generating a cryptographic integrity certificate based on all received scan result data; wherein, the cryptographic integrity certificate is used to prove that the local summary result is complete and has not been tampered with; each aggregation node sends the local summary result and the cryptographic integrity certificate to the master node; after receiving the data packets from each aggregation node, the master node performs the following operations: the master node verifies the integrity of the local summary result using the cryptographic integrity certificate reported by each aggregation node; for the local summary result that passes the integrity verification, a secondary summary is performed to obtain the global integration result; the global integration result is used as the total data volume of the target table and returned to the user.

[0013] This application provides a dynamic sharding scheduling device for distributed databases, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to: construct a global network topology view corresponding to the cluster; wherein the global network topology view includes at least the real-time network latency between each computing node and the real-time load status of each computing node; receive a data volume query request and determine the data shards of the target table on each computing node in the cluster based on the data volume query request; divide the computing nodes into multiple aggregation groups according to the global network topology view, and determine corresponding aggregation nodes for each aggregation group; wherein, by dividing the aggregation groups, the network latency between nodes within the same aggregation group is lower than a preset threshold; determine the scheduling suitability for sending the scanning results of the data shards to the corresponding aggregation nodes based on a dynamic scheduling model; allocate the scanning tasks of each data shard to its corresponding source computing node based on the scheduling suitability, and specify the aggregation nodes to which the scanning results should be sent for incremental task migration; integrate the local summary results of all aggregation nodes, and return the integrated results to the user.

[0014] This application provides a non-volatile computer storage medium storing computer-executable instructions, which are configured to: construct a global network topology view corresponding to a cluster; wherein the global network topology view includes at least the real-time network latency between each computing node and the real-time load status of each computing node; receive a data volume query request and determine the data shards of the target table on each computing node in the cluster based on the data volume query request; divide the computing nodes into multiple aggregation groups according to the global network topology view, and determine the corresponding aggregation nodes for each aggregation group; wherein, by dividing the aggregation groups, the network latency between nodes within the same aggregation group is lower than a preset threshold; determine the scheduling suitability for sending the scanning results of the data shards to the corresponding aggregation nodes based on a dynamic scheduling model; allocate the scanning tasks of each data shard to its corresponding source computing node based on the scheduling suitability, and specify the aggregation nodes to which the scanning results should be sent for incremental task migration; integrate the local summary results of all aggregation nodes, and return the integrated results to the user.

[0015] The above-mentioned technical solutions adopted in this application embodiment can achieve the following beneficial effects: By constructing a global network topology view, this application embodiment can accurately perceive the actual quality differences of cross-data center and cross-cloud links and the status of each node, and can proactively avoid high-latency links and nodes that are about to be overloaded, laying the foundation for optimizing communication costs and achieving load balancing. Secondly, this application embodiment optimizes the data transmission efficiency within the group by dynamically dividing aggregation groups and determining aggregation nodes. At the same time, by electing aggregation nodes through multi-objectives, it not only prevents the overload of a single node, but also improves the fault tolerance of a single task unit by deploying aggregation nodes in physical areas different from multiple data nodes. Furthermore, by calculating scheduling suitability based on a dynamic model, scheduling can adapt to data characteristics and real-time system load to achieve the best match between resources and tasks. When node overload or slow execution is detected, unfinished shards are hot-migrated to healthy nodes online, and the data flow is automatically updated, thereby avoiding the drawbacks of task freezing or needing to be restarted due to single point of failure in traditional solutions, and improving task success rate and efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A system architecture diagram for dynamic sharding scheduling of distributed databases is provided in the embodiments of this application; Figure 2 A flowchart of a dynamic sharding scheduling method for distributed databases provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of a dynamic sharding scheduling device for distributed databases provided in an embodiment of this application.

[0017] Figure label: 200: Dynamic sharding scheduling device for distributed databases; 201: Processor; 202: Memory. Detailed Implementation

[0018] This application provides a dynamic sharding scheduling method, device, and medium for distributed databases.

[0019] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0020] The technical solutions proposed in the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 A system architecture diagram for dynamic sharding scheduling of distributed databases is provided as an embodiment of this application, such as... Figure 1 As shown, by incorporating network topology information and node load status into task scheduling decisions, efficient distributed execution of data volume statistics tasks is achieved. The core system architecture includes the following key components: Master node: As the control center of the system, it is responsible for receiving user query commands, generating distributed execution plans, scheduling task distribution, and integrating the final results.

[0022] Computation nodes: These constitute the data execution plane of the system, where data query operations are actually performed, and data scanning and partial summarization are completed.

[0023] Topology awareness module: Collects network topology information and node load indicators in real time to provide basic data support for scheduling decisions.

[0024] Dynamic scheduling engine: Based on real-time information provided by the topology-aware module, it dynamically calculates the optimal data sharding and task allocation strategy.

[0025] The statistical results integration module is responsible for collecting the local summary results from each computing node, performing global integration, and returning the results to the master node.

[0026] The dynamic sharding scheduling system for distributed databases adopts an architecture design that separates the control plane and the data plane. The master node and scheduling-related modules constitute the control plane, while the cluster of compute nodes constitutes the data plane. The topology-aware module and the dynamic scheduling engine act as bypass systems, continuously providing decision support to the control plane.

[0027] Figure 2 A flowchart of a dynamic sharding scheduling method for distributed databases provided in this application embodiment is shown below. Figure 2 As shown, the dynamic sharding scheduling method for distributed databases includes the following steps: Step 101: Construct a global network topology view corresponding to the cluster.

[0028] In one implementation of this application, a communication delay matrix between nodes is constructed using the network round-trip latency between different computing nodes in the cluster. Available bandwidth between nodes is determined by sending sampled data packets. Real-time load metrics for each computing node are collected at preset intervals. A global network topology view is constructed based on the inter-node communication delay matrix, available bandwidth, and real-time load metrics. The health status and load changes of each computing node are monitored. When a preset trigger event occurs, the global network topology view is updated. The trigger event includes at least one of the following: a node joining or leaving the cluster, a node load exceeding a preset threshold, a sudden increase in inter-node network latency exceeding a preset proportion, and a node losing or recovering connection. The global network topology view includes at least the real-time network latency between computing nodes and the real-time load status of each computing node.

[0029] Specifically, the topology awareness module calculates the network round-trip latency between any two compute nodes using UDP probe packets or TCP handshake measurements, constructing an inter-node communication latency matrix. For clusters deployed across availability zones or regions, this inter-node communication latency matrix accurately reflects the actual quality of network links. It assesses available bandwidth between nodes by sending sampled data packets, identifies network bottleneck links, and provides a basis for data skew scheduling. It collects key load metrics for each compute node in real time, such as CPU utilization, memory usage, network I / O throughput, and disk I / O wait time. The initial collection frequency is set to 30 seconds / time, which can be dynamically adjusted according to the cluster size. Based on the collected information, the topology awareness module constructs a three-dimensional topology view including node attributes, link quality, and real-time load. The inter-node communication latency matrix is ​​stored in an in-memory database in JSON format, supporting millisecond-level queries.

[0030] To ensure the real-time effectiveness of scheduling decisions, the topology sensing module in this embodiment employs a dual update mechanism: Periodic updates: A full topology probe can be performed every 5 minutes to update the network latency matrix and bandwidth data, eliminating the impact of changes in the network environment; Event-driven updates: Real-time monitoring of node health and load changes, triggering a topology update immediately when the following events are detected: (1) Nodes join or leave the cluster; (2) The node load exceeds the preset threshold; (3) Network latency suddenly increases beyond the preset value, such as exceeding 50%; (4) Node disconnection or recovery; The updated topology information is pushed to the dynamic scheduling engine in real time to ensure that scheduling decisions are based on the latest state.

[0031] Step 102: Receive the data volume query request and determine the data shards of the target table on each computing node in the cluster based on the data volume query request.

[0032] In one implementation of this application, when a user initiates a data volume query request (such as the SQL statement "SELECT COUNT(*) FROM distributed table"), the master node receives the SQL instruction and performs syntax analysis and semantic checks using its built-in SQL parser to identify key information such as the table objects involved in the query, filtering conditions, and grouping requirements. The master node obtains the distribution information of the target table from the distributed metadata service, including the number of table shards, the nodes where each shard resides, and the data distribution key. For partitioned tables, it also needs to obtain the partitioning strategy and partition key information. Based on the metadata and query requirements, the master node generates an initial distributed execution plan. This plan breaks down the query into subtasks that can be executed in parallel on each computing node, with each subtask responsible for scanning the data shards held by its local node. The master node requests the latest network topology view and node load information from the topology awareness module, and passes this information, along with the execution plan, to the dynamic scheduling engine, awaiting the injection of the scheduling strategy.

[0033] Step 103: Based on the global network topology view, divide the computing nodes into multiple aggregation groups and determine the corresponding aggregation nodes for each aggregation group.

[0034] In one implementation of this application, a node communication graph is constructed based on the full network delay matrix contained in the global network topology view. Nodes are used as vertices of the communication graph, and the reciprocal of the network delay between any two nodes is used as the weight of the corresponding edge in the communication graph. A community detection algorithm is run on the node communication graph, and the identified node communities with connections are grouped into clusters. Each node within a cluster is considered a candidate cluster node, and a cluster node is selected from the candidate cluster nodes through multi-objective constraints. The clustering is designed to ensure that the network delay between nodes within the same cluster is below a preset threshold.

[0035] Specifically, the scheduling engine first obtains the full network latency matrix from the real-time maintained global network topology view. Using each compute node as a vertex in the graph, a fully connected undirected graph is constructed as the node communication graph. For any two nodes in the graph, the weight is defined as the reciprocal of the network latency between those two nodes; that is, the lower the actual network latency between nodes, the higher the weight of the corresponding edge in the communication graph. On the constructed weighted node communication graph, a community detection algorithm, such as the Louvain algorithm, is run. This algorithm, through iterative optimization, can automatically identify node communities with tightly connected internal connections (i.e., low average latency between nodes within a group) and sparse external connections (i.e., high average latency between nodes across groups). Each identified community constitutes an independent aggregation group. This method adapts to the current network topology and is particularly effective in handling complex deployment scenarios such as cross-data center and cross-availability zone deployments, ensuring efficient data transmission within the group. By setting algorithm parameters or post-processing the results, it can be ensured that the network latency between any two nodes within each aggregation group is below a preset threshold, thus logically forming several low-latency, high-performance data processing units.

[0036] After dividing the aggregation groups, all member nodes within each aggregation group are listed as candidate aggregation nodes. Then, a comprehensive election score is calculated for each candidate node in the group. This score is a quantitative representation under multi-objective constraints, calculated using a predefined scoring function. This function comprehensively considers at least the following three core factors: First, the real-time load status of the candidate node; the lower the load, the higher the score, to avoid creating new performance bottlenecks. Second, the average network latency from all other nodes in the aggregation group to the candidate node; the lower this average latency, the higher the score, to ensure that the selected aggregation node has the lowest communication cost with the majority of members in the group. Third, a reliability factor, which is calculated based on whether the candidate node and the majority of nodes in the group are located in different fault domains, such as different data centers or availability zones, following the principle of physical isolation, aiming to improve the survivability of the entire aggregation group in the event of a single availability zone failure.

[0037] Step 104: Based on the dynamic scheduling model, determine the scheduling suitability of sending the scan results of data sharding to the corresponding aggregation node.

[0038] In one implementation of this application, the real-time load status of each candidate aggregation node is obtained, and the average network latency from other nodes in the aggregation group to the candidate aggregation node is determined. Based on the fault domain isolation information between the candidate aggregation node and several other nodes in the group, a reliability factor is determined. The real-time load status, average network latency, and reliability factor of each candidate aggregation node are substituted into a preset scoring function to determine the comprehensive election score for each candidate aggregation node. The preset scoring function ensures that candidate aggregation nodes whose load is proportional to the average network latency and inversely proportional to the fault domain isolation degree of several nodes in the group receive higher election scores. The candidate aggregation node with the highest comprehensive election score is determined as the aggregation node of the aggregation group.

[0039] Specifically, all nodes within an aggregation group are considered candidate aggregation nodes. For each candidate node (agg) within the group, the scheduling engine calculates its comprehensive score, Score_agg(agg), using the following formula: Score_agg(agg) =α* (1-Load(agg)) +β* (1 / AvgLatency(Group, agg)) +γ*Reliability(agg); Here, AvgLatency(Group, agg) is the average network latency from all other scanning nodes in the group to the candidate aggregation node agg; Reliability(agg) is a reliability factor used to improve the system's fault tolerance, and its value is calculated based on the node's fault domain information. For example, if most scanning nodes in an aggregation group are located in Availability Zone AZ-1, then candidate nodes agg located in different Availability Zones (such as AZ-2) will receive higher Reliability(agg) scores. The scheduling engine selects the node with the highest overall score Score_agg(agg) as the optimal aggregation node for that aggregation group and returns this decision to the master node.

[0040] After calculating the scores of all candidate nodes, the node with the highest comprehensive score is elected as the aggregation node of the aggregation group, which is responsible for receiving and summarizing the intermediate results of all scanning nodes in the group.

[0041] In one implementation of this application, metadata characteristics are determined based on the number of columns and indexes in the table to which the current data shard belongs. Hash sampling is performed based on the data distribution key of the current data shard, and the data distribution characteristics are determined by the skewness of the sampling results. The metadata characteristics and data distribution characteristics are then fused to determine the data complexity. The real-time load status of each candidate aggregation node is obtained. For each aggregation node, the average network latency from the source compute node where the data shard resides to the aggregation group to which that aggregation node belongs is calculated. Based on the type of query request for the current data volume, load balancing weight coefficients and network cost weight coefficients are determined. The data complexity, real-time load status, average network latency, load balancing weight coefficients, and network cost weight coefficients are input into the dynamic scheduling model to calculate the scheduling suitability.

[0042] First, the computational complexity of data sharding is assessed. For the data shard to be processed, metadata of its corresponding table is obtained from the distributed metadata service. The total number of columns and indexes of the table are extracted, normalized, and then weighted and summed to obtain a metadata feature characterizing the table's structural complexity. Simultaneously, the data distribution keys of the shard are hashed at a certain proportion to obtain a sample set of key values. The statistical skewness of this sample set is calculated to characterize the degree of data skewness within the shard. Finally, this metadata feature and the data skewness feature are fused according to preset weights to obtain a comprehensive data sharding computational complexity value, which reflects the relative computational resource overhead required to process the shard. Second, the real-time status and network information of candidate aggregation nodes are collected. The scheduling engine queries and obtains key load indicators such as CPU utilization and memory utilization of each candidate aggregation node from the real-time updated global network topology and load view, and comprehensively calculates a normalized real-time load status value. Meanwhile, for each candidate aggregation node, its aggregation group is determined, and based on the full network latency matrix, the average network latency from the source computing node where the current data shard is located to all nodes in the aggregation group is calculated as the average network latency from the source node to the aggregation group.

[0043] Secondly, the weight coefficients are dynamically determined based on the query type. When parsing query requests with the current data volume, the master node identifies the query type, such as a simple full table row count or an aggregation query with grouping conditions. For query types with very small intermediate result sets, a higher weight is assigned to the load balancing factor, while the weight of the network cost factor is reduced. Conversely, for query types that may generate a large number of intermediate result transmissions, the weight of the network cost factor is increased. The master node retrieves the corresponding load balancing weight coefficients and network cost weight coefficients from the policy library based on the identified type, and finally calculates the scheduling suitability score. The data sharding computational complexity, the real-time load status of candidate aggregation nodes, the average network latency from the source node to the target aggregation group, and the load balancing weight coefficients and network cost weight coefficients adaptive to the query type are input into the dynamic scheduling model. Through this calculation, a quantitative scheduling suitability score is output for the combination of the current data shard and each candidate aggregation node. The higher the score, the higher the overall benefit and lower the cost of sending the scan results of the current data shard to that node for aggregation. The score set of all candidate nodes serves as the direct basis for subsequent task allocation decisions.

[0044] Specifically, for each computation node i, the process of calculating its fitness score Score(i) is as follows: For a data s to be analyzed and a candidate aggregation node agg, the dynamic scheduling model, i.e., the fitness score Score(s, agg), is calculated as follows: Score(s,agg)=α(Q) * (1-Load(agg)) +β(Q) * (1 / AvgLatency(Group(agg)))+ γ* Complexity(s); Where Complexity(s) is the computational complexity factor for piecewise operations, and it is calculated as follows: Metadata characteristics: Complexity_meta = w1 * ColumnCount(s) + w2 * IndexCount(s); Wherein, ColumnCount(s): the number of columns in the table to which shard s belongs. The more columns there are, the higher the scanning and parsing cost. IndexCount(s): The number of indexes in the table to which shard s belongs. The more indexes there are, the higher the maintenance and query costs. Data distribution characteristics: Complexity_skew = Skewness(KeyHash(s)); KeyHash(s): Hash sampling based on the data distribution keys of the shards; Skewness: Calculates the skewness of the sampling results. The higher the skewness, the more skewed the data is, which may lead to uneven I / O or computation hotspots.

[0045] Final complexity: Complexity(s) = Complexity_meta + λ * Complexity_skew; Effect: Shards with high complexity will be assigned a higher Complexity(s) value, and the scheduling engine will tend to allocate them to aggregate nodes with lower load and higher performance.

[0046] α(Q) and β(Q) are both query type adaptive weights.

[0047] Strategy example: Scenario A: Simple aggregate query (SELECT COUNT(*) FROM table): Features: The intermediate result set is extremely small (usually only 1 row and 1 column).

[0048] Strategy: Significantly reduce the network latency weight β (e.g., set it to 0.1) and increase the load balancing weight α (e.g., set it to 0.6). Since network transmission costs are almost negligible at this point, tasks should be allocated purely based on computing power.

[0049] Scenario B: Grouped aggregation query (SELECT col, COUNT(*) FROM table GROUP BY col): Features: The size of the intermediate result set depends on the cardinality of col and can be very large.

[0050] Strategy: Significantly increase the network latency weight β (e.g., set it to 0.5) while maintaining a relatively high α (e.g., 0.4). This is because transmitting a large number of intermediate results across high-latency links will become a bottleneck at this point.

[0051] When generating an execution plan, the master node's SQL parser identifies the query type Q and passes the corresponding (α,β) weight pair to the dynamic scheduling engine.

[0052] `AvgLatency(Group(agg))` represents the average latency within an aggregation group. Calculation method: After determining the aggregation group `Group(agg)`, calculate the average latency from all scanning nodes within that group to the aggregation node `agg`. By selecting the node that minimizes the overall communication cost of the group as the aggregation node, rather than focusing solely on a specific node, global network efficiency is maximized.

[0053] Step 105: Based on scheduling suitability, assign the scanning tasks of each data shard to its corresponding source computing node, and specify the aggregation node to which the scanning results should be sent, in order to perform incremental task migration.

[0054] In one implementation of this application, a sharded query plan is generated based on scheduling suitability and distributed to each source compute node. The sharded query plan includes at least one of the following: a list of table shards to be scanned, the address of the target aggregation node, and query execution parameters. The source compute nodes scan their local metadata tables to determine the data volume of each shard and format the data into structured data records. When the number of data records reaches a preset sending threshold, the data records are sent to the corresponding target aggregation node.

[0055] Specifically, the master node distributes the sharded query plan to each compute node based on scheduling suitability. Each compute node receives a plan containing: a list of table shards to be scanned; the IP address and port of the aggregation node; and query execution parameters.

[0056] The specific steps for a compute node to execute a query are as follows: scan the local metadata table to obtain the physical storage location and file size of each shard; for each shard, calculate the number of data records or the total data volume; format the statistical results as key-value pairs; establish a TCP connection and send the result data to the target aggregation node determined by the scheduling policy.

[0057] To reduce cross-node data transmission overhead, the system adopts the following optimization measures: (1) Compressed transmission: The result data is compressed using LZ4, with a compression rate of 60%-70%; (2) Batch sending: After accumulating a certain number of results, send them in batches to reduce small network packets; (3) Asynchronous non-blocking I / O: Using the IOCP model to improve concurrent transmission efficiency.

[0058] In one implementation of this application, during the migration task execution, the task execution progress and load status of each source computing node are monitored in real time. When any source computing node is detected to be overloaded or experiencing execution lag, the data fragments on the source computing node that have not yet been scanned are identified as fragments to be migrated. The fragments to be migrated are rescheduled to one or more healthy computing nodes, and the aggregation node corresponding to the fragments to be migrated is notified to update the expected data reception so that subsequent results can be received on the new computing nodes; wherein, the aggregation node is configured with an idempotent reception mechanism.

[0059] Specifically, this application embodiment collects detailed execution status of each computing node in real time, and the core monitoring indicators include: Task progress: The number of data shards currently processed and the total number of shards, accurately calculating the completion percentage; Node load: Real-time tracking of CPU, memory and other resource utilization, keenly sensing abnormal load peaks during execution; Network status: Continuously monitoring real-time throughput, latency and retransmission rate of data transmission, and evaluating network link quality.

[0060] Based on the monitoring data above, the engine will automatically trigger task migration to achieve load rebalancing when it detects any of the following situations: 1. Resource overload: The CPU utilization of a certain node is consistently higher than the preset threshold, indicating that it may become a performance bottleneck; 2. Execution lag: The processing speed of a certain node is significantly lower than the average level of the cluster; 3. Network congestion: The quality of network links between nodes deteriorates, resulting in excessively low transmission rates or excessively high retransmission rates, which affects the overall progress.

[0061] Furthermore, the scheduling engine only reschedules incomplete data shards from high-load source nodes to low-load target nodes. For successfully processed shards, their computation results are persisted to shared storage, along with a unique shard ID and version number. These completed results are retained during the migration process to avoid any duplicate computations.

[0062] After triggering the migration, the scheduling engine immediately sends a link redirection instruction to the relevant aggregation nodes to update their data reception expectations: explicitly informing them that "shard S, which was originally handled by node A, will now be provided by node B".

[0063] To prevent the aggregation node from repeatedly receiving the same shard result due to network retransmission or migration command delay, the receiving interface of the aggregation node is designed to be idempotent, that is: each shard result carries a globally unique shard ID and a monotonically increasing version number; the aggregation node maintains a version record table of received shards; when a new result is received, if the shard ID already exists and the new version number is not greater than the recorded version, it is discarded directly; otherwise, it is updated and processed.

[0064] Furthermore, the migration operation employs a lightweight two-phase coordination to ensure state consistency. Specifically, in the preparation phase, the scheduling engine notifies the target node to load the metadata of the shard to be migrated and pre-allocates resources; in the commit phase, the target node returns an ACK after being ready, and the scheduling engine then formally updates the routing table and broadcasts the new link information to the aggregation node; and in the rollback mechanism, if the target node does not respond within the timeout period, the scheduling engine can choose other alternative nodes to retry or maintain the original allocation.

[0065] This application embodiment, through a closed loop of monitoring, decision-making, and incremental migration, can dynamically respond to resource fluctuations and performance imbalances during task execution, continuously optimize resource utilization without interrupting the statistical process, and ensure the overall execution efficiency and timeliness of the task.

[0066] Step 106: Integrate the local summary results of all aggregation nodes and return the integrated results to the user.

[0067] In one implementation of this application, after the aggregation node completes partial aggregation, it generates a partial aggregation result and generates a cryptographic integrity certificate based on all received scan result data. The cryptographic integrity certificate proves that the partial aggregation result is complete and unaltered. Each aggregation node sends the partial aggregation result and the cryptographic integrity certificate to the master node. The master node verifies the integrity of the partial aggregation result using the cryptographic integrity certificates reported by each aggregation node. The partial aggregation result that passes integrity verification undergoes a second aggregation to obtain a global integration result. This global integration result is used as the total data volume of the target table and returned to the user.

[0068] First, the aggregation node synchronously generates a cryptographic integrity credential upon completing partial aggregation. After receiving result data from all scanning nodes within its aggregation group, the aggregation node first performs partial aggregation on these results, calculating a preliminary partial aggregation result. Simultaneously, based on all received original scanning result data, the aggregation node calculates its cryptographic digest, for example, by calculating the Merkle root hash value of all results. This Merkle root hash value is the core component of the cryptographic integrity credential. This credential is mathematically uniquely bound to all received original data; any tampering with the original data or its aggregation result will lead to subsequent verification failure.

[0069] Secondly, the aggregator node reports the local summary result along with the integrity certificate. After generating the local result and certificate, the aggregator node constructs a structured reporting data packet. This data packet explicitly includes the following key fields: the local summary result value generated by the aggregator node, the cryptographic integrity certificate used to prove data integrity, and a metadata list explaining the original data source on which the local summary result depends. This list typically contains identifiers of all expected scan nodes. The aggregator node then sends this data packet to the master node through a secure network channel. This ensures that the master node receives the business result and also obtains the necessary evidence to independently verify the authenticity of the result.

[0070] Next, the master node performs credential verification and integrity checks on the reported results. Upon receiving the reported data packets from each aggregator node, the master node does not immediately trust the partial summary results. The master node first extracts the cryptographic integrity credentials and metadata list from the data packets. Depending on the type of credential, such as a Merkle tree root hash, the master node requests the cryptographic digest of the original scan results or the necessary Merkle tree proof path from the relevant scanning nodes. Using this supplementary information, the master node can independently recalculate or verify the validity of the credentials reported by the aggregator nodes. If the verification passes, it proves that the partial summary result reported by the aggregator node is indeed correctly summarized from all the scan node data it claims to have collected and has not been tampered with; if the verification fails, the result of the aggregator node is marked as untrustworthy.

[0071] Finally, the master node performs selective integration and returns the final result based on the verification results. After verifying the integrity of the results reported by all aggregation nodes, the master node only performs secondary aggregation on the verified and marked as trusted local aggregation results. Specifically, the master node accumulates these trusted local results according to the table identifier, merging them to obtain the total data volume of the global, final target table. For the results of aggregation nodes that fail verification, the master node can choose to trigger an alarm, record an audit log, or initiate a retry mechanism. Finally, the master node returns the global integration result as the authoritative answer to the user for this query. Through this process, the system provides efficient statistical capabilities while additionally providing cryptographic-level result integrity guarantees, improving the applicability of distributed statistical tasks in security-sensitive or high-compliance scenarios.

[0072] In one implementation of this application, an embodiment further includes a fault self-healing mechanism, which monitors the health status of nodes in real time through two layers of monitoring. The master node determines the functional activity of each node by receiving the real-time status and heartbeat feedback of each node's task execution. The topology awareness module actively sends heartbeat packets to each node every 2 seconds to monitor its network connectivity and basic survival status.

[0073] When the system detects three consecutive heartbeat timeouts or critical task execution timeouts on a node, it determines that the node has failed. The topology-aware module then updates the cluster topology view, marking the node as unavailable and isolating it from the current scheduling resource pool. Based on the new topology view, the dynamic scheduling engine immediately recalculates the global scheduling policy. The new policy bypasses the failed node and overwrites the tasks it was originally responsible for. The master node reassigns any unfinished tasks from the failed node to other healthy nodes. Simultaneously, the master node notifies relevant data aggregation nodes to adjust their data reception expectations, redirecting them to receive results from the new task execution nodes, thus ensuring the continuity of the data processing chain.

[0074] The embodiments of this application can respond quickly in the event of node failure, realize seamless migration and continuation of tasks, and maximize the continuity of operations and the overall availability of the system.

[0075] When a node rejoins the cluster after a failure, the system coordination process is as follows: the cluster's topology-aware module detects that the node has returned to online status and updates the system's topology view, reintegrating it into the pool of available node resources. New tasks subsequently started will be considered as available resources during scheduling, allowing them to participate in load balancing. For tasks that were scheduled to other nodes and are currently executing during the node failure, the system will not force them to migrate back to the original recovered node. This design avoids service jitter and performance fluctuations caused by task migration.

[0076] The above strategy ensures that background tasks such as statistics can be executed continuously without interruption in the event of temporary failures at individual nodes, thereby maintaining the integrity and smooth experience of user request processing.

[0077] Figure 3 This is a schematic diagram of the structure of a dynamic sharding scheduling device for distributed databases, provided as an embodiment of this application. Figure 3 As shown, a dynamic sharding scheduling device 200 for distributed databases includes: at least one processor 201; and a memory 202 communicatively connected to the at least one processor 201. The memory 202 stores instructions executable by the at least one processor 201, which, when executed, enable the at least one processor 201 to: construct a global network topology view corresponding to the cluster; wherein the global network topology view includes at least the real-time network latency between each computing node and the real-time load status of each computing node; receive data volume query requests, and determine the location of the target table in each computing node within the cluster based on the data volume query requests. Data sharding on nodes; based on the global network topology view, computing nodes are divided into multiple aggregation groups, and corresponding aggregation nodes are determined for each aggregation group; by dividing into aggregation groups, the network latency between nodes within the same aggregation group is kept below a preset threshold; based on a dynamic scheduling model, the scheduling suitability for sending the scan results of data shards to the corresponding aggregation nodes is determined; based on the scheduling suitability, the scan tasks of each data shard are assigned to their corresponding source computing nodes, and the aggregation nodes to which the scan results should be sent are specified for incremental task migration; the local summary results of all aggregation nodes are integrated, and the integrated results are returned to the user.

[0078] This application provides a non-volatile computer storage medium storing computer-executable instructions, which are configured to: construct a global network topology view corresponding to a cluster; wherein the global network topology view includes at least the real-time network latency between each computing node and the real-time load status of each computing node; receive a data volume query request and determine the data shards of the target table on each computing node in the cluster based on the data volume query request; divide the computing nodes into multiple aggregation groups according to the global network topology view, and determine the corresponding aggregation nodes for each aggregation group; wherein, by dividing the aggregation groups, the network latency between nodes within the same aggregation group is lower than a preset threshold; determine the scheduling suitability for sending the scanning results of the data shards to the corresponding aggregation nodes based on a dynamic scheduling model; allocate the scanning tasks of each data shard to its corresponding source computing node based on the scheduling suitability, and specify the aggregation nodes to which the scanning results should be sent for incremental task migration; integrate the local summary results of all aggregation nodes, and return the integrated results to the user.

[0079] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0080] The above descriptions are merely embodiments of this application and are not intended to limit the scope of this application. For those skilled in the art, various modifications and variations can be made to the embodiments of this application. These modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions in the embodiments of this application.

Claims

1. A dynamic sharding scheduling method for distributed databases, characterized in that, The method includes: Construct a global network topology view corresponding to the cluster; wherein, the global network topology view includes at least the real-time network latency between each computing node and the real-time load status of each computing node; Receive data volume query requests and determine the data shards of the target table on each computing node in the cluster based on the data volume query requests; Based on the global network topology view, the computing nodes are divided into multiple aggregation groups, and corresponding aggregation nodes are determined for each aggregation group; wherein, by dividing the aggregation groups, the network latency between nodes within the same aggregation group is kept below a preset threshold. Based on the dynamic scheduling model, the scheduling suitability for sending the scan results of the data shards to the corresponding aggregation nodes is determined. Based on the scheduling suitability, the scanning tasks of each data shard are assigned to its corresponding source computing node, and the aggregation node to which the scanning results should be sent is specified, so as to perform incremental task migration. The system integrates the local summary results of all aggregation nodes and returns the integrated results to the user.

2. The dynamic sharding scheduling method for distributed databases according to claim 1, characterized in that, The construction of the global network topology view corresponding to the cluster specifically includes: Construct an inter-node communication delay matrix by measuring the network round-trip delay between different computing nodes in the cluster; The available bandwidth between nodes is determined by sending sample data packets; Based on preset time intervals, real-time load metrics of each computing node are collected. Based on the inter-node communication delay matrix, the available bandwidth between nodes, and the real-time load metric, a global network topology view is constructed. The health status and load changes of each computing node are monitored, and the global network topology view is updated when a preset trigger event occurs; wherein the trigger event includes at least one of the following: a node joining or leaving the cluster, a node load exceeding a preset threshold, a sudden increase in network latency between nodes exceeding a preset ratio, and a node losing or recovering.

3. The dynamic sharding scheduling method for distributed databases according to claim 1, characterized in that, The step of dividing the computing nodes into multiple aggregation groups based on the global network topology view, and determining corresponding aggregation nodes for each aggregation group, specifically includes: Based on the full network delay matrix contained in the global network topology view, a node communication graph is constructed; wherein, the computing node is the vertex of the communication graph, and the reciprocal of the network delay between any two nodes is used as the weight of the corresponding edge in the communication graph; Run a community detection algorithm on the node communication graph and aggregate the identified node communities with connections. Each node in the aggregation group is taken as a candidate aggregation node, and the aggregation node is selected from the candidate aggregation nodes through multi-objective constraints.

4. The dynamic sharding scheduling method for distributed databases according to claim 3, characterized in that, The step of selecting the aggregation node from the candidate aggregation nodes through multi-objective constraints specifically includes: Obtain the real-time load status of each candidate aggregation node, and determine the average network latency from each other node in the aggregation group to the candidate aggregation node; Based on the fault domain isolation information between the candidate aggregation node and several other nodes in the group, the reliability factor is determined. The real-time load status, average network latency, and reliability factor corresponding to each candidate aggregation node are substituted into a preset scoring function to determine the comprehensive election score corresponding to each candidate aggregation node. The preset scoring function is used to ensure that candidate aggregation nodes whose load is proportional to the average network latency and whose load is inversely proportional to the degree of isolation of fault domains of several nodes in the group are elected with higher scores. The candidate aggregation node with the highest overall election score is determined as the aggregation node of the aggregation group.

5. A dynamic sharding scheduling method for distributed databases according to claim 1, characterized in that, The dynamic scheduling model determines the scheduling suitability for sending the scan results of the data shards to the corresponding aggregation nodes, specifically including: Metadata characteristics are determined based on the number of columns in the table to which the current data shard belongs and the number of indexes in the table to which the current data shard belongs. Hash sampling is performed based on the data distribution keys of the current data shards, and the data distribution characteristics are determined by the skewness of the sampling results; The data complexity is determined by fusing the metadata features with the data distribution features. Obtain the real-time load status of each candidate aggregation node; For each aggregation node, calculate the average network latency from the source compute node where the data shard is located to the aggregation group to which the aggregation node belongs; Based on the type of query request for the current data volume, determine the load balancing weight coefficient and the network cost weight coefficient; The data complexity, real-time load status, average network latency, load balancing weight coefficient, and network cost weight coefficient are input into the dynamic scheduling model to calculate the scheduling suitability.

6. The dynamic sharding scheduling method for distributed databases according to claim 1, characterized in that, The process of assigning the scanning task of each data shard to its corresponding source computing node specifically includes: Based on the scheduling suitability, a sharded query plan is generated and distributed to each of the source compute nodes; wherein, the sharded query plan includes at least one of the following: a list of table shards to be scanned, the address of the target aggregation node, and query execution parameters; The source computing node scans the local metadata table to determine the data volume of each shard and formats the data volume into structured data records. When the number of data records reaches a preset sending threshold, the data records are sent to the corresponding target aggregation node.

7. A dynamic sharding scheduling method for distributed databases according to claim 1, characterized in that, The incremental task migration specifically includes: During the migration task execution, the task execution progress and load status of each source computing node are monitored in real time; When any of the source computing nodes is detected to be overloaded or experiencing execution lag, the data fragments on the source computing nodes that have not yet been scanned are identified as fragments to be migrated. The shard to be migrated is rescheduled to one or more healthy compute nodes, and the aggregation node corresponding to the shard to be migrated is notified to update the expected data reception so that subsequent results can be received on the new compute node; wherein, the aggregation node is configured with an idempotent reception mechanism.

8. A dynamic sharding scheduling method for distributed databases according to claim 1, characterized in that, The process of integrating the local summary results of all aggregation nodes and returning the integrated results to the user specifically includes: After the aggregation node completes the partial aggregation, it generates a partial aggregation result and generates a cryptographic integrity certificate based on all received scan result data; wherein, the cryptographic integrity certificate is used to prove that the partial aggregation result is complete and has not been tampered with; Each aggregation node sends the local summary result and the cryptographic integrity certificate to the master node; The master node verifies the integrity of the partial summary result by using the cryptographic integrity credentials reported by each of the aggregation nodes; For the local summary results that have passed integrity verification, a second summary is performed to obtain the global integrated result; The global integration result is used as the total data volume of the target table and returned to the user.

9. A dynamic sharding scheduling device for distributed databases, characterized in that, The device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to perform the method described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are capable of performing the method described in any one of claims 1-8.

Images