A database comparison task scheduling method, device and medium

Abstract

This invention discloses a database comparison task scheduling method, device, and medium, relating to the field of electronic digital data processing technology. The method includes: determining the task complexity corresponding to the database comparison task, and the available node capabilities of multiple computing nodes used to execute the database comparison task; determining whether there is a target computing node among the multiple computing nodes capable of scheduling the database comparison task; if not, partitioning the database comparison task into several sub-database comparison tasks based on the allocation constraints formed by the available node capabilities and task complexity of each computing node; initially allocating the sub-database comparison tasks to the corresponding target computing nodes, or allocating the database comparison tasks to target computing nodes whose matching degree is less than a preset matching degree; determining whether the processing progress of the target computing node meets the preset scheduling requirements, so as to reschedule the database comparison task.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, specifically to a database comparison task scheduling method, device, and medium. Background Technology

[0002] With the continuous expansion of enterprise data scale and the widespread application of distributed systems, distributed database comparison technology plays an increasingly important role in scenarios such as data consistency verification, data migration verification, and real-time data synchronization. Generally, large-scale database comparison tasks need to be broken down into multiple sub-tasks and executed in parallel using multiple computing nodes to improve comparison efficiency.

[0003] Existing distributed database comparison systems typically employ static task sharding and simple round-robin allocation strategies, supplemented by timeout retry mechanisms to handle exceptions. However, this approach has significant drawbacks. First, static sharding simply divides the data table into a fixed number of shards according to preset rules, without considering the differences in actual processing complexity among different data tables. This can lead to some computing nodes being assigned to shards with high complexity and large data volumes, while other nodes handle lighter tasks, resulting in an unbalanced load among nodes. Consequently, overall execution efficiency is limited by the slowest node, leading to a decrease in system throughput. Second, existing database comparison task allocation does not fully consider the heterogeneity of computing nodes. Different nodes exhibit significant differences in hardware configuration, real-time load, and network status, yet the round-robin allocation strategy still distributes tasks evenly. This results in high-performance nodes being idle, while low-performance nodes run under high load for extended periods, failing to fully utilize the resource advantages of the distributed architecture and causing low resource utilization efficiency. Furthermore, when a compute node fails or experiences performance degradation, the database comparison task will repeatedly retry on the failed node, which not only prolongs the task execution time but may also cause the task to fail completely due to continuous attempts to unavailable nodes. Summary of the Invention

[0004] To address the above problems, this invention proposes a database comparison task scheduling method, comprising:

[0005] Determine the task complexity corresponding to the database comparison task, and the available node capabilities of the multiple computing nodes used to execute the database comparison task;

[0006] For each computing node, based on the matching degree between the task complexity and the available capabilities of the corresponding computing node, it is determined whether there is a target computing node among multiple computing nodes that can schedule the database comparison task;

[0007] If not, the database comparison task is sharded according to the allocation constraint relationship formed by the available node capacity value corresponding to each computing node and the task complexity, to obtain several sub-database comparison tasks.

[0008] The scheduler initially allocates the sub-database comparison task to the corresponding target computing node based on the complexity of the sub-task corresponding to the sub-database comparison task, or allocates the database comparison task to the target computing node whose matching degree is less than the preset matching degree.

[0009] Based on the matching degree corresponding to the target computing node and the baseline time required to execute the database comparison task, determine whether the processing progress corresponding to the target computing node meets the preset scheduling requirements, so as to reschedule the database comparison task.

[0010] In one implementation of the present invention, determining the task complexity corresponding to the database comparison task, and the available node capabilities of the multiple computing nodes used to execute the database comparison task, specifically includes:

[0011] The task complexity is calculated based on the data characteristics corresponding to the database comparison task; wherein the data characteristics include at least one or more of the following: data volume, field complexity, and index complexity;

[0012] The scheduler collects basic node information and node running status information for each computing node, and calculates the basic node capability and overall node load based on the basic node information and the node running status information.

[0013] Based on the relationship between the overall load of the node and the preset load, the load adjustment factor corresponding to the computing node is determined;

[0014] Obtain the historical task logs corresponding to the computing node, determine the completion rate of the historical database comparison task based on the historical task logs, and determine the stability coefficient corresponding to the computing node based on the mapping relationship between the completion rate and the stability coefficient.

[0015] The available node capacity corresponding to the computing node is calculated based on the product of the node's basic capacity, the load adjustment factor, and the stability coefficient.

[0016] In one implementation of the present invention, determining whether there is a target computing node among multiple computing nodes capable of scheduling the database comparison task, based on the matching degree between the task complexity and the available capabilities of the computing node, specifically includes:

[0017] Based on standard task capability tests performed on standard configuration nodes among multiple computing nodes, the baseline capability of the standard configuration nodes and the baseline complexity of the standard tasks are determined.

[0018] Based on the ratio between the available capabilities of the node and the baseline capabilities, the node processing capability index corresponding to the computing node is determined, and based on the ratio between the task complexity and the baseline complexity, the task complexity index corresponding to the database comparison task is determined.

[0019] The matching degree between the task complexity index and the node processing capacity index is determined based on the ratio between the task complexity index and the available node capacity corresponding to the computing node.

[0020] If there is a computing node among multiple computing nodes with a matching degree greater than a preset matching degree, the computing node is determined as the target computing node for scheduling the database comparison task.

[0021] In one implementation of the present invention, the database comparison task is partitioned according to the allocation constraint relationship formed by the available node capacity value corresponding to each computing node and the task complexity, resulting in several sub-database comparison tasks, specifically including:

[0022] Based on the allocation constraint relationship formed by the available node capacity value corresponding to each computing node and the task complexity, the sharding range corresponding to the database comparison task is determined.

[0023] The mean of the boundary values ​​in the sharding range is rounded to determine the number of shards required to shard the database comparison task, and the target number of computing nodes for the number of shards is selected from multiple computing nodes in descending order of the available capabilities of the nodes.

[0024] Determine the target task complexity to be allocated to each target computing node, and divide the database comparison task into several sub-database comparison tasks according to the proportion of the target task complexity to the total task complexity.

[0025] In one implementation of the present invention, determining the target task complexity to be allocated to each target computing node specifically includes:

[0026] The average subtask complexity corresponding to each target computing node is determined based on the ratio between the task complexity and the number of partitions.

[0027] Based on the available capabilities of the node corresponding to the target computing node, the average task complexity is corrected to obtain the corresponding proposed allocation complexity.

[0028] Based on the proposed allocation complexity, tasks are proposed to be allocated to each target computing node, and the remaining task complexity of the database comparison tasks that have not been allocated is determined according to the proposed allocation results.

[0029] The remaining task complexity is redistributed to the target computing nodes based on the product of the remaining task complexity and the proportion of additional resources corresponding to each target computing node, so as to determine the corresponding target task complexity through the proposed allocation result and the redistribution result.

[0030] In one implementation of the present invention, determining whether the processing progress of the target computing node meets the preset scheduling requirements based on the matching degree corresponding to the target computing node and the baseline time required to execute the database comparison task specifically includes:

[0031] Based on the standard task capability test performed on the standard configuration node, the average running time required for the standard configuration node to execute multiple standard tasks is used as the benchmark running time required for multiple target computing nodes to execute the database comparison task.

[0032] Determine the matching degree corresponding to the target computing node within the current monitoring period, and estimate the expected completion time of the target computing node based on the product of the matching degree, the baseline time consumption, and the preset buffer coefficient; wherein, the buffer coefficient is used to represent the error range of the expected completion time and is negatively correlated with the processing progress of the target computing node;

[0033] If the actual execution time of the target computing node exceeds the expected completion time, it is determined that the processing progress of the target computing node does not meet the preset scheduling requirements.

[0034] In one implementation of the present invention, the sub-database comparison task is initially allocated to the corresponding target computing node according to the complexity of the sub-task corresponding to the sub-database comparison task, specifically including:

[0035] The sub-database comparison tasks are sequentially assigned to target computing nodes whose corresponding target task complexity meets the sub-task complexity.

[0036] During the allocation of the sub-database comparison tasks, if the matching degree between the remaining sub-database comparison tasks and the unallocated target computing nodes is greater than 1, the remaining sub-database comparison tasks are further sharded so that the sub-database comparison tasks obtained after sharding are allocated to the unallocated target computing nodes for processing.

[0037] In one implementation of the present invention, rescheduling the database comparison task specifically includes:

[0038] Determine the timeout reason corresponding to the target computing node; wherein the timeout reason includes at least one or more of the following: node failure, node performance degradation, network failure, database connection failure;

[0039] Based on the timeout reason, the database comparison task is reassigned or timeout controlled to achieve task rescheduling of the database comparison task.

[0040] This invention provides a database comparison task scheduling device, the device comprising:

[0041] At least one processor;

[0042] And, a memory communicatively connected to the at least one processor;

[0043] The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform a database comparison task scheduling method as described above.

[0044] This invention provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0045] A database comparison task scheduling method as described in any of the preceding items.

[0046] The database comparison task scheduling method proposed in this invention can bring the following beneficial effects:

[0047] By evaluating the data volume and node availability of database comparison tasks, precise matching of tasks and resources is achieved, fundamentally eliminating the problem of uneven load distribution among nodes and preventing system throughput from being limited by performance bottleneck nodes. Secondly, by employing intelligent scheduling and dynamic sharding strategies based on matching degree, tasks can be adaptively allocated according to node heterogeneity, ensuring that high-performance nodes handle high-complexity tasks and low-performance nodes handle lightweight tasks, improving resource utilization efficiency. By monitoring task execution progress in real time and triggering differentiated rescheduling based on timeout reasons, invalid retries on faulty nodes are avoided, significantly improving system fault tolerance and task success rate. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0049] Figure 1A flowchart illustrating a database comparison task scheduling method provided in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the structure of a database comparison task scheduling device provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0052] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0053] like Figure 1 As shown in the figure, an embodiment of the present invention provides a database comparison task scheduling method, including:

[0054] S101: Determine the task complexity corresponding to the database comparison task, and the available node capabilities of the multiple computing nodes used to execute the database comparison task.

[0055] Database comparison tasks refer to the data consistency verification between two databases (or tables) in a distributed system, often used for verifying results after data migration, synchronization, or backup. Executing database comparison tasks effectively ensures data integrity and accuracy, improving database reliability. The scheduling of distributed database comparison tasks typically involves decomposing the task into several parallel subtasks, assigning them to different computing nodes for processing, and finally merging the results from each node. This traditional static task allocation method ignores the differences in the data itself and the performance differences of the nodes, resulting in uneven task distribution, uneven utilization of node resources, and poor fault tolerance.

[0056] Based on this, this invention provides a database comparison task scheduling system. By evaluating the data characteristics of the database comparison task and the resource status of multiple computing nodes executing the database comparison task, the system assesses the task complexity of the database comparison task and the available capabilities of the corresponding computing nodes. Then, based on the task complexity and node availability, a scheduling mechanism that allocates tasks according to capabilities is implemented. This accurately matches the task complexity of the database comparison task with the dynamic capabilities of the nodes, thereby achieving efficient, balanced, and reliable distributed data comparison and effectively avoiding insufficient resource utilization and uneven task allocation.

[0057] Specifically, task complexity represents the amount of computational resources and processing difficulty required during the execution of a database comparison task. It needs to be determined through the data characteristics of the database comparison task, which include at least one or more of the following: data volume, field complexity, and index complexity.

[0058] Data volume refers to the amount of table data involved in the database comparison task. It can be characterized by table size, number of rows, average row size, etc., and can be further specified by formulas. The calculation yielded that, These represent the table size (MB), number of rows, and average row size (KB), respectively. For weighting coefficients, field complexity reflects the processing difficulty of different fields. It can be obtained by quantifying the processing cost of different field types and lengths. Therefore, when determining field complexity, a formula can be used. ,in, `w` represents the field type coefficient, which is set based on the processing cost of specific data types. Generally, character fields have a higher weight than numeric fields; `w` represents the field length weight, with longer fields having a higher weight; `n` represents the number of fields in the table. Different index structures lead to differences in data retrieval performance. Therefore, index complexity can also affect the efficiency of data comparison operations. Tables with primary key indexes have the best comparison performance, while tables with composite indexes generally have lower comparison performance. Therefore, the index complexity of a primary key index is lower than that of a composite index. For the database comparison task to be performed, after determining the above data characteristics, it is necessary to follow the formula... Calculate the final task complexity In this formula, This indicates the index complexity. If the database does not have indexes, this value defaults to 1.

[0059] It should be noted that since the system's data processing capability is relatively stable and the parameters obtained in a standard environment are relatively universal, the feature values ​​of each data feature can be obtained by constructing a test dataset for benchmark testing.

[0060] Specifically, a test dataset is constructed. The data in the test dataset includes table sizes of 1MB, 10MB, 100MB, 500MB, 1GB, and 5GB; row counts of 1K, 10K, 100K, 500K, and 1M; field types of pure INT, pure VARCHAR, pure TEXT, and mixed types; and index types of no index, primary key index, and composite index. After constructing the test dataset, a database comparison task is run in a standard hardware environment, recording the actual execution time for each test table. Then, the data features are solved sequentially. When solving for the weight coefficients of each data volume indicator, a test table with pure INT fields and no indexes is used for testing, and then a regression equation is established. c is the regression coefficient, which is solved using multiple linear regression. When calculating field complexity, a test table with a fixed amount of data and no indexes is used, with the INT field as the baseline (coefficient 1.0), to calculate the relative coefficients for VARCHAR, TEXT, and CLOB. Similarly, when calculating index complexity, a test table with a fixed amount of data and field types is used, with no indexes as the baseline (coefficient 1.0), to calculate the relative coefficients for primary key indexes and composite indexes. After confirming the accuracy of the above test data through a validation set, the weight coefficients and complexity coefficients in the above formulas can be determined.

[0061] Node availability represents the capacity of each compute node to process new tasks. To avoid assigning highly complex tasks to compute nodes already under heavy load, the resource utilization of each compute node must be considered during database comparison task scheduling. The availability of a compute node changes dynamically during task execution; therefore, it is necessary to continuously monitor node availability at regular intervals.

[0062] Specifically, the scheduler collects basic node information and node running status information for each computing node. Basic node information refers to the node's static hardware configuration attributes, including the number of CPU cores and clock speed, total memory capacity and speed, disk IOPS, and network bandwidth. Node running status information reflects the dynamic resource consumption of the computing node, including CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization.

[0063] After obtaining the basic information and operational status of the nodes, the basic capabilities and overall load of each node are calculated. This process essentially involves weighted summation of each data metric. Specifically, the first weight is determined by the number of CPU cores and clock speed, the second weight by the total memory capacity and speed, and the third and fourth weights by disk IOPS and network bandwidth, respectively. The basic capabilities of the node are then obtained by weighted summation of these data metrics based on these weights. Similarly, the larger value between disk I / O utilization and network bandwidth utilization is selected as the I / O load value for the computing node. Then, the CPU utilization, memory utilization, and I / O load values ​​are weighted and summed according to their respective weights to obtain the overall load of the node.

[0064] It should be noted that the above weights have a set of default values, and the calculation process is the same as the complexity coefficients mentioned earlier, which can be obtained through regression analysis in a standard hardware environment. As the task continues to run, when the task sample size reaches a certain number, the weights can be dynamically adjusted adaptively.

[0065] Node basic capacity refers to the maximum processing performance of a computing node under ideal no-load conditions. However, in actual task processing, node basic capacity will continuously decrease as the load increases. Therefore, after determining the overall load of each computing node, in order to more clearly understand the actual available capacity of each computing node, it is necessary to introduce a load adjustment factor to characterize the proportion of the computing node's remaining capacity to process new tasks, and adjust the node basic capacity according to the current load of the computing node.

[0066] Specifically, the load adjustment factor for each computing node is determined based on the relationship between the node's overall load and the preset load. The preset load is typically set to 50%. When the node's overall load L is less than the preset load, the load adjustment factor is determined according to the formula... The load adjustment factor is calculated, and when the overall node load is greater than or equal to the preset load, according to the formula... Calculate the load adjustment factor, where b represents the buffer coefficient, typically set to 0.85. This factor is used to handle unpredictable resource fluctuations on compute nodes under high load conditions, preventing resource overload. A higher load adjustment factor indicates that the compute node is more idle and has more available resources.

[0067] When evaluating the availability of compute nodes, in addition to considering the node's inherent capabilities and load impact, it's also necessary to assess the stability of the compute node in executing database comparison tasks, i.e., whether its completion rate is acceptable. If a compute node has a low historical completion rate, it is more likely to encounter errors when executing tasks in the future. To allocate tasks to nodes with lower risk as much as possible, a stability coefficient for the compute nodes needs to be introduced to further adjust the node's availability. This allows for proactive avoidance of risky nodes with low completion rates during task scheduling, thereby improving the success rate of database comparison tasks.

[0068] Specifically, the historical task logs corresponding to the computing nodes are retrieved. These logs record the task status, indicating whether the execution was successful, failed, or timed out. Based on these logs, the completion rate of tasks compared to the historical database can be determined. Then, by establishing a mapping between the completion rate and the stability coefficient, the stability coefficient for each computing node can be determined. For example, a stability coefficient of 1 corresponds to a completion rate of 95%; 0.85 corresponds to a completion rate of 80%-95%; and 0.75 corresponds to a completion rate of less than 80%.

[0069] After obtaining the node's basic capabilities, load adjustment factor, and stability coefficient, multiply them to obtain the final result, which is the node's available capabilities.

[0070] S102: For each computing node, based on the matching degree between the task complexity and the available capabilities of the corresponding computing node, determine whether there is a target computing node among multiple computing nodes that can schedule the database comparison task.

[0071] Because the processing performance varies among computing nodes in a distributed database, maximizing resource utilization requires calculating the matching degree between the task complexity of the database comparison task and the available capabilities of each computing node. The matching degree represents the relative pressure level of a single computing node in handling the database comparison task; a higher matching degree indicates a greater inadequacy in the processing capacity of a single computing node. Therefore, based on the calculated matching degree, it can be determined whether there is a target computing node among multiple computing nodes capable of scheduling the database comparison task. In other words, the matching degree of each computing node is compared with a preset matching degree. If the matching degree is less than the preset matching degree, it means that the computing node can be scheduled to handle the database comparison task; otherwise, it cannot independently undertake the task. Generally, a preset matching degree of 0.85 is used, reserving 15% space for resource buffering.

[0072] In one embodiment, when calculating the matching degree, since the physical meanings of task complexity and node available capabilities are different, it is necessary to eliminate the difference in their units of measurement in order to establish the correspondence between task complexity and node available capabilities. Therefore, this embodiment of the invention uses a task complexity index and a node processing capability index to reflect the matching degree between task complexity and node available capabilities from the side. The above indices need to be determined based on benchmark data obtained through standard task capability testing.

[0073] Specifically, any node with an intermediate hardware configuration among multiple computing nodes is designated as the standard configuration node, and its standard task capability is tested. The standard task capability test aims to assess the baseline capability of a computing node when faced with standard tasks of moderate hardware configuration and complexity. Therefore, after obtaining stable test results through repeated testing, it is necessary to determine the baseline capability of the standard configuration node and the baseline complexity of the standard tasks. Simultaneously, it is also necessary to determine the average execution time required for this computing node to execute multiple standard tasks; this execution time serves as the baseline execution time for the database comparison task among multiple computing nodes. The standard task capability test provides a reference baseline value for each computing node, allowing for subsequent evaluation and calculation of the available capabilities of the computing nodes and the complexity of the tasks based on the baseline capability and baseline complexity.

[0074] Furthermore, for each computing node, a node processing capacity index is determined based on the ratio between its available capacity and baseline capacity. This index reflects the computing node's processing power relative to a standard configuration node. Simultaneously, a task complexity index is determined based on the ratio between the task complexity and baseline complexity. This index reflects the relative complexity of the task compared to the standard task. If both the node processing capacity index and the task complexity index are greater than 1, it indicates that the node's available capacity and the task's complexity are higher than the baseline level; conversely, they are lower than the baseline level.

[0075] Furthermore, after converting task complexity and node processing capacity into their corresponding task complexity index and node processing capacity index, the ratio between the task complexity index and the node processing capacity index is calculated. This ratio represents the matching degree between task complexity and node available capacity. If there are multiple computing nodes with a matching degree less than a preset matching degree, it indicates that the task requirement is less than the node's capacity. In this case, the database comparison task can be completed by a single computing node. This computing node is then determined as the target computing node for scheduling the database comparison task. There can be one or more target computing nodes.

[0076] S103: If not, based on the allocation constraints formed by the available node capacity value and task complexity of each computing node, the database comparison task is sharded to obtain several sub-database comparison tasks.

[0077] If the matching degree for each computing node is greater than the preset matching degree, it means that there is no single computing node capable of independently completing the database comparison task. The complexity of the current task has exceeded the limit that any single node in the cluster can handle. In this case, forcibly allocating the entire task to the most capable computing node would cause that node to be overloaded for an extended period, resulting in slow task execution, frequent timeouts, and blocking the node from receiving other new tasks, creating a bottleneck. Therefore, it is necessary to partition the database comparison task based on the allocation constraints between the available capacity of each computing node and the task complexity. By breaking down the database comparison task into several sub-database comparison tasks with lower complexity, the complexity of each sub-database comparison task can be kept within the capacity of one or more computing nodes, thus enabling the database comparison to be scheduled.

[0078] In one embodiment, the sharding strategy adopted by this invention is no longer to distribute tasks evenly, but to follow the principle of allocation based on capabilities, allocating higher-complexity sub-database comparison tasks to high-performance nodes. This non-uniform sharding method ensures that multiple computing nodes are executing tasks, achieving task load balancing and maximizing resource utilization.

[0079] Specifically, when sharding a database comparison task, the first step is to determine the sharding range corresponding to the task based on the allocation constraints formed by the available capabilities of each computing node and the task complexity. The allocation constraints limit both the upper limit of the available capabilities of the computing nodes and the upper limit of the number of task shards. During sharding, it is necessary to ensure that the task complexity processed by the most powerful high-performance node does not exceed its own capability limit, and at the same time, it is also necessary to ensure that the number of shards does not exceed the number of available nodes to avoid over-sharding.

[0080] Therefore, the allocation constraint relationship can be constructed according to the following formula, and the fragmentation range can be determined through the allocation constraint relationship, where the lower limit of the fragmentation range is expressed as: , The upper limit of the fragment range is expressed as .

[0081] The mean of the boundary values ​​within the partitioned range is rounded down, i.e. The number of shards, k, required to shard the database comparison task needs to be determined. In other words, the database comparison task needs to be divided into k sub-database comparison tasks, and to maintain load balancing across computing nodes, these sub-database comparison tasks also need to be distributed sequentially across k computing nodes for processing. Therefore, after determining the number of shards, the target computing nodes for the required number of shards need to be selected from multiple computing nodes in descending order of their available capabilities. By prioritizing high-performance nodes with strong capabilities, the task execution efficiency can be maximized.

[0082] After selecting k target computing nodes for subsequent task scheduling, only the number of tasks to be sharded is determined, while the task complexity corresponding to each shard remains undetermined. Therefore, it is necessary to determine the target task complexity to be allocated to each target computing node based on its available capabilities. Nodes with higher available capabilities will be assigned sub-database comparison tasks with higher complexity. After allocating the target task complexity to each computing node according to its capabilities, the database comparison tasks are sharded based on the ratio of target task complexity to total task complexity, resulting in several sub-database comparison tasks. Sharding by complexity ratio establishes a matching mechanism between node available capabilities and task complexity, achieving true load balancing.

[0083] In one embodiment, when determining the target task complexity to be allocated to each target computing node, the present invention adopts a two-stage allocation strategy. In the first stage, tasks are to be allocated according to the available capabilities of each target computing node to ensure that the allocated target task complexity does not exceed its available capability limit. In the second stage, the remaining tasks are allocated according to the node capability ratio to ensure that the task allocation is completed.

[0084] Specifically, based on the ratio between task complexity and the number of shards, the average subtask complexity for each target computing node is determined. Average subtask complexity refers to the amount of task each node should handle when all target computing nodes have identical capabilities and are distributed perfectly evenly. Since the available capabilities of each target computing node are different and have a safety upper limit, the average task complexity needs to be adjusted based on the available capabilities of the target computing nodes to obtain the corresponding proposed allocation complexity. Specifically, the smaller value between the product of the safety buffer coefficient and the node's available capability, and the average subtask complexity, is used as the proposed allocation complexity. By adjusting the node's available capability through the safety buffer coefficient, even if the average task complexity is high, it ensures that the amount of task allocated to any node will not exceed its maximum safe load capacity, reserving buffer resources for nodes to cope with performance fluctuations and preventing overload. Thus, a target computing node with stronger capabilities may be allocated a proposed allocation complexity equal to the average subtask complexity, while the allocation for a slightly weaker target computing node will be limited to its own capability upper limit, effectively ensuring allocation safety.

[0085] After the nodes are initially allocated, the proposed allocation complexity of all target computing nodes is summed to obtain the total allocated complexity. Then, the total allocated complexity is subtracted from the task complexity to obtain the remaining task complexity, which needs to be further allocated. Specifically, the remaining task complexity is redistributed to the target computing nodes based on the product of the remaining task complexity and the proportion of additional resources corresponding to each target computing node. The proportion of additional resources is essentially the ratio of the available capacity of a target computing node to the total available capacity of the k target computing nodes. Redistributing tasks according to the proportion of additional resources ensures that the remaining tasks are preferentially assigned to the most capable node after the initial allocation, maximizing the utilization of all nodes' idle resources. Adding the proposed allocation result and the redistribution result of each target computing node during the initial allocation phase to obtain the final target task complexity to be undertaken by that target computing node.

[0086] S104: The scheduler initially allocates the sub-database comparison task to the corresponding target computing node based on the complexity of the sub-task corresponding to the sub-database comparison task, or allocates the database comparison task to the target computing node whose matching degree is greater than the preset matching degree.

[0087] The sharding process described above only provides a preliminary allocation of the workload required by each target computing node. However, in actual task scheduling, the allocation strategy may change as tasks are executed. Therefore, whether the database comparison task is assigned to a single node or sharded and then distributed to multiple nodes, a scheduler is required to schedule and allocate the tasks. When allocating tasks, if a single node is handling the task, the scheduler simply assigns the database comparison task to the target computing node with a matching degree greater than a preset matching degree. If multiple nodes are handling the task, the scheduler needs to initially allocate the sub-database comparison tasks to the corresponding target computing nodes according to the sub-task complexity of each sub-database comparison task mentioned earlier.

[0088] In one embodiment, when the scheduler assigns sub-database comparison tasks to corresponding target computing nodes, it first attempts to assign each sub-database comparison task to the corresponding target computing node according to the pre-calculated target task complexity. This process is usually executed sequentially. The scheduler searches for the node previously selected for each sub-task based on the complexity matching algorithm to ensure that high-capacity nodes prioritize handling sub-database comparison tasks with higher loads. During the allocation process, the scheduler continuously checks whether the matching degree between the remaining sub-database comparison tasks and the unassigned target computing nodes is greater than 1. If, for a certain remaining sub-task, the real-time available capacity of all currently available nodes (i.e., nodes without assigned tasks) is insufficient to handle it, this may be due to dynamic changes in the cluster state, such as a sudden increase in the real-time load of a node causing a sharp decrease in its available capacity, or temporary network fluctuations affecting the effective capacity of the nodes. If the original sharding results are used for allocation at this point, the remaining sub-database comparison tasks will not be executed quickly. Therefore, the scheduler will activate the reallocation mechanism, re-shard the remaining sub-database comparison tasks according to the sharding process described above, and allocate the re-sharded sub-database comparison tasks to the unallocated target computing nodes for processing, thus preventing task blocking and ensuring that all database comparison tasks can be effectively scheduled and processed.

[0089] S105: Based on the matching degree corresponding to the target computing node and the baseline time required to execute the database comparison task, determine whether the processing progress corresponding to the target computing node meets the preset scheduling requirements, so as to reschedule the database comparison task.

[0090] During the execution of the database comparison task, the scheduler continuously monitors the processing progress of the assigned tasks and intervenes promptly when deviations are detected to ensure the task is ultimately completed successfully. Specifically, based on the matching degree of the target computing node and the baseline time required to execute the database comparison task, an estimated completion time is predicted. Whether the target computing node completes the task within the estimated time confirms whether its processing progress meets the preset scheduling requirements. If the scheduling requirements are not met, differentiated task rescheduling is performed based on the specific cause of the failure.

[0091] In one embodiment, after determining the baseline time required for the target computing node to execute the database comparison task, the matching degree between the target computing node and the database comparison task assigned to it within the current monitoring period is determined. This matching degree is dynamically updated; the scheduler recalculates the matching degree based on the node's current real-time availability and the complexity of the subtasks to ensure it reflects the node's latest state. Based on the product of the matching degree, the baseline time, and a preset buffer coefficient, the estimated completion time for the target computing node is predicted. The buffer coefficient represents the error range of the estimated completion time. As the task progresses, the estimated completion time actually gets closer to the actual completion time, so the error range becomes smaller. Therefore, the buffer coefficient is negatively correlated with the processing progress of the target computing node; for example, 1.5 for the first 25%, 1.3 for the 25%-75% stage, and 1.2 for the last 25% stage.

[0092] By continuously monitoring the estimated completion time of the task through the above process, if the actual execution time of the target computing node exceeds the estimated completion time, it indicates that the processing progress of the target computing node does not meet the preset scheduling requirements. The target computing node may have experienced problems such as a sharp decline in performance, severe network congestion, or underlying resource failure, requiring the scheduler to reschedule the task.

[0093] Specifically, the connectivity and performance of the target computing nodes are checked to determine the timeout reasons. Timeout reasons must include at least one or more of the following: node failure, node performance degradation, network failure, and database connection failure. Based on the timeout reason, the database comparison tasks are reassigned or timeout controlled. If it's a node failure, all tasks need to be migrated to other healthy nodes. If it's a node performance degradation, the node's capability rating needs to be lowered, and task allocation reduced. If it's a network failure, the network needs to be restored, and the task timeout time needs to be extended. If it's a database connection failure, the timeout time for all incomplete comparison tasks needs to be extended. Through this differentiated rescheduling strategy, the cost of restarting after a failure can be reduced, ensuring the successful completion of the database comparison tasks.

[0094] The above are embodiments of the method proposed in this invention. Based on the same idea, some embodiments of this invention also provide devices and non-volatile computer storage media corresponding to the above methods.

[0095] Figure 2 This is a schematic diagram of the structure of a database comparison task scheduling device provided in an embodiment of the present invention. Figure 2 As shown, it includes:

[0096] At least one processor; and,

[0097] At least one processor-communication-connected memory; wherein,

[0098] The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform a database comparison task scheduling method as described above.

[0099] This invention provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0100] A database comparison task scheduling method as described in any of the preceding items.

[0101] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are relatively simple in description because they are fundamentally similar to the method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0102] The devices, media, and methods provided in the embodiments of the present invention are one-to-one correspondences. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0103] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0105] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0106] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0107] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0108] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0109] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0110] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0111] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A database comparison task scheduling method, characterized in that, The method includes: Determine the task complexity corresponding to the database comparison task, and the available node capabilities of the multiple computing nodes used to execute the database comparison task; For each computing node, based on the matching degree between the task complexity and the available capabilities of the corresponding computing node, it is determined whether there is a target computing node among multiple computing nodes that can schedule the database comparison task; If not, the database comparison task is sharded according to the allocation constraint relationship formed by the available node capacity value corresponding to each computing node and the task complexity, to obtain several sub-database comparison tasks. The scheduler initially allocates the sub-database comparison task to the corresponding target computing node based on the complexity of the sub-task corresponding to the sub-database comparison task, or allocates the database comparison task to the target computing node whose matching degree is less than the preset matching degree. Based on the matching degree corresponding to the target computing node and the baseline time required to execute the database comparison task, determine whether the processing progress corresponding to the target computing node meets the preset scheduling requirements, so as to reschedule the database comparison task. Based on the allocation constraints formed by the available capabilities of each computing node and the task complexity, the database comparison task is partitioned to obtain several sub-database comparison tasks, specifically including: Based on the allocation constraint relationship formed by the available node capacity value corresponding to each computing node and the task complexity, the sharding range corresponding to the database comparison task is determined. The mean of the boundary values ​​in the sharding range is rounded to determine the number of shards required to shard the database comparison task, and the target number of computing nodes for the number of shards is selected from multiple computing nodes in descending order of the available capabilities of the nodes. Determine the target task complexity to be allocated to each target computing node, and divide the database comparison task into several sub-database comparison tasks according to the proportion of the target task complexity to the total task complexity. Determine the target task complexity to be allocated to each target computing node, specifically including: The average subtask complexity corresponding to each target computing node is determined based on the ratio between the task complexity and the number of partitions. Based on the available capabilities of the node corresponding to the target computing node, the average subtask complexity is corrected to obtain the corresponding proposed allocation complexity. Based on the proposed allocation complexity, tasks are proposed to be allocated to each target computing node, and the remaining task complexity of the database comparison tasks that have not been allocated is determined according to the proposed allocation results. The remaining task complexity is redistributed to the target computing nodes based on the product of the remaining task complexity and the proportion of additional resources corresponding to each target computing node, so as to determine the corresponding target task complexity through the proposed allocation result and the redistribution result.

2. The database comparison task scheduling method according to claim 1, characterized in that, Determine the task complexity corresponding to the database comparison task, and the available node capabilities of the multiple computing nodes used to execute the database comparison task, specifically including: The task complexity is calculated based on the data characteristics corresponding to the database comparison task; wherein the data characteristics include at least one or more of the following: data volume, field complexity, and index complexity; The scheduler collects basic node information and node running status information for each computing node, and calculates the basic node capability and overall node load based on the basic node information and the node running status information. Based on the relationship between the overall load of the node and the preset load, the load adjustment factor corresponding to the computing node is determined; Obtain the historical task logs corresponding to the computing node, determine the completion rate of the historical database comparison task based on the historical task logs, and determine the stability coefficient corresponding to the computing node based on the mapping relationship between the completion rate and the stability coefficient. The available node capacity corresponding to the computing node is calculated based on the product of the node's basic capacity, the load adjustment factor, and the stability coefficient.

3. The database comparison task scheduling method according to claim 1, characterized in that, Based on the matching degree between the task complexity and the available capabilities of the corresponding computing nodes, it is determined whether there is a target computing node among multiple computing nodes capable of scheduling the database comparison task, specifically including: Based on standard task capability tests performed on standard configuration nodes among multiple computing nodes, the baseline capability of the standard configuration nodes and the baseline complexity of the standard tasks are determined. Based on the ratio between the available capabilities of the node and the baseline capabilities, the node processing capability index corresponding to the computing node is determined, and based on the ratio between the task complexity and the baseline complexity, the task complexity index corresponding to the database comparison task is determined. The matching degree between the task complexity index and the node processing capacity index is determined based on the ratio between the task complexity index and the available node capacity corresponding to the computing node. If there is a computing node among multiple computing nodes with a matching degree greater than a preset matching degree, the computing node is determined as the target computing node for scheduling the database comparison task.

4. The database comparison task scheduling method according to claim 3, characterized in that, Based on the matching degree corresponding to the target computing node and the baseline time required to execute the database comparison task, determine whether the processing progress corresponding to the target computing node meets the preset scheduling requirements, specifically including: Based on the standard task capability test performed on the standard configuration node, the average running time required for the standard configuration node to execute multiple standard tasks is used as the benchmark running time required for multiple target computing nodes to execute the database comparison task. Determine the matching degree corresponding to the target computing node within the current monitoring period, and estimate the expected completion time of the target computing node based on the product of the matching degree, the baseline time consumption, and the preset buffer coefficient; wherein, the buffer coefficient is used to represent the error range of the expected completion time and is negatively correlated with the processing progress of the target computing node; If the actual execution time of the target computing node exceeds the expected completion time, it is determined that the processing progress of the target computing node does not meet the preset scheduling requirements.

5. A database comparison task scheduling method according to claim 1, characterized in that, Based on the complexity of the sub-tasks corresponding to the sub-database comparison tasks, the sub-database comparison tasks are initially allocated to the corresponding target computing nodes, specifically including: The sub-database comparison tasks are sequentially assigned to target computing nodes whose corresponding target task complexity meets the sub-task complexity. During the allocation of the sub-database comparison tasks, if the matching degree between the remaining sub-database comparison tasks and the unallocated target computing nodes is greater than 1, the remaining sub-database comparison tasks are further sharded so that the sub-database comparison tasks obtained after sharding are allocated to the unallocated target computing nodes for processing.

6. The database comparison task scheduling method according to claim 1, characterized in that, The database comparison task is rescheduled, specifically including: Determine the timeout reason corresponding to the target computing node; wherein the timeout reason includes at least one or more of the following: node failure, node performance degradation, network failure, database connection failure; Based on the timeout reason, the database comparison task is reassigned or timeout controlled to achieve task rescheduling of the database comparison task.

7. A database comparison task scheduling device, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform a database comparison task scheduling method as described in any one of claims 1-6.

8. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: A database comparison task scheduling method as described in any one of claims 1-6.

Images