Reconciliation distributed task coordination processing method supporting heterogeneous data source

Abstract

The invention provides a reconciliation distributed task coordination processing method supporting a heterogeneous data source, which comprises the following steps of: constructing a task dependency graph data structure which comprises an adjacent table and a to-be-counted data container; creating an empty queue, traversing the data to be counted, and putting the task nodes with the in-degree being zero into the queue to obtain a queue to be sorted; traversing the queue to be sorted to take out the current node, comparing the current node with the node in the adjacent table, and when the sum of the v value of the current node and one is equal to the v value of the adjacent node, putting the adjacent node into the queue to obtain a topological sorting queue; storing a to-be-executed task by using a task queue, and storing server node information by using a node array; starting a scheduling thread to obtain tasks from the task queue, and allocating task execution according to healthy nodes in the node array; starting a health detection thread to monitor a node state and update a node array; a task ID and parameters are obtained before a task is executed, the parameters are subjected to dictionary sorting and then spliced with the task ID, and Hash values generated by SHA-256 are used as unique keys for idempotent judgment.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a method for coordinating and processing distributed reconciliation tasks that supports heterogeneous data sources. Background Technology

[0002] In the field of modern information technology, task scheduling, as a core technology supporting complex business operations, plays an irreplaceable role. Task scheduling directly affects the smoothness of business processes and the overall stability of the system. However, with the expansion of business scale and the increasing complexity of scenarios, how to efficiently manage tasks and rationally allocate resources has become a pressing challenge.

[0003] Current task scheduling methods often reveal significant shortcomings when dealing with complex business needs. Most of these methods struggle to adapt to the diverse demands arising from multi-domain integration, especially when task dependencies are complex and resource requirements fluctuate greatly, easily leading to execution chaos or resource waste. More seriously, existing methods lack effective mechanisms to handle system failures or task anomalies, increasing the risk of business interruption.

[0004] A deeper analysis of the challenges in this field reveals that task dependency management is the most critical technical factor. Task dependency management refers to ensuring that tasks are executed in a predetermined order, avoiding business logic errors caused by disordered execution. Because this factor is not adequately addressed, systems often struggle when handling large-scale tasks. For example, in order processing on e-commerce platforms, inventory deduction must occur after payment confirmation. If the task order cannot be guaranteed, incorrect inventory deductions or even overselling may occur. Further, when multiple tasks involve inventory operations simultaneously, failure to accurately coordinate dependencies can lead to data inconsistencies, thereby impacting user experience and business accuracy.

[0005] Therefore, how to ensure tasks are executed in sequence and avoid data conflicts through effective task dependency management in complex business scenarios has become a critical issue that urgently needs to be addressed in the field of task scheduling. Solving this problem not only concerns the stability of system operation but also directly impacts the business efficiency and reliability of enterprises in multi-domain integrated environments. Summary of the Invention

[0006] This invention provides a method for coordinating and processing distributed reconciliation tasks that support heterogeneous data sources, mainly including: Construct a task dependency graph data structure, including an adjacency table and a container for data to be collected; Create an empty queue and iterate through the data to be collected, adding task nodes with an in-degree of zero to the queue to obtain a queue to be sorted. Traverse the queue to be sorted, retrieve the current node, compare it with the nodes in the adjacent table, and when the v value of the current node plus one equals the v value of the adjacent node, put the adjacent node into the queue to obtain the topological sort queue; Use a task queue to store tasks to be executed, and use a node array to store server node information; Start the scheduling thread to retrieve tasks from the task queue and allocate tasks for execution based on healthy nodes in the node array; A health check thread is initiated to monitor node status and update the node array. Before task execution, the task ID and parameters are obtained, the parameters are sorted lexicographically, and then concatenated with the task ID. A hash value is generated using SHA-256 as a unique key for idempotency testing. Further, the construction of the task dependency graph data structure includes: an adjacent table storing the dependency relationships between task nodes; a container for data to be analyzed storing the initial information of all task nodes, including the v-value and in-degree of each task node; traversing each task node in the container for data to be analyzed, if the in-degree of a task node is zero, then the task node is placed in an empty queue; the empty queue maintains the topological sorting order through the first-in-first-out property. Further, traversing the queue to be sorted and retrieving the current node, comparing it with nodes in the adjacent table, includes: retrieving the current node from the head of the queue to be sorted and obtaining the v-value of the current node; traversing the downstream nodes of the current node in the adjacent table, if the v-value of the downstream node is equal to the v-value of the current node plus one, then the in-degree of the downstream node is decremented by one; if the in-degree of the downstream node is zero, then the downstream node is placed in the queue to be sorted; repeating the above process until the queue to be sorted is empty, resulting in a topologically sorted queue. Furthermore, the step of using a task queue to store tasks to be executed and a node array to store server node information includes: the task queue stores tasks to be executed sequentially according to topological sorting, with each task carrying execution rules; the node array stores multiple server nodes, with each server node's information including CPU utilization, memory usage, and running status; and the scheduling thread obtains a list of healthy nodes from the node array in real time. Furthermore, the step of starting the scheduling thread to obtain tasks from the task queue and assigning tasks for execution based on healthy nodes in the node array includes: the scheduling thread obtaining tasks to be executed from the head of the task queue; traversing the node array; if a server node's running status is healthy and its CPU utilization is below a preset threshold and its memory usage meets the requirements, then the task to be executed is assigned to that server node for execution; if there are no healthy nodes, the task to be executed is kept in the task queue to wait. Furthermore, the step of starting the health detection thread to monitor the node status and update the node array includes: the health detection thread periodically traversing each server node in the node array; for each server node, obtaining the current CPU utilization, memory usage, and running status; if the CPU utilization exceeds a preset threshold, or the memory usage is abnormal, or the running status is faulty, then the server node is marked as unhealthy; the node array updates the health status of each server node in real time.Furthermore, the step of obtaining the task ID and parameters before task execution, sorting the parameters lexicographically and concatenating them with the task ID, and using SHA-256 to generate a hash value as a unique key for idempotency determination includes: obtaining the task ID and parameter list of the task to be executed; sorting the parameter list lexicographically to obtain sorting parameters; concatenating the task ID and sorting parameters into a string and calculating the hash value using the SHA-256 algorithm; querying whether the hash value exists in the preset storage; if it exists, determining that the task has been executed; if it does not exist, storing the hash value and executing the task.

[0007] Furthermore, the method for coordinating and processing reconciliation distributed tasks that support heterogeneous data sources also includes providing a unified API for each project to support developers in writing code to deploy tasks.

[0008] Furthermore, the method for coordinating and processing distributed reconciliation tasks that support heterogeneous data sources also includes: Update the task execution progress and record the task log in real time; Simultaneously mark failed tasks and the abnormal tasks corresponding to the call exception task logs; A retry mechanism list is created in real time for multiple failed and abnormal tasks. Once the task nodes in the queue to be sorted have completed their work, the task nodes on the retry mechanism list are started.

[0009] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This invention discloses a distributed task coordination method for reconciliation supporting heterogeneous data sources. It addresses the challenges of task dependency management and efficient resource allocation in complex business scenarios, particularly ensuring sequential task execution, dynamic resource balancing, and data consistency in scenarios involving the integration of tasks from multiple industries. This invention ensures ordered task execution according to dependencies by constructing a task dependency graph and topological sorting; it employs a dynamic task allocation and health monitoring mechanism to monitor node status in real time and optimize resource allocation, avoiding overload and failure impacts; and it introduces an idempotency judgment mechanism, using hash value verification to prevent duplicate task execution and ensure data consistency. For multi-scenario integration issues, this invention provides a unified interface to support task configuration and failover, combining priority strategies and predictive failover to ensure priority execution of critical business tasks and system stability. Ultimately, this invention achieves efficient scheduling and rational resource utilization for large-scale tasks, significantly improving the system's adaptability, reliability, and execution efficiency in complex business environments, and providing comprehensive support for cross-domain task management. Attached Figure Description

[0010] Figure 1 This is a flowchart of a distributed task coordination and processing method for reconciliation that supports heterogeneous data sources, according to the present invention. Figure 2This is a schematic diagram of a specific operation process in a distributed task coordination and processing method for reconciliation that supports heterogeneous data sources according to the present invention. Figure 3 This is a schematic diagram of the structure of a storage medium for applying the distributed task coordination and processing method for reconciliation of heterogeneous data sources provided by the present invention. Detailed Implementation

[0011] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] The technical solutions of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings of the embodiments. The described embodiments are merely some embodiments of the present invention.

[0013] like Figure 1 As shown, this embodiment of a method for coordinating and processing distributed reconciliation tasks that supports heterogeneous data sources may specifically include: This invention provides a method for coordinating and processing distributed reconciliation tasks that support heterogeneous data sources, aiming to solve the problems of large-scale task dependency management and efficient resource allocation. The implementation process of this invention is described in detail below with reference to specific embodiments to make the objectives, technical solutions, and advantages of this invention clearer.

[0014] In one embodiment, the distributed task coordination and processing method for reconciliation supporting heterogeneous data sources provided by this invention mainly addresses scheduling needs in complex task dependency scenarios. By constructing a task dependency graph, performing topology sorting, dynamically allocating tasks, and ensuring the idempotency of task execution, it achieves efficient task execution and rational resource utilization. This method is applicable to large-scale distributed systems, such as enterprise-level data processing platforms and cloud computing task scheduling systems. The following will describe the specific implementation of each step according to the logical flow of the method.

[0015] Step S1 involves constructing a task dependency graph data structure, including an adjacency table and a container for data to be collected. Specifically, the task dependency graph is a directed acyclic graph used to represent the dependencies between tasks. The adjacency table, as a storage structure, stores the direct downstream tasks of each task node, i.e., the list of tasks that depend on the current task. The container for data to be collected stores the initial information of all task nodes, including the identifier of each task node, its initial in-degree value, and an auxiliary value for sorting (hereinafter referred to as the v value). The in-degree value represents the number of upstream tasks that depend on the current task, and the v value is used to determine the hierarchical relationship of tasks during subsequent topology sorting.

[0016] In one possible implementation, when constructing the task dependency graph data structure, the dependency data of the tasks first needs to be read from the task configuration file or database. For example, in a data processing platform, suppose there are 10 tasks to be scheduled, where task A depends on tasks B and C, and task D depends on task A. For this dependency relationship, the system will record tasks B and C as upstream tasks of task A, and set the in-degree value of task A to 2. Simultaneously, the system will initialize the v value of each task to 0, as the baseline value for subsequent sorting. In the adjacent tables, the records of task B and task C will each point to task A, indicating that task A is their downstream task. The data collection container stores the in-degree value and v value of each task in key-value pairs for easy subsequent traversal and updates.

[0017] It's important to note that when constructing the task dependency graph, the system checks for circular dependencies. If a circular dependency is found—for example, task A depends on task B, and task B depends on task A—the system throws an exception and prompts the user to adjust the task configuration to ensure the task dependency graph is a directed acyclic graph (DAG). This checking mechanism effectively prevents task scheduling from getting stuck in an infinite loop, ensuring the correctness of subsequent topology sorting. Once constructed, the task dependency graph data structure provides the foundation for subsequent queue initialization and sorting.

[0018] In step S11, when constructing the adjacent table and the container for data to be collected, different storage methods can be selected based on the task scale. For example, when the number of tasks is small, in-memory lists and dictionaries can be used to store the adjacent table and the container for data to be collected. However, when the number of tasks reaches hundreds of thousands or even millions, a distributed storage system can be used to shard the adjacent table and the container for data to be collected across multiple nodes to improve read efficiency. For instance, in an enterprise-level data analysis platform, task dependencies may involve multiple business modules, and the task dependency data of each module is stored on different servers. The system reads this data through a unified interface and constructs a global task dependency graph. This approach can significantly improve the parallelism of data processing and shorten the time required to construct the task dependency graph.

[0019] Step S12, regarding the construction of the task dependency graph, can also introduce a verification mechanism to ensure data integrity and consistency. For example, when building the adjacency table, the system checks whether each task node has been correctly recorded. If a task node is found to have neither upstream nor downstream tasks, it will be marked as an orphan task, and the scheduling administrator will be notified for processing. This verification mechanism can avoid task omissions due to data entry errors and ensure the integrity of the scheduling process. In addition, the system can also record the construction time and data scale of the task dependency graph during the construction process, which is convenient for subsequent performance optimization and problem troubleshooting.

[0020] It's important to note that constructing the task dependency graph is not merely about initializing the data structure; it also lays the foundation for subsequent task sorting and scheduling. By appropriately designing the adjacency table and the container for data to be collected, the system can quickly locate critical task nodes within complex task dependencies, thereby improving overall scheduling efficiency. In practical applications, this construction method can effectively handle scenarios with a surge in the number of tasks, ensuring system stability under high load.

[0021] Step S2: Create an empty queue and iterate through the data to be analyzed, adding task nodes with an in-degree of zero to the queue to obtain the unsorted queue. Specifically, the empty queue is a first-in, first-out (FIFO) data structure used to store task nodes to be sorted. The system iterates through all task nodes in the data container, checking the in-degree value of each task node. If the in-degree of a task node is 0, it means that the task has no upstream dependencies and can be executed directly; therefore, it is added to the empty queue, forming the unsorted queue.

[0022] In one possible implementation, suppose a distributed data processing system stores information about 100 task nodes in a container for the data to be analyzed. When the system traverses these nodes, it finds five task nodes with an in-degree of 0. These task nodes may be data acquisition tasks or initial preprocessing tasks. The system sequentially places these five task nodes into an empty queue, forming a queue to be sorted. The first-in, first-out (FIFO) property of the empty queue ensures that tasks without dependencies are processed first during subsequent task sorting, providing a basis for topological sorting.

[0023] Step S21: When traversing the container for data to be collected, priority filtering can be performed based on the attributes of the task nodes. For example, in some business scenarios, some tasks with an in-degree of 0 may have higher priority, such as system initialization tasks or critical data loading tasks. During traversal, the system can prioritize placing high-priority task nodes into an empty queue to ensure these tasks are executed first in subsequent scheduling. For example, in a financial data processing platform, the market data loading task has a higher priority than the log cleanup task; even if both have an in-degree of 0, the system will prioritize placing the market data loading task into an empty queue. This priority filtering mechanism can effectively improve the response speed of critical tasks.

[0024] Step S22, regarding the initialization of the empty queue, can also introduce a parallel processing mechanism to improve efficiency. For example, when there are many task nodes, the system can divide the container for data to be collected into slices and distribute them to multiple threads for parallel traversal. Each thread is responsible for checking the in-degree values ​​of a portion of the task nodes and placing task nodes with an in-degree of 0 into a shared empty queue. To avoid thread conflicts, the system can introduce a locking mechanism or atomic operations to ensure that task nodes are not added repeatedly. This parallel processing method can significantly shorten the queue initialization time in scenarios with a large task scale.

[0025] It's important to note that forming the queue to be sorted is the first step in topology sorting, and its purpose is to identify task nodes that can be executed immediately. By prioritizing task nodes with an in-degree of 0 in the queue, the system ensures that subsequent task sorting processes conform to task dependencies. This approach is not only logically clear but also effectively reduces task waiting time and improves overall scheduling efficiency in practical applications.

[0026] Step S3: Traverse the queue to be sorted, retrieve the current node, and compare it with the nodes in the adjacent table. When the v value of the current node plus one equals the v value of the adjacent node, put the adjacent node into the queue to obtain the topological sorting queue.

[0027] Specifically, the system retrieves the current task node from the head of the unsorted queue, obtains its v value, and iterates through the downstream nodes connected to the current node in the adjacent table. If the v value of a downstream node is equal to the v value of the current node plus 1, it means that the downstream node is adjacent to the current node in the dependency level, and its in-degree value is decremented by 1. If the in-degree value of a downstream node becomes 0, it means that all its upstream tasks have been processed, and it can be added to the unsorted queue. This process is repeated until the unsorted queue is empty, ultimately forming a topologically sorted queue.

[0028] In one possible implementation, assume there is a task node in the unsorted queue with a v value of 0. The system finds two downstream nodes connected to this node in the adjacency list, with v values ​​of 1 and 2 respectively. The system finds that the v value of one of the downstream nodes is equal to the v value of the current node plus 1, i.e., 1, so it decrements the in-degree of this downstream node by 1. Assuming the in-degree of this downstream node was originally 1, it becomes 0 after decrementing, and the system adds it to the unsorted queue. The v value of the other downstream node does not meet the condition, so it is not processed. The system continues to take the next task node from the unsorted queue and repeat the above operation, eventually forming a topologically sorted queue that conforms to the dependency order.

[0029] See Figure 2In step S31, a hierarchical marking mechanism can be introduced to optimize sorting efficiency during the topology sorting process. For example, when traversing adjacent tables, the system can dynamically update the v value for each task node to ensure that the v value accurately reflects the task dependency hierarchy. If the v value of a task node has not yet been set, the system can infer it based on the v value of its upstream task. For example, in a data processing workflow, if the v value of the data cleaning task is 0, the v value of its downstream data analysis task should be 1, and the v value of the next downstream data visualization task should be 2. Through this hierarchical marking, the system can quickly determine the dependencies between tasks and reduce unnecessary traversal operations.

[0030] Step S32, regarding the generation of the topology sorting queue, can also introduce an exception handling mechanism to handle special cases. For example, when traversing the queue to be sorted, if it is found that the in-degree value of a certain task node cannot be reduced to 0, it may be due to undetected circular dependencies or data errors in the task dependency graph. The system will mark the task node as an exception, suspend its subsequent processing, and notify the scheduling administrator to investigate. This exception handling mechanism can effectively avoid scheduling failures caused by data problems and ensure the stability of the system.

[0031] It's important to note that generating the topology-sorted queue is a core step in task scheduling, ensuring tasks are executed in an ordered manner according to their dependencies. By dynamically updating the in-degree and v-values ​​of task nodes, the system can accurately identify executable task nodes and add them to the queue. This approach is particularly crucial in complex task dependency scenarios, effectively preventing disordered task execution.

[0032] In one possible implementation, the generation of the topology sorting queue can also be optimized by incorporating task priorities. The system will prioritize adding urgent report generation tasks to the sorting queue to ensure their prompt execution. This priority optimization mechanism can significantly improve the timeliness of task response in critical business scenarios.

[0033] Step S33: For the generation process of the topology sorting queue, visualization tools can be introduced to facilitate debugging and monitoring. For example, the system can record the generation process of the topology sorting queue as a log file and generate a visual view of the task dependency graph, showing the hierarchical relationship and sorting order of each task node. The scheduling administrator can quickly understand the execution order of tasks and troubleshoot potential problems through the visual view.

[0034] It's important to note that the generation process of the topological sorting queue involves more than just arranging tasks in sequence; it also lays the foundation for subsequent task allocation and execution. Through well-designed sorting algorithms and exception handling mechanisms, the system can accurately identify executable tasks amidst complex task dependencies, ensuring a smooth scheduling process. In practical applications, this approach effectively improves task execution efficiency and reduces resource waste caused by incorrect dependencies.

[0035] In one possible implementation, the generation of the topology sorting queue can be optimized by combining it with a distributed computing framework. For example, when there are many task nodes, the system can shard the queue to be sorted and the adjacent tables across multiple computing nodes, with each node responsible for handling the sorting operations of a portion of the task nodes. After processing, the results from each node are aggregated to form a global topology sorting queue. To avoid data consistency issues, the system can introduce distributed locks or version control mechanisms to ensure that state updates of each task node do not conflict. This distributed processing approach can significantly improve sorting efficiency in scenarios with large task scales.

[0036] Step S34: Regarding the generation of the topology sorting queue, a dynamic adjustment mechanism can be introduced to adapt to real-time changes in task dependencies. For example, in some business scenarios, task dependencies may change due to external factors, such as the addition or deletion of tasks. During the topology sorting process, the system can monitor updates to task dependency data in real time. If a change is detected, the current sorting operation is paused, the task dependency graph is reconstructed, and the queue to be sorted is updated. This dynamic adjustment mechanism can effectively cope with dynamic changes in task dependencies and ensure the accuracy of scheduling results.

[0037] It's important to note that generating the topology sorting queue is a crucial step in coordinating distributed reconciliation tasks across heterogeneous data sources. Its accuracy and efficiency directly impact the effectiveness of subsequent task allocation and execution. By introducing mechanisms such as priority optimization, distributed processing, and dynamic adjustment, the system can flexibly handle the complexity of task dependencies in different business scenarios, ensuring smooth task scheduling. This approach significantly improves the system's adaptability and stability in practical applications, providing a reliable guarantee for large-scale task scheduling.

[0038] Step S35: Regarding the generation of the topology sorting queue, a task grouping mechanism can be introduced to improve scheduling flexibility. For example, in some business scenarios, task nodes can be grouped according to business modules or execution environments. When generating the topology sorting queue, the system prioritizes processing task nodes within the same group to reduce the switching costs of cross-group tasks. For instance, in a multi-tenant cloud computing platform, the system can group task nodes belonging to the same tenant and prioritize the generation of the tenant's topology sorting queue to ensure that its tasks enter the execution phase as soon as possible. This grouping mechanism can effectively improve the targeting of task scheduling and reduce resource contention.

[0039] Step S36: Regarding the generation of the topology sorting queue, a weighting mechanism for task dependencies can be introduced to optimize the sorting results. For example, in some business scenarios, the dependencies between tasks may have different importance. The system can assign weights to each dependency edge, and dependencies with higher weights will be processed first. When traversing the adjacent table, the system prioritizes checking downstream nodes with higher weights to ensure that critical dependencies are processed as quickly as possible.

[0040] In one possible implementation, the generation of the topological sorting queue can also be optimized in conjunction with the resource requirements of the tasks. For example, the system can adjust the position of tasks in the queue based on their resource requirements, such as computing or storage resources, prioritizing tasks with lower resource requirements to reduce resource contention.

[0041] Step S37: For the generation of the topology sorting queue, a task failure retry mechanism can be introduced to improve system fault tolerance. For example, during the generation of the topology sorting queue, if a task node cannot process normally due to data errors or other reasons, the system will mark it as a failed state and attempt to recalculate its in-degree and v values. If the retry still fails, the system will suspend the processing of that task node and notify the scheduling administrator to intervene. This retry mechanism can effectively prevent the entire sorting process from failing due to a problem with a single task node, ensuring system stability.

[0042] Step S4 involves using a task queue to store tasks to be executed and a node array to store server node information. Specifically, the task queue is a first-in, first-out (FIFO) data structure used to store tasks in the order they are topologically sorted. Each task carries specific execution rules, such as the task's execution time window and resource requirements. The node array stores the status information of multiple server nodes, including key metrics such as current CPU usage, memory usage, and running status. Through the collaborative work of the task queue and the node array, the system provides data support for subsequent task allocation and execution.

[0043] In one possible implementation, the task queue initialization process is based on the topology-sorted queue generated in the aforementioned steps. The system transfers task nodes from the topology-sorted queue to the task queue one by one, ensuring that tasks are arranged in dependency order. For example, in a data processing platform, the topology-sorted queue contains three tasks: data acquisition, data cleaning, and data analysis. The system stores them in the task queue in this order and attaches execution rules to each task. For instance, the data acquisition task needs to be executed every morning, and the data cleaning task requires at least two computing units. The node array records the status of all server nodes in the system. For example, a server node's CPU utilization is 30%, memory utilization is 40%, and its operating status is normal. The system obtains the health information of the server nodes in real time through the node array, providing a basis for task allocation.

[0044] Step S41: Regarding the storage method of the task queue, different implementation methods can be selected according to the task scale. For example, when the number of tasks is small, the task queue can be directly stored in memory to improve access speed. When the number of tasks is large, the system can store the task queue in a distributed caching system, improving read and write efficiency through sharded storage. For example, the task queue may contain tens of thousands of task nodes, and the system can shard and store them across multiple cache nodes, with each cache node responsible for storing a portion of the task data, thereby avoiding single-point performance bottlenecks. This storage method can significantly improve the processing capacity of the task queue.

[0045] Step S42, regarding the storage and updating of the node array, a hierarchical management mechanism can be introduced to improve efficiency. For example, the system can group server nodes according to geographical location or business module, and record the summary and detailed information of the node status of each group in the node array. When allocating tasks, the system first selects a suitable node group based on the business attributes of the task, and then selects a specific server node from that group. It should be noted that the construction of the task queue and node array lays the foundation for subsequent task allocation and execution. Through reasonable design of the storage structure and update mechanism, the system can quickly obtain task and node information in a complex distributed environment, ensuring smooth task scheduling. This approach can effectively improve resource utilization and reduce task waiting time in practical applications.

[0046] Step S5: The scheduling thread is started to retrieve tasks from the task queue and allocate tasks for execution based on healthy nodes in the node array. Specifically, the scheduling thread is a continuously running background process responsible for retrieving tasks to be executed from the head of the task queue and allocating them to suitable healthy nodes for execution based on the server node status information in the node array. The criteria for determining a healthy node include normal operating status, CPU utilization below a preset threshold, and memory usage meeting task requirements. If no healthy node meets the conditions, the system will keep the task in the task queue, awaiting subsequent allocation.

[0047] In one possible implementation, after the scheduling thread retrieves a task from the task queue, it iterates through the node array, checking the running status, CPU utilization, and memory usage of each server node. Assuming a preset CPU utilization threshold of 70% and a requirement of at least 50% free memory, the scheduling thread will select a server node that meets these conditions to execute the task.

[0048] For example, in a data processing platform, the scheduling thread retrieves data cleaning tasks from the task queue. After traversing the node array, it finds three server nodes that meet the criteria. The system then selects the node with the lowest CPU utilization to execute the task. If no node meets the criteria, the data cleaning task will be temporarily held in the task queue, awaiting the next scheduling iteration.

[0049] Step S51: Regarding the task allocation process, a load balancing mechanism can be introduced to optimize resource utilization. For example, when allocating tasks, the system can comprehensively consider the current and historical load of server nodes, prioritizing the execution of tasks on nodes with lower loads. For instance, in the data processing workflow of an e-commerce platform, the scheduling thread retrieves order data processing tasks from the task queue. When traversing the node array, it finds multiple healthy nodes that meet the criteria. Based on the load records of the past hour, the system selects the node with the lowest load to execute the task. This load balancing mechanism can effectively prevent certain nodes from becoming overloaded, ensuring the stability of the overall system performance.

[0050] In one embodiment, the scheduling thread retrieves multiple tasks from the task queue. After traversing the node array, it finds five server nodes running normally with CPU utilization below 70% and memory usage meeting requirements. The system selects the node with the lowest CPU utilization to execute the task. If no healthy node is found when a subsequent transportation route planning task is retrieved, the system will retain the task in the task queue, waiting for the next scheduling. This approach ensures the rationality of task allocation and avoids resource waste.

[0051] Step S52: To address task allocation failures, a retry mechanism can be introduced to improve system fault tolerance. For example, if the scheduling thread finds no healthy nodes when allocating a task, the system will retain the task in the task queue and retry the allocation in the next scheduling cycle. Simultaneously, the system can set a maximum number of retries; if the maximum is reached and allocation still fails, the system will mark the task as failed and notify the scheduling administrator for handling.

[0052] It's important to note that the task allocation process of the scheduling thread is one of the core components supporting the coordinated processing of distributed reconciliation tasks from heterogeneous data sources, and its efficiency and accuracy directly impact the task execution results. By introducing load balancing and retry mechanisms, the system can rationally allocate tasks in resource-constrained scenarios, ensuring smooth task execution. This approach significantly improves system stability and resource utilization efficiency in practical applications.

[0053] For task retry methods, the following implementation methods can also be adopted: The method for coordinating and processing reconciliation distributed tasks that supports heterogeneous data sources also includes task retry methods: Update the task execution progress and record the task log in real time; Simultaneously mark failed tasks and the abnormal tasks corresponding to the call exception task logs; A retry mechanism list is created in real time for multiple failed and abnormal tasks. Once the task nodes in the queue to be sorted have completed their work, the task nodes on the retry mechanism list are started.

[0054] Step S6: Start the health check thread to monitor node status and update the node array. Specifically, the health check thread is a periodically running background process responsible for traversing each server node in the node array and obtaining information such as its current CPU usage, memory usage, and running status. If the CPU usage of a server node exceeds a preset threshold, or its memory usage is abnormal, or its running status is faulty, the system will mark that node as unhealthy. The node array will be updated in real time with the health status of each server node, providing the scheduling thread with the latest node information.

[0055] In one possible implementation, a health check thread runs every 30 seconds, traversing all server nodes in the node array and checking the running status of each node. Assuming a CPU utilization threshold of 80%, if a node's CPU utilization reaches 85%, memory utilization exceeds 90%, or a heartbeat detection indicates the node is offline, the system marks the node as unhealthy and updates the record in the node array. For example, in a data processing platform, if the health check thread finds that a server node's CPU utilization is consistently above the threshold, the system marks it as unhealthy, and the scheduling thread will no longer consider this node in subsequent task allocation until its status returns to normal.

[0056] Step S61: For the health check process, a multi-dimensional detection mechanism can be introduced to improve accuracy. For example, the system can not only check CPU usage and memory usage, but also combine indicators such as network latency and disk read / write speed to comprehensively judge the health status of nodes. For instance, in a financial data processing platform, if the health check thread finds that the CPU usage of a server node is normal, but the network latency exceeds a preset threshold, the system will still mark it as unhealthy to avoid delays caused by network problems during task execution. This multi-dimensional detection mechanism can more comprehensively evaluate the node status and improve system reliability.

[0057] Step S62, regarding the updating of health check results, a recovery mechanism can be introduced to improve system flexibility. For example, if a server node is marked as unhealthy, and the health check thread finds that its status has returned to normal in subsequent checks, the system will remark it as healthy and update the record in the node array. For instance, if a server node is marked as unhealthy due to temporary overload, but subsequent checks find that its CPU utilization has dropped to 50% and memory usage has returned to normal, the system will remark it as healthy and allow it to participate in task allocation again. This recovery mechanism can effectively utilize system resources and avoid resource waste caused by temporary failures.

[0058] It's important to note that the health check thread is crucial for coordinating distributed reconciliation tasks across heterogeneous data sources. Its purpose is to monitor the server node status in real time, ensuring the accuracy and reliability of task allocation. Through multi-dimensional detection and recovery mechanisms, the system can dynamically adjust node states in complex distributed environments, ensuring smooth task execution. This approach significantly improves the system's fault tolerance and stability in practical applications.

[0059] Step S7: Before task execution, obtain the task ID and parameters. Sort the parameters lexicographically and concatenate them with the task ID. A hash value is generated using a specific hash algorithm and used as a unique key for idempotency testing. Specifically, before task execution, the system first obtains the task ID and parameter list of the task to be executed. Then, it sorts the parameter list lexicographically to obtain the sorted parameter content. Next, it concatenates the task ID and the sorted parameter content into a string and calculates the hash value using a specific hash algorithm. The system checks if the hash value exists in the preset storage. If it exists, the task has already been executed and will not be repeated; if it does not exist, the system stores the hash value and executes the task.

[0060] It should be noted that during task execution, we can obtain the task ID and parameters. In this embodiment of the invention, all parameters are sorted lexicographically, the task ID is added, and the concatenated string is then encrypted using SHA-256 to obtain an encrypted string. This encrypted string is a unique value that can be used as a basis for idempotency testing. SHA-256 is chosen because of its high security, extremely low collision probability of 2^-256, faster computation speed than SHA-512, and native JDK support, eliminating the need for additional third-party libraries, ensuring security and reliability.

[0061] SHA-256 is a cryptographic hash function whose output is usually called a hash or digest. Many people in the field refer to hashing as "encryption," meaning that encrypting a concatenated string using SHA-256 yields an encrypted string, which is essentially the hash value.

[0062] Step S71: For the dictionary sorting process of parameters, custom sorting rules can be introduced to adapt to different business needs. For example, in some business scenarios, parameter fields may have different priorities. The system can sort according to field priority, rather than simple dictionary order. For instance, in the data processing flow of an e-commerce platform, the order number field has higher priority than the timestamp field. The system prioritizes the order number field when sorting to ensure the stability of hash value generation. This custom sorting rule can effectively cope with complex business scenarios and improve the accuracy of idempotency judgment.

[0063] Step S72: For the storage and retrieval of hash values, a distributed storage mechanism can be introduced to improve efficiency. For example, when there are many tasks, the system can store hash values ​​in a distributed cache system, improving query speed through sharding. For instance, in an advertising platform, the system processes hundreds of thousands of task execution records daily, with hash values ​​stored on multiple cache nodes, each responsible for storing a portion of the hash value data. When querying a hash value, the system locates the corresponding cache node based on the hash value of the task ID, thus quickly determining whether the task has been executed. This distributed storage mechanism can significantly improve the efficiency of idempotency determination.

[0064] It's important to note that idempotency checks are crucial for coordinating distributed reconciliation tasks across heterogeneous data sources. Their purpose is to prevent resource waste and business errors caused by duplicate task execution. By hashing the task ID and parameters, the system can accurately identify executed tasks, ensuring their uniqueness. This approach significantly improves system reliability and data consistency in practical applications, guaranteeing smooth business processes.

[0065] Step S73: For the idempotency determination process, an expiration mechanism can be introduced to optimize storage space. For example, the system can set an expiration time for stored hash values, automatically deleting them after expiration to prevent unlimited storage growth. For instance, the system can set a 7-day expiration time for each hash value, automatically deleting unused hash value data after 7 days, thereby freeing up storage space. This expiration mechanism can effectively control storage costs while ensuring the accuracy of idempotency determination.

[0066] Step S74, regarding the selection of the hash algorithm for idempotency determination, can be adjusted according to business needs. For example, in scenarios with high data security requirements, the system can choose a hash algorithm with higher computational complexity to improve security; while in scenarios with high performance requirements, the system can choose a hash algorithm with faster computation speed to improve efficiency. For instance, in a financial data processing platform, the system selects a hash algorithm with higher security to calculate the hash value, ensuring that task execution records cannot be maliciously tampered with. This flexible algorithm selection mechanism can effectively adapt to the needs of different business scenarios.

[0067] It should be noted that the idempotency judgment mechanism is designed with full consideration of the complexity of distributed systems and the variability of business requirements. Through reasonable parameter sorting and hash calculation, the system can accurately identify duplicate tasks, ensuring the uniqueness of task execution. Meanwhile, through distributed storage and expiration mechanisms, the system can operate efficiently in high-concurrency scenarios, providing a reliable guarantee for large-scale task scheduling.

[0068] Step S8, for the overall implementation of the distributed task coordination and processing method for reconciliation supporting heterogeneous data sources, also provides a unified interface for each project, supporting developers to write code to deploy tasks and establish task dependencies. Specifically, the system provides a set of standardized interfaces through which developers can define the execution logic, dependencies, and execution rules of tasks. The system also supports basic failover functionality; for example, when a server node fails, the system automatically reassigns tasks to other healthy nodes for execution. Furthermore, idempotency protection is maintained by the access party, and the scheduling algorithm is fixed, ensuring the stability and predictability of the system.

[0069] In one possible implementation, the system provides a unified interface that includes a task creation interface, a dependency setting interface, and a task execution rule configuration interface. Developers define basic task information, such as task name and execution script, through the task creation interface; define dependencies between tasks, such as task A depending on task B, through the dependency setting interface; and set the task's execution time window, resource requirements, etc., through the task execution rule configuration interface.

[0070] Step S81: To improve development efficiency, a templated configuration mechanism can be introduced for the use of the unified interface. For example, the system can provide a variety of task configuration templates, allowing developers to select the appropriate template based on business needs and quickly complete task definition and dependency settings.

[0071] Step S82: For the failover function, a priority transfer mechanism can be introduced to optimize resource allocation. For example, when a server node fails, the system will prioritize high-priority tasks and transfer them to other healthy nodes for execution.

[0072] It's worth noting that the unified interface and failover functionality are designed with developer convenience and system fault tolerance in mind. Through standardized interfaces and flexible configuration mechanisms, the system supports task scheduling needs across various business scenarios, ensuring smooth task deployment and execution. This approach significantly improves system usability and stability in practical applications, providing developers with reliable task scheduling support.

[0073] Step S83: To address the scalability of the unified interface, a version control mechanism can be introduced to support feature iteration. For example, the system sets a version number for the unified interface; when the interface functionality is upgraded, developers can choose to use the new version or continue using the old version.

[0074] Step S84: For the use of the unified interface, a permission management mechanism can be introduced to enhance system security. For example, the system can restrict a developer's access to task configuration and execution rules based on their role and permissions. It should be noted that the design and implementation of the unified interface fully considers the system's usability and security. Through mechanisms such as templated configuration, version control, and permission management, the system can support task scheduling needs in different business scenarios, ensuring smooth task deployment and execution. This approach significantly improves the system's adaptability and reliability in practical applications, providing developers with flexible and secure task scheduling support.

[0075] Step S85: To further optimize system performance, a predictive failover mechanism can be introduced to support the failover function. For example, the system can predict potential node failures based on historical data and node status trends, and transfer tasks to other healthy nodes before a failure occurs. For instance, in an online payment platform, if the system predicts that a server node may fail due to excessive load, it can proactively transfer the transaction data entry task currently being executed on that node to other healthy nodes, ensuring that task execution is unaffected. This predictive failover mechanism effectively reduces the impact of failures on task execution and improves the overall stability of the system.

[0076] Step S86: For the use of the unified interface, a logging mechanism can be introduced to facilitate debugging and monitoring. For example, the system can record detailed operation logs of developers creating and configuring tasks through the unified interface, including task name, dependencies, execution rules, and other information. Scheduling administrators can quickly understand the task configuration status and troubleshoot potential problems through these logs. For instance, in a news content distribution platform, the system records operation logs of developers creating news data collection tasks and content classification tasks. Administrators can use these logs to discover that the resource requirements for the content classification task were configured too high, causing task allocation failure. Timely adjustments to the configuration ensure the task executes correctly. This logging mechanism effectively improves system maintainability.

[0077] It should be noted that the design of the unified interface and failover function fully considers the system's usability and fault tolerance. Through various optimization mechanisms, the system can support task scheduling requirements in different business scenarios, ensuring smooth task deployment and execution. This approach can significantly improve the system's adaptability and reliability in practical applications, providing developers with flexible and efficient task scheduling support.

[0078] Step S87: For the overall implementation of distributed task coordination and processing for reconciliation supporting heterogeneous data sources, a monitoring and alarm mechanism can be introduced to improve system maintainability. For example, the system can monitor key indicators such as the length of the task queue, the success rate of task execution, and the health status of server nodes in real time. When an indicator exceeds a preset range, the system will automatically generate an alarm message and notify the scheduling administrator.

[0079] Step S88: For the overall implementation of distributed task coordination and processing for reconciliation supporting heterogeneous data sources, a performance optimization mechanism can be introduced to improve system efficiency. For example, the system can dynamically adjust the processing priority of the task queue and the resource allocation strategy of server nodes based on historical task execution data. For instance, if the system finds that the delivery status update task has a long execution time and is likely to become a bottleneck, the system can dynamically adjust its priority, allocating more resources to ensure its rapid completion. This performance optimization mechanism can effectively improve the overall system efficiency and reduce task waiting time.

[0080] Step S89: For the overall implementation of distributed task coordination and processing for reconciliation supporting heterogeneous data sources, a data backup mechanism can be introduced to improve system fault tolerance. For example, the system can periodically back up critical data in the task queue, node array, and hash value storage. When a system failure occurs, the task scheduling state can be quickly restored using the backup data. For instance, the system can back up the task queue and node array data hourly. When the system restarts due to an unexpected failure, the task scheduling state can be restored using the backup data, ensuring that critical tasks are not lost due to the failure. This data backup mechanism effectively improves the system's fault tolerance and ensures business continuity.

[0081] Step S90: For the overall implementation of distributed task coordination and processing for reconciliation supporting heterogeneous data sources, a task execution result verification mechanism can be introduced to improve data reliability. For example, after a task is completed, the system can verify whether the task execution result meets expectations. If it does not meet expectations, the system will re-add the task to the task queue, waiting for the next execution. For example, in an online payment platform, after the transaction data entry task is completed, the system verifies whether the entered data is complete. If data is found to be missing, the system will re-add the task to the task queue to ensure the completeness of data entry. This result verification mechanism can effectively improve data reliability and ensure the accuracy of business processes.

[0082] Example 2 On the other hand, this second embodiment, based on the distributed reconciliation task coordination and processing method supporting heterogeneous data sources provided in the first embodiment, also provides a computer storage medium 1140 (hereinafter referred to as the storage medium). For example... Figure 3 The diagram shown is a schematic of a computer storage medium structure framework provided in Embodiment 3 of the present invention, which includes: Memory 1130 is used to store computer programs; The communication interface 1120 is used to connect the memory 1130 to the processor 1110; Processor 1110 is configured to execute a computer program to implement an embodiment of a distributed reconciliation task coordination processing method supporting heterogeneous data sources, as disclosed in any combination of the above embodiments.

[0083] This invention discloses a distributed task coordination method for reconciliation supporting heterogeneous data sources. It addresses the challenges of task dependency management and efficient resource allocation in complex business scenarios, particularly ensuring sequential task execution, dynamic resource balancing, and data consistency in scenarios involving the integration of tasks from multiple industries. This invention ensures ordered task execution according to dependencies by constructing a task dependency graph and topological sorting; it employs a dynamic task allocation and health monitoring mechanism to monitor node status in real time and optimize resource allocation, avoiding overload and failure impacts; and it introduces an idempotency judgment mechanism, using hash value verification to prevent duplicate task execution and ensure data consistency. For multi-scenario integration issues, this invention provides a unified interface to support task configuration and failover, combining priority strategies and predictive failover to ensure priority execution of critical business tasks and system stability. Ultimately, this invention achieves efficient scheduling and rational resource utilization for large-scale tasks, significantly improving the system's adaptability, reliability, and execution efficiency in complex business environments, and providing comprehensive support for cross-domain task management.

[0084] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; those skilled in the art can modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reconciliation distributed task coordination processing method supporting heterogeneous data sources, characterized in that, include: Construct a task dependency graph data structure, including an adjacency table and a container for data to be collected; Create an empty queue and iterate through the data to be collected, adding task nodes with an in-degree of zero to the queue to obtain a queue to be sorted. Traverse the queue to be sorted, retrieve the current node, compare it with the nodes in the adjacent table, and when the v value of the current node plus one equals the v value of the adjacent node, put the adjacent node into the queue to obtain the topological sort queue; Use a task queue to store tasks to be executed, and use a node array to store server node information; Start the scheduling thread to retrieve tasks from the task queue and allocate tasks for execution based on healthy nodes in the node array; Start a health check thread to monitor node status and update the node array; Before the task is executed, the task ID and parameters are obtained. The parameters are sorted lexicographically and then concatenated with the task ID. A hash value is generated using SHA-256 and used as a unique key for idempotency testing.

2. The method of claim 1, wherein, The construction task dependency graph data structure includes: The adjacent table stores the dependencies between task nodes, and the container for statistics to be collected stores the initial information of all task nodes, including the v value and in-degree value of each task node. Iterate through each task node in the container of data to be collected. If the in-degree of a task node is zero, then put the task node into an empty queue. The empty queue maintains the topological sorting order through the first-in-first-out (FIFO) property.

3. The method of claim 1, wherein, The step of traversing the queue to be sorted, retrieving the current node, and comparing it with nodes in adjacent tables includes: Retrieve the current node from the head of the queue to be sorted, and obtain the v value of the current node; Iterate through the downstream nodes of the current node in the adjacent list. If the v value of the downstream node is equal to the v value of the current node plus one, then decrement the in-degree value of the downstream node by one. If the in-degree value of the downstream node is zero, then add the downstream node to the unsorted queue. Repeat the above process until the queue to be sorted is empty, thus obtaining a topologically sorted queue.

4. The method of claim 1, wherein, The use of a task queue to store tasks to be executed and a node array to store server node information includes: The task queue stores tasks to be executed sequentially in a topologically sorted queue, and each task carries an execution rule. The node array stores multiple server nodes, and the information for each server node includes CPU utilization, memory usage, and running status. The scheduling thread obtains the list of healthy nodes from the node array in real time.

5. The method of claim 1, wherein, The startup scheduling thread retrieves tasks from the task queue and allocates tasks for execution based on healthy nodes in the node array, including: The scheduling thread retrieves tasks to be executed from the head of the task queue; Traverse the node array. If the server node is in a healthy running state, the CPU utilization is below the preset threshold, and the memory usage meets the requirements, then assign the task to be executed to that server node for execution. If there are no healthy nodes, the tasks to be executed will be kept in the task queue and wait.

6. The method of claim 1, wherein, The step of starting a health check thread to monitor node status and update the node array includes: The health monitoring thread periodically traverses each server node in the node array; For each server node, obtain the current CPU usage, memory usage, and running status. If the CPU usage exceeds the preset threshold, the memory usage is abnormal, or the running status is faulty, then mark the server node as unhealthy. The node array is updated in real time with the health status of each server node.

7. The method of claim 1, wherein, The step of obtaining the task ID and parameters before task execution, sorting the parameters lexicographically, concatenating them with the task ID, and using SHA-256 to generate a hash value as a unique key for idempotency determination includes: Get the task ID and parameter list of the task to be executed; Sort the parameter list lexicographically to obtain the sorting parameters; Concatenate the task ID and sorting parameters into a string, and calculate the hash value using the SHA-256 algorithm; Check if the hash value exists in the preset storage. If it exists, determine that the task has been executed. If it does not exist, store the hash value and execute the task.

8. The method of claim 1, wherein, The method for coordinating and processing distributed reconciliation tasks that supports heterogeneous data sources also includes providing a unified API for each project to support developers in writing code to deploy tasks.

9. The method of claim 1, wherein, The method for coordinating and processing distributed reconciliation tasks that supports heterogeneous data sources also includes: Update the task execution progress and record the task log in real time; Simultaneously mark failed tasks and the abnormal tasks corresponding to the call exception task logs; A retry mechanism list is created in real time for multiple failed and abnormal tasks. Once the task nodes in the queue to be sorted have completed their work, the task nodes on the retry mechanism list are started.

10. A storage medium, characterized by The storage medium stores a computer program, which, when executed by a processor, implements the steps of the distributed task coordination and processing method for reconciliation supporting heterogeneous data sources as described in any one of claims 1-8.

Images