An execution routing method and system for master-slave task orchestration

Abstract

The present application relates to the field of data resource scheduling, and more particularly to an execution routing method and system for master-slave task orchestration, which first responds to the initiation of an aggregated task request, extracts a corresponding database advisory lock to elect a corresponding master node from a preset service cluster; then, in response to the master node starting, triggers an execution router to schedule multiple extractors from a preset data extractor thread pool to perform data extraction tasks; wherein the execution engine built into each extractor is configured to: divide the date range of the aggregated task request; execute the sub-request tasks assigned by the execution router and return the execution results within the date range; push the execution results to a preset result merger; and finally, drive the result merger to output the aggregated results. Unlike the prior art, the present application can overcome the technical problems of poor convenience, low stability and low security of existing cross-border data return task scheduling technology.

Description

Technical Field

[0001] This invention relates to the field of data resource scheduling technology. More specifically, this invention relates to an execution routing method and system for master-slave task orchestration. Background Technology

[0002] Multinational corporations typically rely on third-party APIs to obtain feedback data from overseas advertising campaigns. This feedback data can often reach tens or even hundreds of millions of records per day, so distributed task scheduling and general-purpose tools are generally used to support task orchestration and data packet acquisition. Popular general-purpose tools include ZooKeeper, etcd, Kubernetes, Airflow, Celery, and Quartz Scheduler. However, these tools have the following drawbacks when applied to this scenario:

[0003] Firstly, traditional distributed systems rely entirely on external heavyweight distributed coordination service tools (ZooKeeper / etcd) for cluster master node election and node coordination. These tools primarily support single-point-aware responses, meaning one request corresponds to one response. However, they do not support scenarios where requests for date ranges are split or multiple reports are integrated, i.e., one request needs to return multiple pieces of information or multiple information aggregations. Enterprises still need to deploy back-end information distribution and aggregators, which not only increases the complexity of system deployment and operation and maintenance but also introduces additional single-point failure risks. The overall cross-border data return process is not secure, convenient, or stable enough, and it also increases operation and maintenance costs.

[0004] Secondly, some lightweight master node election schemes are overly bound to specific container orchestration environments (such as Leader Election which relies on Kubernetes), making them unsuitable for use in bare metal or hybrid cloud environments. Overall, they lack versatility and convenience.

[0005] Third, when performing large-scale, time-consuming third-party API retrieval tasks, if any worker node crashes, traditional solutions often have to retry the entire large task from the beginning, resulting in loss of memory state, waste of API call quota, and repeated data retrieval.

[0006] In summary, existing cross-border data return task scheduling technologies suffer from technical problems such as poor convenience, low stability, and low security. Summary of the Invention

[0007] To address the technical problems of poor convenience, low stability, and low security in the aforementioned cross-border data return task scheduling technology, this invention discloses an execution routing method and system for master-slave task orchestration.

[0008] In a first aspect, the present invention discloses an execution routing method for master-slave task orchestration, comprising: S10: In response to the initiation of the aggregation task request, extract the corresponding database query lock to elect the corresponding master node from the preset service cluster.

[0009] S20: In response to the master node startup, trigger the execution router to schedule multiple extractors from the preset data extractor thread pool to perform data extraction tasks; wherein, the execution engine built into each extractor is configured as follows: Define the date range for the aggregated task request; execute the sub-request tasks assigned by the execution router and return the execution results within the date range; push the execution results to the preset result merger; S30: Drives the result merger to output the aggregated result.

[0010] Beneficial effects: When an aggregation task request is initiated, the corresponding database query lock is first extracted to elect the corresponding master node from the preset service cluster. This process does not require the intervention of external tools such as ZooKeeper, etcd, or Redis, eliminating the excessive reliance on external heavyweight distributed coordination service tools and improving security. Then, with the startup of the local master node as the trigger condition, the execution router is triggered to assign tasks to multiple extractors and execute the sub-request tasks related to the aggregation task request in parallel, thereby improving convenience. Finally, the execution results are pushed to the preset result merger to achieve the aggregation of cross-border returned data.

[0011] Preferably, step S10 includes: Each node in the driver service cluster calls the non-blocking query lock function of the preset relational database to acquire the cluster lock; If the current node successfully acquires the lock and is in a non-master node state, then the current node is switched to master node; the instance information and heartbeat time of the master node are updated to the database, and dead node records that have timed out and not been updated are cleaned up; If the current node fails to acquire the lock and is in the master node state, the current node will be demoted to a slave node and the scheduling operations related to the slave node will be stopped.

[0012] Preferably, after electing the corresponding master node, the method of the present invention further includes: Based on the configuration information of the master node, a unique instance identifier and a registration request for announcing its own capacity are generated; the registration request includes the host address, port information, and maximum concurrent task processing attributes.

[0013] Preferably, the method of the present invention further includes: In response to the successful registration request, the driver extractor initiates a cycle to execute tasks: Send heartbeat information to the master node; The heartbeat information includes AK / SK authentication information, heartbeat setting time, current number of jobs, current execution status, and the address of the host to which the job belongs.

[0014] Preferably, the date range for defining aggregated task requests includes: Calculate the time span between the start and end dates of the aggregate task request; If the time span exceeds the maximum number of days threshold for a single task execution request, the maximum number of days threshold is accumulated in a loop to generate multiple consecutive sub-date range blocks until the time period to which the last sub-date range block belongs covers the end date. Each sub-date range is configured with a sub-request task.

[0015] Preferably, executing the sub-request task assigned by the execution router includes: Extract the paging cursor of the sub-request task; whereby the paging cursor serves as a checkpoint for the sub-job to record the data retrieval progress; Update the pagination cursor to the database in real time.

[0016] Preferably, executing the sub-request task assigned by the execution router further includes: When an extractor error is detected or a master node state switch occurs, the recovery processor is triggered. The execution recovery processor is used to read the latest stored subjob checkpoint in the database and instruct the reassigned extractor to continue executing the subrequest task from where the page cursor was interrupted.

[0017] Preferably, the triggering router schedules multiple extractors from a preset data extractor thread pool to perform data extraction tasks, including: In the timed scheduling loop, the logical judgment branches of the master and slave node roles in the previous execution cycle are cleared.

[0018] Preferably, the output aggregation result of the driving result merger includes: Barriers are completed in the pre-tasks of the results merger; The task completion barrier is used to intercept and suspend the aggregation operation until it receives completion or failure signals for all execution results. Then, the completion barrier is released and the aggregation results are output in a non-polling blocking state.

[0019] Secondly, the present invention also discloses an execution routing system for master-slave task orchestration, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the execution routing method for master-slave task orchestration described in the first aspect is implemented.

[0020] The advantages of this invention, which differ from existing technologies, are as follows: First, the method of this invention supports master node election for local database clusters, eliminating the need for external tools such as ZooKeeper, etcd, or Redis, thus enhancing information security. During the return process of the aggregation task, multiple extractors' built-in execution engines are used to execute subtasks, resulting in shorter execution time and greater stability.

[0021] Secondly, with the local database cluster as the sole trusted source, the newly elected master and slave nodes can recover their state using the recovery processor, achieving zero memory loss and greater stability.

[0022] Third, the worker nodes announce their capacity upon startup, ensuring that the load is updated with each heartbeat, meaning the router always has the latest data, further enhancing stability. Attached Figure Description

[0023] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein: Figure 1 This is a flowchart of the execution routing method for master-slave task orchestration in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the execution routing system for master-slave task orchestration in Embodiment 2 of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0026] Example 1 like Figure 1 As shown, this embodiment discloses an execution routing method for master-slave task orchestration, including: S10: In response to the initiation of the aggregation task request, extract the corresponding database query lock to elect the corresponding master node from the preset service cluster.

[0027] Step S10 includes: S11: Each node in the driver service cluster calls the non-blocking query lock function of the preset relational database to obtain the cluster lock.

[0028] In this embodiment, the service cluster (dubhe-service cluster) has multiple nodes, each of which is configured with an execution instance to handle the corresponding aggregation task request. When one node acts as the master node, the other nodes are in an idle state.

[0029] S12: If the current node successfully acquires the lock and is in a non-master node state, then switch the current node to master node; update the instance information and heartbeat time of the master node to the database, and clean up dead node records that have timed out and not been updated.

[0030] S13: If the current node fails to acquire the lock and is in the master node state, the current node will be downgraded to a slave node and the scheduling operations related to the slave node will be stopped.

[0031] Optionally, in a Java environment, each node in the service cluster can attempt to acquire the lock using the PostgreSQL-specific non-blocking function `pg_try_advisory_lock`. If the lock acquisition is successful, the dead node cleanup function is executed and scheduling is initiated; if it was originally the master node but failed to acquire the lock (network disconnection or lock release), a seamless and smooth degradation mechanism is executed to ensure that the system does not experience a split-brain scenario. This approach effectively improves the stability of the method of this invention. Furthermore, through the self-developed design of steps S11-S13 above, excessive reliance on external heavyweight distributed coordination service tools can be eliminated, resulting in higher security.

[0032] It should be noted that after electing the corresponding master node, the method in this embodiment also includes: Based on the configuration information of the master node, a unique instance identifier and a registration request to declare its capacity are generated. The registration request includes the host address, port information, and maximum concurrent task processing attributes.

[0033] Through the above technical solution, the master node can actively announce its capacity at startup, so that the router always has the latest data when the load is updated with each heartbeat. This further enhances the real-time synchronization capability and stability of the method in this embodiment.

[0034] S20: In response to the master node startup, trigger the execution router to schedule multiple extractors from the preset data extractor thread pool to perform data extraction tasks.

[0035] It is important to note that in the timed scheduling loop of step S20 above, the logical judgment branches of the master and slave node roles from the previous execution cycle need to be cleared to ensure that there is no interference from role-related if-else information in the hot path during task mode scheduling. In this embodiment, the state design pattern is used to configure the scheduling hot path of the execution router. Combined with the above technical solution, the execution efficiency of the hot path in high-concurrency scenarios can be improved.

[0036] Furthermore, each extractor has a built-in execution engine configured with the following functions: S201: Define the date range for aggregate task requests.

[0037] Specifically, step S201 includes: First, calculate the time span between the start and end dates of the aggregated task request. If the time span is greater than the maximum number of days threshold for a single task execution request, then cyclically accumulate the maximum number of days threshold to generate multiple consecutive sub-date range blocks until the time period of the last sub-date range block covers the end date.

[0038] It should be noted that each sub-date range is configured with a sub-request task.

[0039] More specifically, the relevant functional scripts are used to calculate the span. If it exceeds the threshold, a while loop is used to split the large-span date range (e.g., one year) into multiple chunk lists based on maxDaysPerRequest (e.g., a maximum of 7 days per retrieval). This perfectly adapts to the hard limitations of single API queries on overseas advertising platforms. Through the above date range splitting strategy, on the one hand, it ensures that the returned aggregated information is comprehensive, reliable, and highly consistent with the original content; on the other hand, it avoids system errors caused by API rate limiting, thereby improving the stability of the method in this embodiment.

[0040] S202: Execute the sub-request task assigned by the execution router and return the execution results within the date range.

[0041] Specifically, in implementing step S202 above, it is necessary to extract the paging cursor of the sub-request task and update the paging cursor to the database in real time. When an error is detected in the extractor or a state switch occurs in the master node, the execution of the recovery processor is triggered.

[0042] The execution recovery processor reads the latest stored subjob checkpoint from the database and instructs the reassigned fetcher to continue executing the sub-request task from where the paging cursor was interrupted. The paging cursor serves as the subjob checkpoint for recording the data fetching progress.

[0043] To illustrate further, for daily ad feedback that may contain tens of thousands of pages of data, the above-described solution allows the extractor to avoid keeping track of progress in local memory. Instead, it flushes the cursor status to the database and creates an execution snapshot after fetching each page (or batch). If a node crashes while fetching tens of thousands of pages, the reassigned task can directly resume fetching from the cursor in the database. This means that when the extractor restarts, it resumes within the specified date range, completely solving the problem of "wasted API call bandwidth and duplicate data fetching due to retries from the beginning" in existing technologies. This significantly reduces bandwidth waste and improves the stability of the entire method.

[0044] It is important to note that when the registration request is approved, the following tasks must also be executed during the extractor startup cycle: Send heartbeat information to the master node; The heartbeat information includes AK / SK authentication information, heartbeat setting time, current number of jobs, current execution status, and the address of the host to which the job belongs.

[0045] It should be explained that AK / SK authentication information is an identity authentication mechanism based on the digital signature of API requests using access keys. It is used to verify the sender's identity and ensure that the request has not been tampered with.

[0046] Using the above technical solution, the router can accurately calculate the remaining idle computing power of each extractor, achieving intelligent load balancing at extremely low cost. Furthermore, worker nodes declare their capacity upon startup and update their load with each heartbeat, ensuring the router always has the latest data.

[0047] S203: Push the execution results to the preset result merger.

[0048] S30: Drives the result merger to output the aggregated result.

[0049] Specifically, step S30 above includes: The barrier is completed before the result merger pre-task.

[0050] The task completion barrier is used to intercept and suspend the aggregation operation until it receives completion or failure signals for all execution results. Then, it releases the completion barrier and outputs the aggregation results in a non-polling blocking state.

[0051] By configuring the barrier as described above, waiting for N heterogeneous asynchronous API reports before aggregation can avoid blocking the polling thread and further improve the stability of the method in this embodiment.

[0052] Based on the above technical disclosures, the method in this embodiment combines PostgreSQL advisory lock master node election with self-registration, capacity reporting type extractor worker nodes, and subjob checkpoints, requiring no external coordination services whatsoever. It has high security and adaptability to cross-border aggregation task returns.

[0053] In addition, the method in this embodiment also introduces a pluggable splitting strategy and a completion barrier, which requires requests to be executed in blocks by date and multiple asynchronous reports to be considered consistent before the data is considered consistent, making the returned aggregate task data more complete.

[0054] Example 2 like Figure 2 As shown, this embodiment discloses an execution routing system for master-slave task orchestration, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the execution routing method for master-slave task orchestration described in Embodiment 1 is implemented.

[0055] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0056] In the description of this specification, "multiple" means at least two, such as two, three or more, etc., unless otherwise expressly and specifically defined.

[0057] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. An execution routing method for master-slave task orchestration, characterized in that, include: S10: In response to the initiation of the aggregation task request, extract the corresponding database query lock to elect the corresponding master node from the preset service cluster; S20: In response to the startup of the master node, the execution router is triggered to schedule multiple extractors from a preset data extractor thread pool to perform data extraction tasks; wherein, the execution engine built into each extractor is configured as follows: Divide the date range of the aggregated task request; execute the sub-request tasks assigned by the execution router and return the execution results within the date range; push the execution results to a preset result merger; S30: Drive the result merger to output the aggregated result.

2. The execution routing method for master-slave task orchestration according to claim 1, characterized in that, Step S10 includes: Drive each node in the service cluster to call the non-blocking query lock function of the preset relational database to obtain the cluster lock; If the current node successfully acquires the lock and is in a non-master node state, then the current node is switched to master node; the instance information and heartbeat time of the master node are updated to the database, and dead node records that have timed out and not been updated are cleaned up. If the current node fails to acquire the lock and is in the master node state, the current node will be downgraded to a slave node, and the scheduling operations related to the slave node will be stopped.

3. The execution routing method for master-slave task orchestration according to claim 1, characterized in that, After electing the corresponding master node, the method further includes: Based on the configuration information of the master node, a unique instance identifier and a registration request for announcing its own capacity are generated; wherein, the registration request includes the host address, port information and maximum concurrent task processing attributes.

4. The execution routing method for master-slave task orchestration according to claim 3, characterized in that, Also includes: In response to the successful registration request, the extractor is driven to initiate a periodic task execution process: Send heartbeat information to the master node; The heartbeat information includes AK / SK authentication information, heartbeat setting time, current number of jobs, current execution status, and the address of the host to which it belongs.

5. The execution routing method for master-slave task orchestration according to claim 1, characterized in that, The date range for the aggregated task request is divided into: Calculate the time span between the start and end dates of the aggregated task request; If the time span is greater than the maximum number of days threshold for a single task execution request, the maximum number of days threshold is accumulated cyclically to generate multiple consecutive sub-date range blocks until the time period to which the last sub-date range block belongs covers the end date; Each sub-date range is configured with a sub-request task.

6. The execution routing method for master-slave task orchestration according to claim 1, characterized in that, Executing the sub-request task assigned by the execution router includes: Extract the pagination cursor of the sub-request task; wherein the pagination cursor serves as a sub-job checkpoint for recording the data retrieval progress; The pagination cursor is updated to the database in real time.

7. The execution routing method for master-slave task orchestration according to claim 6, characterized in that, Executing the sub-request task assigned by the execution router further includes: When an error is detected in the extractor or a state switch occurs in the master node, the recovery processor is triggered. The execution recovery processor is used to read the latest subjob checkpoint stored in the database and instruct the reassigned extractor to continue executing the subrequest task from the point where the page cursor was interrupted.

8. The execution routing method for master-slave task orchestration according to claim 1, characterized in that, The triggering router schedules multiple extractors from a pre-defined data extractor thread pool to perform data extraction tasks, including: In the timed scheduling loop, the logical judgment branches of the master and slave node roles in the previous execution cycle are cleared.

9. The execution routing method for master-slave task orchestration according to claim 1, characterized in that, The result merger is driven to output the aggregation result, including: The pre-task completion barrier is established before the result merger; The task completion barrier is used to intercept and suspend the aggregation operation until completion or failure signals of all execution results are received. Then, the completion barrier is released and the aggregation result is output in a non-polling blocking state.

10. An execution routing system for master-slave task orchestration, characterized in that, It includes a processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement the execution routing method for master-slave task orchestration as described in any one of claims 1-9.

Images