Cooperative assembly line task scheduling method and system suitable for hierarchical federated network

Abstract

The invention discloses a collaborative assembly line task scheduling method and system suitable for a hierarchical federated network, and belongs to the field of cross-data center collaborative analysis. The method comprises the following steps: randomly disassembling a collaborative pipeline into at least one field subtask according to data requirements of a directed acyclic graph and a subtask, and constructing at least one field subtask on the basis of a topological structure of a hierarchical federated network, a data set on a leaf node and the data requirements of the field subtask; scheduling the subtasks in the field to corresponding branch center nodes in the upper federated network; and on the basis of the resource demand and the data demand of the subtask, disassembling the field subtask into at least one node subtask, and scheduling the node subtask to the corresponding leaf node in the lower-layer field subnet in combination with the data set on the leaf node. According to the method, interdisciplinary field seamless cooperation of complex scientific research tasks can be supported.

Description

Technical Field

[0001] The present invention belongs to the field of cross-data center collaborative analysis and proposes a collaborative pipeline task scheduling method and system suitable for hierarchical federated networks. Background Art

[0002] With the rise of new research paradigms such as "convergent science" and "AI4Science," scientific research is gradually moving toward multidisciplinary, cross-disciplinary collaboration. The core of this paradigm lies in integrating data resources and algorithmic models from different fields to jointly solve complex scientific problems. For example, the identification of silt dams and the assessment of their ecological value require the integration of remote sensing, channel, elevation, and topographic data; while space astronomy and high-energy physics share observational data and theoretical models. This cross-disciplinary collaboration relies heavily on efficient scientific data management and collaborative capabilities.

[0003] A hierarchical federated network is a distributed data management and collaboration network suitable for solving cross-domain collaborative analysis challenges. Its typical structure includes:

[0004] The upper-level federated network is composed of multiple geographically dispersed subject-matter sub-centers (e.g., ecology, space, astronomy, etc.). These sub-centers support cross-disciplinary collaborative data analysis through network access.

[0005] Lower-level domain subnet: It consists of several internal data center nodes affiliated with the domain sub-center. There is a high degree of trust between the nodes, and they are responsible for aggregating and processing data in this field.

[0006] Hierarchical federated networks provide an effective infrastructure for cross-domain collaborative analysis, but in practical applications, existing collaborative task scheduling methods have limitations. For example, although there are a variety of pipeline-based cross-center collaborative computing methods (such as "Orchestration method and system for cross-center collaborative computing based on pipeline mechanism" patent number: 2022101459584), they do not take into account network hierarchy and load balancing, resulting in low scheduling efficiency under hierarchical federated networks. The pipeline computing method proposed in Chinese patent application CN115062329A is mainly aimed at privacy computing and federated learning scenarios, combining hardware such as FPGA, CPU, and GPU to improve computing efficiency, but does not consider multi-data center scenarios. The federated learning acceleration method in Chinese patent application CN118917387A allocates parallel steps to different engines for execution. Although it can improve the efficiency of encryption operations, it does not solve the data transmission overhead problem in multi-data center scenarios, and the task types are limited to data encryption algorithms, and the scope of application is relatively narrow. Summary of the Invention

[0007] The present invention provides a collaborative pipeline task scheduling method and system suitable for hierarchical federated networks, which adapts to the characteristics of hierarchical federated networks and can support seamless cross-disciplinary collaboration of complex scientific research tasks.

[0008] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0009] A collaborative pipeline task scheduling method applicable to a hierarchical federated network, wherein the hierarchical federated network is composed of an upper-layer federated network and a lower-layer domain subnet, the method comprising:

[0010] Obtain the topology of the hierarchical federated network and the datasets on each leaf node in the lower domain subnet;

[0011] Obtain the collaborative pipeline task, the directed acyclic graph of the collaborative pipeline task, and the resource requirements and data requirements of each subtask in the collaborative pipeline task;

[0012] Decompose the collaborative pipeline task into at least one domain subtask based on the directed acyclic graph and the data requirements of the subtasks. Then, schedule the domain subtask to the corresponding sub-center node in the upper-layer federated network based on the topology of the hierarchical federated network, the dataset on the leaf node, and the data requirements of the domain subtask. The domain subtask is composed of at least one of the subtasks.

[0013] Based on the resource and data requirements of the subtask, the domain subtask is decomposed into at least one node subtask, and combined with the data set on the leaf node, the node subtask is scheduled to the corresponding leaf node in the lower-level domain subnet; wherein, the node subtask is composed of at least one of the subtasks.

[0014] Furthermore, the step of decomposing the collaborative pipeline into at least one domain subtask according to the directed acyclic graph includes:

[0015] Obtain the dependency relationships of each subtask in the collaborative pipeline task based on the directed acyclic graph;

[0016] The collaborative pipeline task is disassembled according to the dependency relationship of the subtasks, and each domain subtask only depends on the dataset of one domain, so the collaborative pipeline task is disassembled into at least one domain subtask.

[0017] Furthermore, the subtasks in this field are dispatched to the corresponding sub-center nodes in the upper-layer federated network, including:

[0018] Compute dependencies between domain subtasks;

[0019] The domain subtask is dispatched to the corresponding sub-center node in the upper-level federated network, and the execution order of the domain subtasks is set according to the dependencies between the domain subtasks to ensure that the subsequent domain subtasks are carried out after the previous domain subtasks are completed.

[0020] Furthermore, the domain subtask is decomposed into at least one node subtask based on the resource and data requirements of the subtask, and the node subtask is scheduled to the corresponding leaf node in the lower domain subnet in combination with the data set on the leaf node, including:

[0021] Obtain the subtasks in the domain subtask and determine whether the subtask has high resource requirements based on the resource requirements of the subtask;

[0022] If the subtask is a resource-intensive algorithm and does not directly depend on any dataset, a resource suitability score is generated for each leaf node based on the leaf node's resource situation, and the subtask is scheduled to the corresponding leaf node in the lower-level domain subnet based on the resource suitability score;

[0023] When the subtask is not an algorithm with high resource requirements and does not directly depend on any data set, a node subtask is generated based on the data requirements of the subtask, and the node subtask is scheduled to the corresponding leaf node in the lower-level domain subnet based on the data set on the leaf node.

[0024] Furthermore, a resource suitability score is generated for each leaf node based on the resource status of the leaf node, and the subtask is dispatched to the corresponding leaf node in the lower domain subnet according to the resource suitability score, including:

[0025] Get the total CPU resources, total memory resources, currently used CPU resources, and currently used memory resources of the leaf node;

[0026] Get the CPU and memory requirements of the subtask;

[0027] Calculate the CPU resource score of the leaf node based on the total CPU resources, the currently used CPU resources, and the CPU demand of the subtask. cpu ;

[0028] Calculate the memory resource score of the leaf node based on the total memory resources, the currently used memory resources, and the memory requirements of the subtask. mem ;

[0029] Calculate the resource adaptation score of the subtask Score = Score cpu w cpu +Score memw mem ; Among them, w cpu and w mem Represents the CPU weight coefficient and memory weight coefficient respectively, and w cpu +w mem =1;

[0030] A leaf node is selected in the lower domain subnet according to the resource adaptability score Score, and the subtask is scheduled to the corresponding leaf node in the lower domain subnet.

[0031] Furthermore, when the subtask is a computationally intensive task, the CPU weight coefficient is greater than the memory weight coefficient; when the subtask is a memory intensive task, the CPU weight coefficient is less than the memory weight coefficient.

[0032] A collaborative pipeline task scheduling system for a hierarchical federated network, wherein the hierarchical federated network consists of an upper-layer federated network and a lower-layer domain subnet, wherein:

[0033] The sub-center nodes in the upper-layer federated network include:

[0034] The task acquisition module is used to obtain the topology of the hierarchical federated network and the data sets on each leaf node in the lower domain subnet; obtain the collaborative pipeline task, the directed acyclic graph of the collaborative pipeline task, and the resource requirements and data requirements of each subtask in the collaborative pipeline task;

[0035] A domain subtask scheduling module is used to decompose the collaborative pipeline task into at least one domain subtask based on the directed acyclic graph and the data requirements of the subtask, and schedule the domain subtask to the corresponding sub-center node in the upper-level federated network based on the topology of the hierarchical federated network, the data set on the leaf node, and the data requirements of the domain subtask; wherein the domain subtask is composed of at least one of the subtasks;

[0036] The node subtask scheduling module is used to decompose the domain subtask into at least one node subtask based on the resource requirements and data requirements of the subtask, and schedule the node subtask to the corresponding leaf node in the lower-level domain subnet in combination with the data set on the leaf node; wherein the node subtask is composed of at least one of the subtasks.

[0037] An electronic device, characterized in that the electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements any of the above-mentioned collaborative pipeline task scheduling methods applicable to hierarchical federated networks.

[0038] A computer-readable storage medium, characterized in that computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, any of the above-mentioned collaborative pipeline task scheduling methods applicable to a hierarchical federated network is implemented.

[0039] A computer program product, characterized in that when the computer program product is run on a computer device, the computer device executes any of the above-mentioned collaborative pipeline task scheduling methods applicable to a hierarchical federated network.

[0040] Compared with the prior art, the present invention has at least the following beneficial effects.

[0041] 1) Through multi-level task splitting and scheduling strategies, the present invention can significantly improve the execution efficiency and load balancing capabilities of collaborative pipelines in hierarchical federated networks, thereby improving the overall efficiency of collaborative analysis in interdisciplinary fields.

[0042] 2) This invention adapts to the characteristics of hierarchical federated networks and is applicable to a wide range of tasks, including general big data processing tasks as well as machine learning, deep learning, and other tasks, without limiting the hardware type used for task calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is the overall flow chart of collaborative pipeline scheduling.

[0044] Figure 2 Schematic diagram of collaborative analysis in a hierarchical federated network.

[0045] Figure 3 Schematic diagram for subtask division.

[0046] Figure 4 This is the pipeline scheduling flow chart in the upper-layer federated network.

[0047] Figure 5 This is the pipeline scheduling flow chart in the lower-level domain subnet. DETAILED DESCRIPTION

[0048] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0049] This invention provides a novel hierarchical task splitting and scheduling strategy. Specifically, tasks are first dispatched to the data center node corresponding to the domain to which they depend, followed by the data center node closest to the data source. Resource-intensive algorithms are then divided into separate subtasks and dispatched to the most appropriate data center node based on resource suitability scores. This approach not only improves the efficiency of collaborative analysis tasks but also ensures efficient resource utilization and balanced load distribution.

[0050] like Figure 1 As shown, the collaborative pipeline task scheduling method applicable to a hierarchical federated network of the present invention includes the following steps.

[0051] (1) Network topology awareness and resource monitoring.

[0052] Analyze the topological structure of the hierarchical federated network, clarify the topological level of each node and the distribution of data sets. The domain sub-center node not only represents itself, but also represents all the data center nodes below it, and is responsible for providing a list of data sets published by all data centers within the domain subnet.

[0053] In one embodiment, assume that there is a domain sub-center node Tier1_A, and the lower layer contains three leaf nodes Tier2_A1, Tier2_A2, and Tier2_A3. The Tier1_A dataset provides the union of the datasets of all data center nodes in the subnet, such as Figure 2 shown.

[0054] (2) Generation of collaborative pipeline tasks.

[0055] Users log in to any domain sub-center node or leaf node in the network, use data and algorithms to orchestrate collaborative analysis pipelines, organize and submit collaborative tasks to a domain sub-center node according to DAG (directed acyclic graph) dependencies.

[0056] (3) Generation of domain subtasks (sub-pipelines).

[0057] The domain sub-center node obtains the collaborative tasks to be executed and decomposes the pipeline into several domain sub-tasks (i.e. sub-pipelines) according to the DAG graph. Each domain sub-task is presented in the form of a pipeline, including multiple data sets, algorithms and data flows. Figure 3 As shown in the figure, suppose there is a collaborative pipeline task T in an interdisciplinary field. The input data Data1 and Data2 are stored in the lower leaf nodes of the two domain sub-center nodes Tier1_A and Tier1_B respectively. After preprocessing (T A1 、T B1 ), feature extraction (T A2 、T B2 ), model training (T A3 、T B3 ) and then perform aggregation analysis T'. According to the above principles, it can be divided into three areas of sub-tasks, namely (T A1 、T A2 、T A3 )、(T B1 、T B2 、T B3 ),(T').

[0058] In one embodiment, cross-domain collaborative pipeline tasks are split to ensure that each domain subtask relies only on datasets from the same domain. If certain algorithms rely on datasets from multiple domains, the upstream tasks of the algorithm are divided into different subtasks to reduce the complexity of cross-domain dependencies.

[0059] (4) Upper-level federated network task scheduling.

[0060] According to the dependency relationship of subtasks in the directed acyclic graph (DAG), the domain subtasks are scheduled in sequence. In the above example, the domain subtask (T A1 、T A2 、T A3 ) and (T B1 、T B2 、T B3 ) to the sub-center nodes Tier1_A and Tier1_B of the subnet where the data source is located. For subtask (T') that requires input data from two fields at the same time, it can be dispatched to the central node of any subnet where the dependent data source is located. The scheduling process is as follows Figure 4 shown.

[0061] Scheduling is performed in the upper-level interdisciplinary federated network, which assigns domain subtasks to each sub-center for execution. The scheduling considers the dependencies between tasks, ensuring that subsequent tasks are performed only after the predecessor tasks have been completed.

[0062] (5) Lower-level domain subnet task scheduling.

[0063] Each domain subtask is scheduled in the lower-level domain subnet, and the domain sub-center node is further split into node subtasks, that is, the granularity that can be executed by each data center node in the subnet. Try to ensure that each node subtask after splitting only depends on the data set of the same data center node and is scheduled to the data center node for execution. Algorithms with high resource requirements (such as AI model training) are divided into a separate subtask. For such high-resource-demand subtasks that do not directly depend on the data set, computing resource adaptation is given priority during scheduling. Based on the available CPU and memory resources, as well as the subtask's demand for computing resources, a "resource adaptability" score is calculated for each node in the subnet, and it is scheduled to the data center node with the highest score for execution, thereby ensuring efficient use of computing resources.

[0064] Specifically, the domain subtasks are further split into granularities that can be executed by a single data center, while considering the resources required by each algorithm in the subtask. In the above example, for example, algorithm T A3 and T B3 It requires more computing resources. A1 、T A2 、T A3 ) is split into (T A1、T A2 ) and (T A3 ) two node subtasks, (T B1 、T B2 、T B3 ) is split into (T B1 、T B2 ) and (T B3 ) two node subtasks. A1 、T A2 )、(T B1 、T B2 ) is dispatched to the data center node where the input data Data1 and Data2 are located for execution. (T A3 ) and (T B3 ) is a task with high resource requirements and does not directly rely on the data provided by the node. The system calculates the resource adaptability score for each node and prioritizes scheduling to the node with the highest score. The scheduling process is as follows Figure 5 shown.

[0065] In one embodiment, any data center node s is scored for a high resource demand subtask j as follows:

[0066] Score=Score cpu w cpu +Score mem w mem

[0067]

[0068] where w cpu and w mem are weight coefficients, representing the relative importance of CPU and memory resources in the score, and w cpu +w mem = 1. cpu_total and mem_total represent the total amount of CPU and memory resources on node s, respectively.

[0069] cpu_used and mem_used represent the amount of CPU and memory resources currently used on node s. cpu_req and mem_req represent the amount of CPU and memory resources required by subtask j with high resource requirements. When the task is computationally intensive (such as AI model training), w can be increased. cpu The weight value of w is given priority to the matching degree of CPU resources; when the task is memory intensive, w mem The weight value of .

[0070] In summary, the scheduling strategy of this invention adheres to the principle of "moving computation, not data" in the upper-level federated network, scheduling subtasks to the data center nodes in the domain to which they depend. Further decomposition is performed in the lower-level domain subnets, ensuring balanced computational load while dispatching to the data center nodes closest to the data source. This reduces the data transmission load and ensures efficient execution of collaborative tasks.

[0071] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A collaborative pipeline task scheduling method suitable for hierarchical federated networks, characterized in that: The hierarchical federated network is composed of an upper-layer federated network and a lower-layer domain subnet. The method includes: Obtain the topology of the hierarchical federated network and the datasets on each leaf node in the lower domain subnet; Obtain the collaborative pipeline task, the directed acyclic graph of the collaborative pipeline task, and the resource requirements and data requirements of each subtask in the collaborative pipeline task; Decompose the collaborative pipeline task into at least one domain subtask based on the directed acyclic graph and the data requirements of the subtasks. Then, schedule the domain subtask to the corresponding sub-center node in the upper-layer federated network based on the topology of the hierarchical federated network, the dataset on the leaf node, and the data requirements of the domain subtask. The domain subtask is composed of at least one of the subtasks. Based on the resource and data requirements of the subtask, the domain subtask is decomposed into at least one node subtask, and combined with the data set on the leaf node, the node subtask is scheduled to the corresponding leaf node in the lower-level domain subnet; wherein, the node subtask is composed of at least one of the subtasks.

2. The method according to claim 1, characterized in that The collaborative pipeline is decomposed into at least one domain subtask according to the directed acyclic graph, including: Obtain the dependency relationships of each subtask in the collaborative pipeline task based on the directed acyclic graph; The collaborative pipeline task is disassembled according to the dependency relationship of the subtasks, and each domain subtask only depends on the dataset of one domain, so the collaborative pipeline task is disassembled into at least one domain subtask.

3. The method according to claim 1, characterized in that Dispatching the subtasks in this field to the corresponding sub-center nodes in the upper-layer federated network includes: Compute dependencies between domain subtasks; The domain subtask is dispatched to the corresponding sub-center node in the upper-level federated network, and the execution order of the domain subtasks is set according to the dependencies between the domain subtasks to ensure that the subsequent domain subtasks are carried out after the previous domain subtasks are completed.

4. The method according to claim 1, wherein The process of decomposing a domain subtask into at least one node subtask based on the resource and data requirements of the subtask and scheduling the node subtask to the corresponding leaf node in the lower domain subnet in combination with the data set on the leaf node includes: Obtain the subtasks in the domain subtask and determine whether the subtask has high resource requirements based on the resource requirements of the subtask; If the subtask is a resource-intensive algorithm and does not directly depend on any dataset, a resource suitability score is generated for each leaf node based on the leaf node's resource situation, and the subtask is scheduled to the corresponding leaf node in the lower-level domain subnet based on the resource suitability score; When the subtask is not an algorithm with high resource requirements and does not directly depend on any data set, a node subtask is generated based on the data requirements of the subtask, and the node subtask is scheduled to the corresponding leaf node in the lower-level domain subnet based on the data set on the leaf node.

5. The method according to claim 4, characterized in that Generate a resource suitability score for each leaf node based on the resource status of the leaf node, and schedule the subtask to the corresponding leaf node in the lower domain subnet based on the resource suitability score, including: Get the total CPU resources, total memory resources, currently used CPU resources, and currently used memory resources of the leaf node; Get the CPU and memory requirements of the subtask; Calculate the CPU resource score of the leaf node based on the total CPU resources, the currently used CPU resources, and the CPU demand of the subtask. cpu ; Calculate the memory resource score of the leaf node based on the total memory resources, the currently used memory resources, and the memory requirements of the subtask. mem ; Calculate the resource adaptation score of the subtask Score = Score cpu w cpu +Score mem w mem ; Among them, w cpu and w mem Represents the CPU weight coefficient and memory weight coefficient respectively, and w cpu +w mem =1; A leaf node is selected in the lower domain subnet according to the resource adaptability score Score, and the subtask is scheduled to the corresponding leaf node in the lower domain subnet.

6. The method according to claim 5, characterized in that When the subtask is a computationally intensive task, the CPU weight coefficient is greater than the memory weight coefficient; If the subtask is a memory-intensive task, the CPU weight coefficient is smaller than the memory weight coefficient.

7. A collaborative pipeline task scheduling system suitable for hierarchical federated networks, characterized in that: The hierarchical federation network consists of an upper-layer federation network and a lower-layer domain subnet, wherein: The sub-center nodes in the upper-layer federated network include: The task acquisition module is used to obtain the topology of the hierarchical federated network and the data sets on each leaf node in the lower domain subnet; obtain the collaborative pipeline task, the directed acyclic graph of the collaborative pipeline task, and the resource requirements and data requirements of each subtask in the collaborative pipeline task; A domain subtask scheduling module is used to decompose the collaborative pipeline task into at least one domain subtask based on the directed acyclic graph and the data requirements of the subtask, and schedule the domain subtask to the corresponding sub-center node in the upper-level federated network based on the topology of the hierarchical federated network, the data set on the leaf node, and the data requirements of the domain subtask; wherein the domain subtask is composed of at least one of the subtasks; The node subtask scheduling module is used to decompose the domain subtask into at least one node subtask based on the resource requirements and data requirements of the subtask, and schedule the node subtask to the corresponding leaf node in the lower-level domain subnet in combination with the data set on the leaf node; wherein the node subtask is composed of at least one of the subtasks.

8. An electronic device, characterized in that: The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the collaborative pipeline task scheduling method applicable to a hierarchical federated network as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the collaborative pipeline task scheduling method applicable to a hierarchical federated network as described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the computer program product is run on a computer device, the computer device is caused to execute the collaborative pipeline task scheduling method applicable to a hierarchical federated network as described in any one of claims 1 to 6.

Images