Distributed scheduling method, apparatus, device, and storage medium

Abstract

The application discloses a distributed scheduling method and device, equipment and a storage medium, relates to the technical field of multi-party secure calculation, federated learning and distributed scheduling. A specific embodiment of the method comprises: sending a registration application of nodes of a plurality of participants to a central coordinator; periodically sending heartbeat information to the central coordinator, wherein the heartbeat information comprises current system resource states and job states of the nodes of the plurality of participants; starting a resource container on a node of a target participant and jointly running a job with nodes of other participants. The embodiment provides a distributed scheduling mechanism for a distributed system to solve the problem of autonomous coordination of each participant of the distributed system. When applied to a federated computing system, each participant can autonomously participate in federated computing, thereby improving the operation efficiency of the federated computing system and promoting the efficient achievement of federated computing cooperation.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, in particular to the technical field of multi-party secure computation, federated learning, distributed scheduling, and more particularly to a distributed scheduling method, device, equipment and storage medium. BACKGROUND

[0002] Federated computing is a new computing mode that integrates MPC (Secure Multi-party Computation), TEE (Trusted Execution Environment) and data desensitization and other data security and privacy protection technologies to provide a platform for multiple participants to jointly coordinate secure computing.

[0003] Federated computing first needs to solve the coordination and management problem of each participant. Therefore, the current solution represented by federated learning and MPC focuses on how to design a safe and efficient algorithm, and each participant uses the algorithm to complete a specific multi-party secure computation task. Common implementation schemes include software schemes represented by MPC and federated learning and hardware schemes represented by TEE trusted secure hardware. Among them, the software-based technical solution includes MPC, federated learning, etc. MPC is based on Oblivious Transfer (OT), Homomorphic Encryption (HE), Secret Sharing (SS) and other cryptographic protocols to realize secure data computation. The hardware-based technical solution needs the support of hardware manufacturers. Neither of the above two implementation schemes involves the scheduling of each participant. SUMMARY

[0004] Embodiments of the present application propose a distributed scheduling method, device, equipment and storage medium.

[0005] In a first aspect, embodiments of the present application propose a distributed scheduling method, comprising: sending a registration application of nodes of a plurality of participants to a central coordinator; periodically sending heartbeat information to the central coordinator, wherein the heartbeat information includes current system resource states and job states of the nodes of the plurality of participants; starting a resource container on a node of a target participant, and jointly running a job with nodes of other participants, wherein the target participant is selected by the central coordinator from the plurality of participants based on resource requirements of the job.

[0006] In a second aspect, an embodiment of the present application provides a distributed scheduling apparatus, comprising: a registration application module configured to send a registration application of nodes of a plurality of participants to a central coordinator; a heartbeat sending module configured to periodically send heartbeat information to the central coordinator, wherein the heartbeat information comprises current system resource states and job states of the nodes of the plurality of participants; and a joint running module configured to start a resource container on a node of a target participant and jointly run a job with nodes of other participants, wherein the target participant is selected by the central coordinator from the plurality of participants based on resource requirements of the job.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any of the implementations of the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described in any of the implementations of the first aspect.

[0009] The distributed scheduling method, apparatus, device and storage medium provided by the embodiments of the present application can be used to solve the problem of autonomous coordination of participants in a distributed system. When applied to a federated computing system, the participants can autonomously participate in federated computing, thereby improving the running efficiency of the federated computing system and promoting efficient achievement of federated computing cooperation. In addition, the distributed scheduling mechanism can be applied to coordination and scheduling of other distributed systems, including but not limited to distributed scheduling of a big data platform.

[0010] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the drawings. The drawings are intended to better understand the present application and do not constitute a limitation thereof. Among them:

[0012] Figure 1is a schematic diagram of a node scheduling mechanism;

[0013] Figure 2 is a flow chart of one embodiment of a distributed scheduling method according to the present application;

[0014] Figure 3 is a flow chart of one embodiment of a job running step of a distributed scheduling method according to the present application;

[0015] Figure 4 is a flow chart of one embodiment of a task running step of a distributed scheduling method according to the present application;

[0016] Figure 5 is a schematic diagram of a job scheduling mechanism;

[0017] Figure 6 is a structural schematic diagram of one embodiment of a distributed scheduling apparatus according to the present application;

[0018] Figure 7 is a block diagram of an electronic device for implementing a distributed scheduling method according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are meant to be exemplary and not limiting. Therefore, it should be recognized that many modifications and variations of the exemplary embodiments described herein can be made without departing from the scope and spirit of the application. Also, for the purpose of clarity and the brevity, the description below omits the description of well-known functions and structures.

[0020] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] The distributed scheduling method of the embodiments of the present application introduces a distributed scheduling mechanism into a distributed system to solve the problem of autonomous coordination of each participant in the distributed system. The main principle includes two layers of scheduling mechanisms: the first layer of scheduling is a node scheduling mechanism; and the second layer of scheduling is a job scheduling mechanism.

[0022] A participant can deploy a node. The node can provide necessary computing power resources for the participant of the distributed system, including but not limited to CPU (Central Processing Unit), memory, network, storage, etc. Therefore, when the participant participates in the calculation, the participant first needs to ensure that the node can provide the computing power resources required for the calculation.

[0023] Figure 1 A schematic diagram of the node scheduling mechanism is shown. As shown in the schematic diagram, the node scheduling mechanism includes a node manager and a node worker. The node manager is responsible for receiving a task from a task queue and sending the task to the node worker. The node worker is responsible for receiving the task from the node manager and performing the task.Figure 1 As shown, the central coordinator 1011 is deployed on the master server 101. The node 1021, the database 1022 and the resource container 1023 are deployed on the local server of the participant 102. Similarly, the node 1031, the database 1032 and the resource container 1033 are deployed on the local server of the participant 103. The node scheduling mechanism can collect the states of the node 1021 of the participant 102 and the node 1031 of the participant 103, and synchronize the state of the node 1021 of the participant 102 to the participant 103 and the state of the node 1031 of the participant 103 to the participant 102 through the central coordinator 1011. When the participant 102 participates in the calculation, the node 1021 is deployed on the local server, and the node 1021 is used to connect the database 1022 to obtain the private data of the participant 102. Similarly, when the participant 103 participates in the calculation, the node 1031 is deployed on the local server, and the node 1031 is used to connect the database 1032 to obtain the private data of the participant 103. When running the job, the participant 102 and the participant 103 can start the job running calculation task according to the job scheduling mechanism.

[0024] The important value of the distributed system is to protect the sensitive data of the participants, and one principle of the distributed system is to ensure that the original data does not go out of the domain. The central coordinator 1011 of the distributed system only collects the states of the node 1021 of the participant 102 and the node of the participant 103 through the heartbeat mechanism, and does not touch the sensitive data of the participant 102 and the participant 103. The sensitive data is always stored on the local server of the participant 102 and the participant 103. When the node 1021 needs to jointly calculate with the node 1022, all the nodes participating in the joint calculation start the job task, and perform the calculation through the security protocol. The state of the node can include but is not limited to whether the node has been started, whether a running error has occurred, whether the task has been completed, etc.

[0025] It should be understood that the number of participants of the distributed system supports any number of parties, Figure 1 The number of participants in the above is only illustrative. According to the needs of implementation, there can be any number of participants. When applied in a federated computing system, the node is a federated computing node, the calculation is a federated calculation, and the job can be a federated computing job.

[0026] With reference to Figure 2 , which shows a flow 200 of one embodiment of the distributed scheduling method according to the present application.

[0027] In the embodiment, when the distributed system is initialized, the central coordinator is started on the master server, and the nodes of the multiple participants are started on the cluster of participant servers. The cluster of participant servers can include multiple local servers of the participants, and one local server of a participant can start one node. Then, the distributed scheduling method is executed, which includes the following steps.

[0028] In step 201, the registration application of the nodes of the multiple participants is sent to the central coordinator.

[0029] In the embodiment, after the nodes of the multiple participants are started, the registration application can be sent to the central coordinator respectively. The central coordinator can verify the identities of the multiple participants, and the nodes of the participants that pass the verification are used as the computing nodes available for the distributed system.

[0030] In step 202, the heartbeat information is sent to the central coordinator periodically.

[0031] In the embodiment, the nodes of the multiple participants can send the heartbeat information to the central coordinator periodically respectively, so that the central coordinator receives the heartbeat information from the nodes of the multiple participants periodically. The heartbeat information of one node of a participant can include the current system resource state and the job state of the node of the participant. The current system resource state can include but is not limited to the states of CPU, memory, and IO (Input Output).

[0032] In step 203, the resource container is started on the node of the target participant, and the job is run jointly with the nodes of other participants.

[0033] In the embodiment, the central coordinator can receive the resource application of the job, and select the target participant from the multiple participants according to the resource requirement of the job. The node of the target participant can start the resource container to run the job. Meanwhile, the nodes of other participants can jointly run the job. The resource container can include the basic computing resource required for running the job task, including but not limited to CPU, memory, IO, etc. Each resource container runs only one task of the job. The other participants can be the participants other than the target participant in the multiple participants.

[0034] The distributed scheduling method provided by the embodiments of the present application can be used for a distributed system, and can solve the problem of autonomous coordination of each participant in the distributed system. When applied to a federated computing system, each participant can autonomously participate in federated computing, so as to improve the operation efficiency of the federated computing system and promote efficient achievement of federated computing cooperation. In addition, the distributed scheduling mechanism can be applied to coordination and scheduling of other distributed systems, including but not limited to distributed scheduling of a big data platform.

[0035] Further reference is made to Figure 3 FIG. 3 shows a flow 300 of one embodiment of a job running step of the distributed scheduling method according to the present application.

[0036] In the present embodiment, the job scheduling mechanism can start a job running federated secure computing task. The job scheduling mechanism is based on the premise that the nodes can work normally, and uses the CPU, memory, network, storage and other basic computing resources provided by the nodes of the participants to run the job.

[0037] The job running step can be decomposed as follows:

[0038] Step 301: adding data resources of the job on the node of the target participant, and authorizing the nodes of other participants to view metadata information of the data resources.

[0039] In the present embodiment, the target participant can add data resources needed for computing on the node of the participant, and authorize the nodes of other participants to view metadata information of the data resources.

[0040] In the distributed system, the data belongs to the participants, and the distributed system will not send the data of one participant to other participants in plaintext. However, in order to enable other participants to use the data for computing in the distributed system, the other participants need to be authorized to see the metadata information of the data. When the other participants see the metadata information, they can use the data for computing.

[0041] Step 302: creating a job on the node of the target participant, and authorizing the nodes of other participants to view the job.

[0042] In the present embodiment, the target participant can create a job on the node of the participant.

[0043] In the distributed system, the job created by the participant in the distributed system is also private data of the participant, and therefore the central coordinator of the distributed system does not store the job. The job created by the participant at the node end of the participant is stored on the hard disk or other storage device of the node end of the participant. The distributed system analyzes the participant involved in the job, and the participant involved in the job can see the details of the job. In this way, the participant can perform authorization operation on the job according to actual conditions.

[0044] Step 303, starting the job by the node of the target participant, and authorizing the execution of the job by the node of the other participant.

[0045] In the embodiment, the job can be started by the node of the target participant, and the execution of the job is authorized by the node of the other participant.

[0046] After each participant knows the job logic, the participant can decide whether to participate in the job, and select a specific authorization mode to authorize the execution of the job.

[0047] The authorization mode of the job can include but is not limited to at least one of the following:

[0048] 1. Full authorization: authorizing the job to run multiple times (infinite times), and the job logic does not need to be authorized again after modification and adjustment.

[0049] 2. No change authorization: authorizing the job to run multiple times (infinite times), and the job logic needs to be authorized again after modification.

[0050] 3. Run only once: authorizing the job to run only once, and the next time the job needs to be authorized again.

[0051] 4. Refuse to execute: refuse to run the job.

[0052] The job running step provided in the embodiment of the application does not need to send the data resource and the job of the participant to other participants in the process of running the job, and other participants can see the metadata information of the data resource and the details of the job through the authorization mode, thereby ensuring the confidentiality and security of the data resource and the job of the participant.

[0053] Further reference Figure 4 Fig. 4 shows a flow 400 of one embodiment of the task running step of the distributed scheduling method according to the application.

[0054] Step 401, generating AppMaster and multiple AppExecutors in the job running.

[0055] In the embodiment, when the job is started, a job running instance is generated. Two types of tasks, AppMaster and AppExecutor, are generated in the job running. The AppMaster is responsible for monitoring the running state of the job task, and the AppExecutor is responsible for running the job task. The node of the target participant can apply to the central coordinator to start the AppMaster. The central coordinator selects the node with idle resources from the nodes of multiple participants to start the AppMaster. By default, the central coordinator selects the node of the creator of the job to start the AppMaster, and can also specify a specific node to start the AppMaster through configuration.

[0056] In step 402, the job is parsed and compiled into a subtask sequence.

[0057] In the embodiment, after the AppMaster is started, the job logic can be parsed and compiled into a series of subtasks.

[0058] In step 403, a DAG graph is generated based on the upstream and downstream dependency relationship of the subtask sequence.

[0059] In the embodiment, the AppMaster can form a DAG (Directed Acyclic Graph) graph based on the upstream and downstream dependency relationship of the subtask sequence. The AppMaster is responsible for monitoring and maintaining the running state of the DAG graph of all subtasks.

[0060] The execution of these subtasks will be responsible for the AppExecutor. The AppMaster will be responsible for applying computing resources for the task, and will be responsible for monitoring and maintaining the running state of the DAG graph. When all subtasks in the DAG graph are successfully run, it is considered that the job instance is successfully run. If any subtask fails to run, it is considered that the job instance fails to run.

[0061] In step 404, the AppMaster submits multiple AppExecutors to the central coordinator according to the dependency relationship of the DAG graph.

[0062] In the embodiment, the AppMaster can submit multiple AppExecutor tasks to the central coordinator according to the dependency relationship of the DAG graph.

[0063] In step 405, the subtask sequence pushed by the central coordinator is received.

[0064] In the embodiment, the central coordinator can push the subtask sequence to multiple participant nodes at the same time according to the computing resources required by the AppExecutor task.

[0065] Step 406, if the computing resources are sufficient, start multiple AppExecutors to execute the subtask sequence, and register with the AppMaster and periodically report heartbeat information by the multiple AppExecutors.

[0066] In this embodiment, if the computing resources of the nodes of the multiple participants are sufficient, multiple AppExeutor tasks can be respectively started, and registered with the AppMaster and periodically report heartbeat information. If there is not enough computing resources, it can enter a waiting state.

[0067] Step 407, send a start running instruction by the AppMaster to the multiple AppExecutors.

[0068] In this embodiment, the AppMaster can send a start running instruction to the multiple AppExecutors respectively.

[0069] Step 408, run the subtask sequence by the multiple AppExecutors.

[0070] In this embodiment, after receiving the start running instruction, the multiple AppExecutors can respectively run the subtasks in the subtask sequence.

[0071] Step 409, feed back the state of the subtask sequence to the AppMaster by the multiple AppExecutors during running.

[0072] In this embodiment, the multiple AppExecutors can feed back the state of the subtasks in the subtask sequence to the AppMaster during running.

[0073] Step 410, update the subnode state of the DAG graph by the AppMaster, and trigger the subsequent subtasks in the subtask sequence.

[0074] In this embodiment, the AppMaster can update the subnode state of the DAG graph according to the received state, and trigger the subsequent subtasks in the subtask sequence.

[0075] Step 411, if the state of all subnodes of the DAG graph is complete, exit the multiple AppExecutors and the AppMaster.

[0076] In this embodiment, if the state of all subnodes of the DAG graph is complete, the AppMaster can notify the multiple AppExeutor, and the multiple AppExeutor exits in turn, and the AppMaster also exits subsequently. At this time, the whole job running is finished.

[0077] The task running step provided in the embodiments of the present application is to apply computing resources for a job by the AppMaster, and to execute a subtask sequence by the AppExecutor, so that distributed scheduling of the job is implemented. The upstream and downstream dependency relationship and running state of the subtask sequence of the job are recorded by using the DAG graph, so that the relationship between the subtask sequence of the job and the execution of the subtask sequence are facilitated. The subtask sequence of the job is executed based on the DAG graph, so that the subtask sequence of the job can be executed in order. After the job is executed, the AppExecutor and the AppMaster are exited in time, so that the occupation of system resources by the old job is released in time, and the smooth execution of the new job is facilitated.

[0078] Figure 5 A schematic diagram of a job scheduling mechanism is shown. As shown in Figure 5 , a job is created by a participant A, and needs to be participated in computing by a participant B. Two types of tasks, AppMaster and AppExecutor, are generated in the job running. The task running steps are as follows:

[0079] 1. The participant A applies to start the AppMaster to the Coordinator (central coordinator).

[0080] 2. The Coordinator selects the Agent A of the participant A to start the AppMaster.

[0081] 3. After the AppMaster is started, the job logic is parsed and compiled to form a series of subtasks, and the upstream and downstream dependency relationship of the subtasks is formed into a DAG graph.

[0082] 4. The AppMaster submits the AppExecutor task to the Coordinator in turn according to the dependency relationship of the DAG graph.

[0083] 5. The Coordinator pushes the subtasks to the Agent A of the participant A and the Agent B of the participant B at the same time according to the computing resources required by the AppExecutor task.

[0084] 6. If the Agent A and the Agent B have sufficient computing resources, the AppExeutor A and the AppExecutor B will be started respectively. If the computing resources are not sufficient, the Agent A and the Agent B will wait.

[0085] 7. The AppExecutor A and the AppExecutor B register with the AppMaster and report heartbeat information in time.

[0086] 8、AppMaster sends start running instruction to AppExecutor A and AppExecutor B. After receiving the start running instruction, AppExecutor A and AppExecutor B run the subtasks.

[0087] 9、AppExecutor A and AppExecutor B feed back the subtask state to AppMaster during the running process. AppMaster updates the node state of the subtasks corresponding to the DAG graph, and triggers to start the subsequent subtasks. After all the subtasks of the entire DAG graph are run, AppMaster notifies AppExeutor, and AppExeutor A and AppExecutor B quit in turn, and AppMaster also quits subsequently.

[0088] Further referring to Figure 6 , as an implementation of the method shown in the above figures, the present application provides an embodiment of a distributed scheduling device, which corresponds to the method embodiment shown in Figure 2 , and the device can be specifically applied to various electronic devices.

[0089] As shown in Figure 6 , the distributed scheduling device 600 of the embodiment can include a registration application module 601, a heartbeat sending module 602, and a joint running module 603. The registration application module 601 is configured to send a registration application of nodes of a plurality of participants to a central coordinator. The heartbeat sending module 602 is configured to periodically send heartbeat information to the central coordinator, wherein the heartbeat information includes current system resource states and job states of the nodes of the plurality of participants. The joint running module 603 is configured to start a resource container on a node of a target participant, and jointly run a job with nodes of other participants, wherein the target participant is selected by the central coordinator from the plurality of participants based on resource requirements of the job.

[0090] In the embodiment, in the distributed scheduling device 600: the specific processing of the registration application module 601, the heartbeat sending module 602, and the joint running module 603 and the technical effects brought by the specific processing can be respectively referred to the related descriptions of steps 201-203 in the corresponding embodiment, which will not be repeated here. Figure 2

[0091] ​In some optional implementation of the embodiment, the joint running module 603 comprises: a resource adding submodule (not shown in the figure) configured to add data resources of the job on the node of the target participant, and authorize the nodes of the other participants to view metadata information of the data resources; a job creating submodule (not shown in the figure) configured to create the job on the node of the target participant, and authorize the nodes of the other participants to view the job; a joint running submodule (not shown in the figure) configured to start the job by the node of the target participant, and authorize the nodes of the other participants to execute the job.

[0092] In some optional implementation of the embodiment, the joint running submodule comprises: a generating unit (not shown in the figure) configured to generate the AppMaster and the plurality of AppExecutors in the job running; a compiling unit (not shown in the figure) configured to compile the job into the subtask sequence; an executing unit (not shown in the figure) configured to apply for computing resources for the job by the AppMaster, and execute the subtask sequence by the plurality of AppExecutors.

[0093] In some optional implementation of the embodiment, the joint running submodule further comprises: a generating unit (not shown in the figure) configured to generate the DAG graph based on the upstream and downstream dependencies of the subtask sequence; a maintaining unit (not shown in the figure) configured to maintain the running state of the DAG graph by the AppMaster.

[0094] In some optional implementation of the embodiment, the executing unit comprises: a submitting subunit (not shown in the figure) configured to sequentially submit the plurality of AppExecutors to the central coordinator according to the dependencies of the DAG graph by the AppMaster; a receiving subunit (not shown in the figure) configured to receive the subtask sequence pushed by the central coordinator; an executing subunit (not shown in the figure) configured to start the plurality of AppExecutors to execute the subtask sequence if the computing resources are sufficient.

[0095] In some optional implementation of the embodiment, the executing unit further comprises: a waiting subunit (not shown in the figure) configured to enter a waiting state if the computing resources are insufficient.

[0096] In some optional implementation of the embodiment, the executing subunit is further configured to: register and periodically report heartbeat information to the AppMaster by the plurality of AppExecutors; send a start running instruction to the plurality of AppExecutors by the AppMaster; execute the subtask sequence by the plurality of AppExecutors.

[0097] In some optional implementations of the embodiment, the maintenance unit is further configured to: feed back, by the plurality of AppExecutors, the state of the subtask sequence to the AppMaster during running; update, by the AppMaster, the subnode state of the DAG graph, and trigger the subsequent subtask in the subtask sequence; and exit the plurality of AppExecutors and the AppMaster if all the subnode states of the DAG graph are in the completed state.

[0098] In some optional implementations of the embodiment, the node of the target participant applies to the central coordinator to start the AppMaster, and the central coordinator selects the node with idle resources from the nodes of the plurality of participants to start the AppMaster.

[0099] In some optional implementations of the embodiment, the authorization mode of the node of the other participant includes at least one of the following: full authorization, no-change authorization, run only once, and refuse execution.

[0100] In some optional implementations of the embodiment, the central coordinator is started on a master server, and the nodes of the plurality of participants are started on a participant server cluster.

[0101] According to the embodiments of the present application, the present application further provides an electronic device and a readable storage medium.

[0102] As Figure 7 shown in FIG. 1, is a block diagram of an electronic device for implementing a distributed scheduling method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown in FIG. 1, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0103] As Figure 7As shown, the electronic device includes one or more processors 701, memory 702, and interfaces 703 for connecting various parts of the electronic device, including high-speed interfaces and low-speed interfaces. Various parts are connected using different buses, and can be mounted on a common motherboard or otherwise mounted as appropriate. The processor can process instructions for execution within the electronic device, including instructions stored in the memory or on the memory to display graphical information for a GUI on an external input / output device, such as a display device coupled to the interface. In other implementations, multiple processors and / or multiple buses can be employed as appropriate, as well as multiple memories and types of memory. Also, various parts of the electronic device can be implemented using one or more integrated circuits, for example, an ASIC. It is to be understood that the electronic device can employ any of a number of different hardware configurations, software configurations, and firmware configurations. Figure 7 The processor 701 is taken as an example in the present application.

[0104] The memory 702 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by at least one processor, so that the at least one processor executes the distributed scheduling method provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to execute the distributed scheduling method provided by the present application.

[0105] The memory 702, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules of the distributed scheduling method in the embodiments of the present application (for example, the registration application module 601, the heartbeat sending module 602 and the joint running module 603 shown in the figure). The processor 701 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 702, that is, implements the distributed scheduling method in the above method embodiments. Figure 6 The registration application module 601, the heartbeat sending module 602 and the joint running module 603 shown in the figure). The processor 701 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 702, that is, implements the distributed scheduling method in the above method embodiments.

[0106] The memory 702 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function; the data storage area can store data created according to the use of the electronic device of the distributed scheduling method and the like. In addition, the memory 702 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 702 can optionally include a memory remotely arranged with respect to the processor 701, which can be connected to the electronic device of the distributed scheduling method through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0107] The electronic device of the distributed scheduling method can further include an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703, and the output device 704 can be connected through a bus or other means, Figure 7 The connection through the bus is taken as an example.

[0108] The input device 703 can receive inputted digital or character information, and generate key signal input related to user settings and function control of the electronic device of the distributed scheduling method, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 704 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor), etc. The display device can include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0109] Various embodiments of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0110] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0111] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0112] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0113] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0114] According to the technical solution of the present application, the nodes of multiple participants can first send a registration application to the central coordinator, and periodically send heartbeat information to the central coordinator; then start a resource container on the node of the target participant selected by the central coordinator, and jointly run a job with the nodes of other participants. A distributed scheduling mechanism is provided for the distributed system to solve the problem of autonomous coordination of the participants of the distributed system. When applied in a federated computing system, the participants can autonomously participate in federated computing, thereby improving the operation efficiency of the federated computing system and promoting the efficient achievement of federated computing cooperation. In addition, the distributed scheduling mechanism can be applied to the coordination and scheduling of other distributed systems in addition to the federated computing system, including but not limited to the distributed scheduling of a big data platform.

[0115] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps described in this application can be performed in parallel, in series, in a different order, or any combination thereof, as long as the desired results of the technology disclosed in this application are achieved, which is not limited herein.

[0116] The specific embodiments described above are not intended to be limiting, and any modifications, combinations, sub-combinations and alternatives are intended to be included herein. Thus, while there has been set forth above certain embodiments of this application, various modifications, changes and variations can be made thereto and substitutions of equivalents can occur to those skilled in the art without departing from the spirit and scope of the application as set forth in the following claims.

Claims

1. A distributed scheduling method, comprising: Send registration requests from multiple participating nodes to the central coordinator; Heartbeat information is periodically sent to the central coordinator, wherein the heartbeat information includes the current system resource status and job status of the nodes of the multiple participating parties, and is sent by the nodes of the multiple participating parties to the central coordinator. A resource container is started on the node of the target participant and the job is run jointly with the nodes of other participants, wherein the target participant is selected by the central coordinator from the plurality of participants based on the resource requirements of the job; The step of starting resource containers on the target participant's nodes and jointly running jobs with the nodes of other participants includes: Add the data resources of the task to the node of the target participant, and authorize the nodes of the other participants to view the metadata information of the data resources; Create the job on the node of the target participant, and authorize the nodes of the other participants to view the job; The job is initiated by a node of the target participant and authorized for execution by nodes of other participants. The authorization methods of the nodes of other participants include at least one of the following: full authorization, authorization without changes, run only once, and refuse to execute.

2. The method according to claim 1, wherein, The process of initiating the job by a node of the target participant and authorizing the execution of the job by nodes of the other participants includes: During the execution of the job, an AppMaster and multiple AppExecutors are generated. The task is parsed and compiled into a sequence of subtasks; The AppMaster requests computing resources for the job, and the plurality of AppExecutors execute the sequence of subtasks.

3. The method according to claim 2, wherein, The process of initiating the job by a node of the target participant and authorizing the execution of the job by nodes of the other participants further includes: A DAG graph is generated based on the upstream and downstream dependencies of the sub-task sequence; The AppMaster maintains the running state of the DAG graph.

4. The method according to claim 3, wherein, The process of the AppMaster requesting computing resources for the job and the plurality of AppExecutors executing the subtask sequence includes: The AppMaster submits the multiple AppExecutors to the central coordinator sequentially according to the dependencies in the DAG graph; Receive the sub-task sequence pushed by the central coordinator; If sufficient computing resources are available, the multiple AppExecutors are launched to execute the subtask sequence.

5. The method according to claim 4, wherein, The process of the AppMaster requesting computing resources for the job and the plurality of AppExecutors executing the subtask sequence further includes: If computing resources are insufficient, enter a waiting state.

6. The method according to claim 4, wherein, The step of initiating the execution of the subtask sequence by the plurality of AppExecutors includes: The multiple AppExecutors register with the AppMaster and periodically report heartbeat information. The AppMaster sends a start-run command to the multiple AppExecutors; The sequence of subtasks is run by the plurality of AppExecutors.

7. The method according to claim 6, wherein, The maintenance of the running state of the DAG graph by the AppMaster includes: The multiple AppExecutors shall feed back the status of the subtask sequence to the AppMaster during the execution process; The AppMaster updates the state of the child nodes in the DAG graph and triggers subsequent subtasks in the subtask sequence; If all child nodes of the DAG graph are in a completed state, exit the multiple AppExecutors and the AppMaster.

8. The method according to any one of claims 2-7, wherein, The nodes of the target participants request the central coordinator to start the AppMaster, and the central coordinator selects nodes with idle resources from the nodes of the multiple participants to start the AppMaster.

9. The method according to any one of claims 1-7, wherein, Start the central coordinator on the master server and start the nodes of the multiple participants on the participant server cluster.

10. A distributed scheduling device, comprising: The registration application module is configured to send registration applications from multiple participating nodes to the central coordinator; The heartbeat sending module is configured to periodically send heartbeat information to the central coordinator, wherein the heartbeat information includes the current system resource status and job status of the nodes of the multiple participating parties, and is sent by the nodes of the multiple participating parties to the central coordinator. The joint operation module is configured to launch resource containers on the nodes of the target participant and jointly run jobs with the nodes of other participants, wherein the target participant is selected by the central coordinator from the plurality of participants based on the resource requirements of the job; The joint operation module includes: The resource addition submodule is configured to add the data resources of the job to the nodes of the target participant, and to authorize the nodes of the other participants to view the metadata information of the data resources; The job creation submodule is configured to create the job on the node of the target participant and authorize the nodes of the other participants to view the job; The joint operation submodule is configured to initiate the job by a node of the target participant and authorize the execution of the job by nodes of other participants, wherein the authorization methods of the nodes of other participants include at least one of the following: full authorization, authorization without changes, run only once, and refuse to execute.

11. The apparatus according to claim 10, wherein, The joint operation submodule includes: The generation unit is configured to generate an AppMaster and multiple AppExecutors during the execution of the job; The compilation unit is configured to parse and compile the job into a sequence of subtasks; The execution unit is configured to request computing resources for the job by the AppMaster and to execute the subtask sequence by the plurality of AppExecutors.

12. The apparatus according to claim 11, wherein, The joint operation submodule also includes: The generation unit is configured to generate a DAG graph based on the upstream and downstream dependencies of the subtask sequence; The maintenance unit is configured to maintain the running state of the DAG graph by the AppMaster.

13. The apparatus according to claim 12, wherein, The execution unit includes: Submission subunits are configured to be submitted by the AppMaster to the central coordinator in sequence according to the dependencies of the DAG graph; The receiving subunit is configured to receive the sequence of subtasks pushed by the central coordinator; The execution subunit is configured to launch the plurality of AppExecutors to execute the sequence of subtasks if sufficient computing resources are available.

14. The apparatus according to claim 13, wherein, The execution unit further includes: The waiting subunit is configured to enter a waiting state if computing resources are insufficient.

15. The apparatus according to claim 13, wherein, The execution subunit is further configured to: The multiple AppExecutors register with the AppMaster and periodically report heartbeat information. The AppMaster sends a start-run command to the multiple AppExecutors; The sequence of subtasks is run by the plurality of AppExecutors.

16. The apparatus according to claim 15, wherein, The maintenance unit is further configured to: The multiple AppExecutors shall feed back the status of the subtask sequence to the AppMaster during the execution process; The AppMaster updates the state of the child nodes in the DAG graph and triggers subsequent subtasks in the subtask sequence; If all child nodes of the DAG graph are in a completed state, exit the multiple AppExecutors and the AppMaster.

17. The apparatus according to any one of claims 11-16, wherein, The nodes of the target participants request the central coordinator to start the AppMaster, and the central coordinator selects nodes with idle resources from the nodes of the multiple participants to start the AppMaster.

18. The apparatus according to any one of claims 10-16, wherein, Start the central coordinator on the master server and start the nodes of the multiple participants on the participant server cluster.

19. An electronic device comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

20. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

Images