Efficient scheduling method and system for multi-job longitudinal federated learning training based on rolling time domain

Abstract

The present invention relates to a method and system for efficiently scheduling multi-job longitudinal federated learning training based on a rolling time domain. It mainly includes S100, initializing the first time domain as the current time domain, denoted as O horizon , the current schedule S current Initialized to empty, S200, current time domain O horizon Contains a set of operations to be scheduled, using a method g(•) to solve the current time domain O horizon The scheduling problem of the current time domain O is obtained through g(•) horizon Temporary solution S horizon ,S300, use temporary solution S horizon The first m operations in form a temporary schedule S temp , through temporary scheduling S temp Update the current schedule S current If the problem is solved, return to the current schedule S current Otherwise, set the new time domain to the current time domain O horizon , and returns to step S200. The method of the present application achieves an overall balance between job completion time and computing efficiency, and can obtain a scheduling solution with a shorter total completion time for multiple jobs under limited scheduling overhead.

Description

Technical Field

[0001] The present invention relates to the field of multi-job longitudinal federated learning training, and in particular to an efficient scheduling method and system for multi-job longitudinal federated learning training based on a rolling time domain. Background Art

[0002] In recent years, federated learning has become an effective method for collaborative training of decentralized data. Vertical federated learning (VFL) is a scenario of cross-data island collaboration in federated learning, in which multiple participants with different features of the same dataset jointly train machine learning models without exposing their original data.

[0003] As society develops, it is sometimes necessary to process multiple VFL jobs at the same time. For example, an e-commerce company cooperates with a bank to build a VFL model. The e-commerce company wants to recommend products to customers, while the bank wants to predict credit risk. In this case, two VFL models are trained simultaneously using the same participants. Some of these operations can be performed in parallel. To solve this problem, the multi-job VFL framework is introduced.

[0004] like Figure 1 As shown in the figure, the training multi-job VFL framework is characterized by a coordinator to control the scheduling of multiple jobs. The coordinator is in charge of the scheduling of jobs, and the data party performs specific model training. The coordinator can be a trusted third party or any party in the system. Before the training starts, the coordinator makes a plan to determine the execution order of jobs in each participant. The participants then communicate with the coordinator to update their own plans. Unlike the single-job VFL, in the first step of the initialization of the multi-job VFL framework, in addition to the PSI (private set intersection) process, the coordinator also collects the status of the current job and sends a scheduling plan to the data party. The coordinator is in charge of the scheduling of jobs, and the data party performs specific model training.

[0005] A difficult problem to solve in the multi-job VFL framework is how to find a suitable scheduling method to make the total completion time of multiple jobs as short as possible. Summary of the invention

[0006] Based on this, an efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain is provided. This method is conducive to finding a suitable scheduling scheme to make the total completion time of multiple jobs as short as possible.

[0007] An efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain, including:

[0008] S100, initialize the first time domain as the current time domain, denoted as O horizon , the current schedule Scurrent Initialized to empty,

[0009] S200, current time domain O horizon Contains a set of operations to be scheduled, using a method g(•) to solve the current time domain O horizon The scheduling problem of the current time domain O is obtained through g(•) horizon Temporary solution S horizon ,

[0010] S300, use temporary solution horizon The first m operations in form a temporary schedule S temp , through temporary scheduling S temp Update the current schedule S current If the problem is solved, return to the current schedule S current Otherwise, set the new time domain to the current time domain O horizon , and returns to step S200.

[0011] In one embodiment, the g(•) is a commercial solver.

[0012] In one embodiment, the g(•) is OR-Tools.

[0013] In one embodiment, in step S100, an O is defined. left Collection, O left The collection contains all operations to be scheduled.

[0014] In step S200, the current time domain O horizon All operations in O left Set, and from O left Select the distance S from the current schedule in the collection current Each operation within the h distance forms the current time domain O horizon In step S300, 0 <m<h。

[0015] In one embodiment, in step S300, a temporary solution S is used. horizon The first m operations in form a temporary schedule S temp Afterwards, from O left Set to remove temporary schedule S temp The operations included in the above, and the said if the problem is solved means O left The collection is empty.

[0016] In one embodiment, further comprising S250, if O left Remove the temporary solution S from the set horizon After that, O leftThe set is empty, directly use the temporary solution S horizon Join the current schedule S current , and the current schedule S current Return, if O left Remove the temporary solution S from the set horizon After that, O left If the set is not empty, execute step S300.

[0017] In one embodiment, by temporarily scheduling S temp Update the current schedule S current This means that each temporary schedule S temp The operations contained are sequentially spliced ​​into the current schedule S current .

[0018] An efficient scheduling system for multi-job longitudinal federated learning training based on rolling time domain, including a scheduling information acquisition module and a scheduling information processing module.

[0019] The scheduling information acquisition module is used to obtain all operations to be scheduled.

[0020] The scheduling information processing module is used to execute operations corresponding to the efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain.

[0021] A computer storage medium stores at least one executable instruction, which enables a processor to execute the operation of the efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain.

[0022] A computer device includes: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus, and the memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain.

[0023] The beneficial effects of this application are:

[0024] 1. The method of the present application is conducive to finding a suitable scheduling scheme to make the total completion time of multiple jobs as short as possible.

[0025] 2. The method of the present application achieves an overall balance between job completion time and computational efficiency, and can obtain a scheduling solution with a shorter total completion time for multiple jobs under limited scheduling overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a multi-job VFL framework diagram.

[0027] Figure 2 This is a flowchart of an efficient scheduling method for multi-job longitudinal federated learning training based on a rolling time domain according to an embodiment of the present application.

[0028] Figure 3 The comparison chart of different scheduling methods under the first experimental condition is shown in Figure 2. The experimental conditions are: K=3, J=3, r j =[1, 10].

[0029] Figure 4 This is a comparison chart of different scheduling methods under the second experimental condition. The experimental conditions are: K=3, J=3, r j =[1, 50].

[0030] Figure 5 The comparison chart of different scheduling methods under the third experimental condition is shown in Figure 2. The experimental conditions are: K=3, r j =3, J=[1, 10]. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0032] An embodiment of the present application provides an efficient scheduling method for multi-job vertical federated learning training based on a rolling time domain. The pseudo code corresponding to the efficient scheduling method for multi-job vertical federated learning training based on a rolling time domain of the present application is shown in Table 1. Here, the symbol To indicate that o is In a multi-job VFL process, all worker nodes have a set of pending operations. In scheduling, each worker node is assigned a set of operations, where Is a working node The scheduling.

[0033] For two schedules = , …, > and =< , …, >, use ≺ To express and The concatenation of two schedules, namely ≺ =< , …, , , …, >. The dth operation in job j is represented by It is associated with three elements: a set of preconditions , Processing time and worker nodes Specifically, the operation Need to be on the working node An uninterrupted processing time . Precondition set is a set of Preconditions to be completed before starting. The prerequisites of all operations can be represented as a directed acyclic graph (DAG), where nodes represent operations and edges represent preconditions. The prerequisites of operations and working nodes are determined by the model training process. Processing time It can be estimated by previous training results.

[0034] Table 1

[0035]

[0036] Combining Table 1 and Figure 2 As shown, the method of one embodiment of the present application includes the following steps.

[0037] S100, initialize the first time domain as the current time domain, denoted as O horizon , the current schedule S current Initialize to empty, define an O left Collection, O left The collection contains all pending operations.

[0038] S200, current time domain O horizon Contains a set of operations to be scheduled, the current time domain O horizon All operations in O left Set, and from O left Select the distance S from the current schedule in the collection current Each operation within the h distance forms the current time domain O horizon , using a method for solving small-scale problems g(•) to solve the current time domain O horizon The scheduling problem of the current time domain O is obtained through g(•) horizon Temporary solution S horizon .

[0039] S250, if O left Remove the temporary solution S from the set horizon After the corresponding operations, O left If the set is empty, it means that this is the last temporary solution. In this case, directly use the temporary solution S horizon Splice to the current schedule Scurrent , and return the current schedule S current . That is, at this time, a reasonable scheduling scheme has been obtained. Under this scheme, the total completion time of multiple jobs is relatively short. If O left After removing the temporary solution S horizon from the corresponding operations in the set, if O left The set is not empty, then execute step S300.

[0040] S300: Use the first m operations in the temporary solution S horizon to form a temporary schedule S temp , where 0 < m < h. Remove the operations included in the temporary schedule S left from the O temp set, and update the current schedule S temp through the temporary schedule S current . If the O left set is empty, return the current schedule S current , otherwise, set the new time domain as the current time domain O horizon , and return to step S200.

[0041] It can be understood that after the current schedule S current is returned, it means that all the operations in the O left set are spliced into the current schedule S current . The current schedule S current is a relatively reasonable scheduling scheme. Under this scheme, the total completion time of multiple jobs is relatively short.

[0042] It should be noted that updating the current schedule S temp through the temporary schedule S current means splicing the operations included in each temporary schedule S temp into the current schedule S current in turn.

[0043] In one embodiment, the above g(•) is a commercial solver. Here, g(•) can apply various existing commercial solvers. For example, the g(•) is OR-Tools.

[0044] It should be noted that the operations within h distance from the current schedule S current are selected to form the current time domain O horizon . The distance from any node to the current schedule S current is defined in the prerequisite directed acyclic graph (DAG). In the directed acyclic graph (DAG), the weights of all edges are set to 1. The operations in the current schedule S current form a node set. The h distance is from The shortest distance to any node in the node set. All operations without prerequisites are connected to a virtual node in the directed acyclic graph (DAG). If ,but arrive The distance is The operations whose distance to the virtual node is less than or equal to m from one of the entry nodes are called the first m operations. The temporary solution is expressed as , select The first m operations in , if it is the last time range, that is, a certain is the last temporary solution, then select all operations. If an operation exist If there are no incoming arcs from other nodes in the node set, then it is defined as The entry node.

[0045] The effect of the above method of the present application is verified by specific experiments below.

[0046] In the experiment, parties with different computing and communication resources were simulated. A set of VFL training jobs with different computing and communication resource requirements were randomly generated for scheduling. Google's OR-Tools was used as the g(·) function in the efficient scheduling method for multi-job longitudinal federated learning training based on rolling horizon in this application. Each experiment was repeated 50 times, and the average results of 50 runs were reported.

[0047] For comparison, the following three scheduling methods were selected as control experiments:

[0048] 1. Sequential method: The order of jobs is arbitrary. The operations in the schedule are ordered according to the jobs they belong to. This means that the models are trained one after another, which is the same as training a single job serially.

[0049] 2. Greedy method: Start with an empty schedule and keep adding new operations to the schedule until all operations are scheduled. Each time a new operation is added, its start time is calculated as described in formulas (1) and (2). The first operation to start among all the remaining operations is always added to the existing schedule. The start time of operation d in job j is expressed as Given a schedule, the start time of all operations can be estimated. Assume that in a multi-job VFL system, all operations start immediately once they are ready. For operations other than transmission tasks, it can be expressed as:

[0050] , (1)

[0051] in, Represents the end time of the prerequisite operation. The prerequisite operation means that, in terms of logic, B needs A to be done first, so A is the prerequisite operation of B. Indicates the end time of other operations on the same working node before this operation. is the first job to be processed, defining the start time as 0. For a pair of transfer operations d1 and d2, there is an additional constraint that they need to start at the same time. Therefore, the earliest start time is the later of the two operations. This can be expressed as:

[0052] , (2)

[0053] and It is calculated by formula (1).

[0054] 3. Solver method: Use OR-Tools as the optimization solver for the entire problem.

[0055] The experimental results are as follows Figures 3 to 5 As shown, the rolling horizon in the figure is the efficient scheduling method for multi-job vertical federated learning training based on rolling horizon of the present application.

[0056] Figure 3 Shows when the round number r j Comparison results of different scheduling methods when increasing, here r j =[1, 10]. This includes job completion time and scheduling overhead. Figure 3 Figure (a) is the job completion time graph, and Figure (b) is the scheduling overhead graph. The number of participants K is fixed at K=3, and the number of jobs J is fixed at J=3. According to the experimental results, the solver method can always produce a solution with the minimum job completion time. However, the scheduling overhead of the solver method is significantly greater than that of other methods, which limits its use in practical systems. At the same time, the method of the present application (rolling horizon) is superior to the sequential method and the greedy method in terms of job completion time. Although the scheduling overhead is higher than that of the sequential method and the greedy method, it is still within an acceptable range and is much smaller than the scheduling overhead of the solver method.

[0057] When the number of rounds r is further increased jWhen the number of rounds is larger, the scheduling overhead of the solver method becomes very large and has no practical application value. j Here we only compare the performance of the method of this application (rolling horizon) with the sequential method and the greedy method. Figure 4 shows the performance of the method when the number of rounds r is increased. j The range is to r j =[1, 50]. Figure 4 Figure (a) is the job completion time graph, and Figure (b) is the scheduling overhead graph. The method (rolling horizon) of this application is superior to the sequential method (sequential) and the greedy method (greedy) in terms of job completion time, and has an acceptable scheduling overhead.

[0058] Figure 5 shows the comparison results of different scheduling methods when increasing the number of jobs J, J = [1, 10]. Figure 5 Figure (a) is the job completion time diagram, and Figure (b) is the scheduling cost diagram. At this time, the number of participants K is K = 3, and the number of rounds is r j = 3. Similar results can be observed, that is, the method of the present application (rolling horizon) achieves a good balance between job completion time and scheduling overhead. That is, a scheduling scheme can be obtained with a small scheduling overhead, which can make the job completion time very short.

[0059] Based on the above experiments, the following conclusions can be drawn: In terms of job completion time, the solution generated by the solver method (solver) is better than the method of this application (rolling horizon). The method of this application (rolling horizon) is better than the sequential method (sequential) and the greedy method (greedy). However, algorithms that achieve better results also require more overhead. The solver method (solver) requires exponential time (scheduling overhead). The method of the application (rolling horizon) requires linear time (scheduling overhead). The method of this application (rolling horizon) achieves an overall balance between effect and efficiency, and has more practical value.

[0060] An embodiment of the present application provides an efficient scheduling system for multi-job longitudinal federated learning training based on a rolling time domain, including a scheduling information acquisition module and a scheduling information processing module.

[0061] The scheduling information acquisition module is used to obtain all operations to be scheduled.

[0062] The scheduling information processing module is used to execute operations corresponding to the efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain.

[0063] An embodiment of the present application provides a computer storage medium, in which at least one executable instruction is stored, and the executable instruction enables a processor to execute the operations of the efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain.

[0064] An embodiment of the present application provides a computer device, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus, and the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain.

[0065] As an embodiment of the present invention, the communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0066] As an implementation mode of the present invention, the communication interface is used for communication between the above-mentioned terminal and other devices.

[0067] As an embodiment of the present invention, the memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0068] As an implementation mode of the present invention, the above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0069] Different from the prior art, the method of the present application is conducive to finding a suitable scheduling solution to make the total completion time of multiple jobs as short as possible. The method of the present application achieves an overall balance between the calculation results and the calculation efficiency, that is, the method of the present application can obtain an acceptable scheduling solution with a small scheduling overhead, which can make the total completion time of multiple jobs as short as possible.

[0070] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain, characterized in that: include: S100, initialize the first time domain as the current time domain, denoted as O horizon , the current schedule S current Initialize to empty, define an O left Collection, O left The collection contains all operations to be scheduled. S200, current time domain O horizon All operations in O left Set, and from O left Select the distance S from the current schedule in the collection current Each operation within the h distance forms the current time domain O horizon , using a method for solving small-scale problems g(•) to solve the current time domain O horizon The scheduling problem is g(•) is a commercial solver OR-Tools. The current time domain O is obtained through g(•). horizon Temporary solution S horizon , S250, if O left Remove the temporary solution S from the set horizon After that, O left The set is empty, directly use the temporary solution S horizon Join the current schedule S current , and the current schedule S current Return, if O left Remove the temporary solution S from the set horizon After that, O left If the set is not empty, then execute step S300. S300, use temporary solution horizon The first m operations in form a temporary schedule S temp , each temporary schedule S temp The operations contained are sequentially spliced ​​into the current schedule S current , if O left If the collection is empty, return the current schedule S current Otherwise, set the new time domain to the current time domain O horizon , the new time domain is O left Remove the temporary solution S from the set horizon After that, select the distance from the current schedule S current Each operation within the distance h, and return to step S200, In the step S300, the first m (0 < m < h) operations in the temporary solution S horizon are used to form a temporary schedule S temp After that, the operations included in the temporary schedule S left are removed from the set O temp ​ In the directed acyclic graph DAG, the weights of all edges are set to 1, and the current schedule S current The operations in form a node set, and the h distance is from The shortest distance to any node in the node set, the dth operation in job j is expressed as .

2. An efficient scheduling system for multi-job longitudinal federated learning training based on rolling time domain, characterized in that: It includes a scheduling information acquisition module and a scheduling information processing module. The scheduling information acquisition module is used to obtain all operations to be scheduled. The scheduling information processing module is used to perform operations corresponding to the efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain described in claim 1.

3. A computer storage medium, characterized in that: The computer storage medium stores at least one executable instruction, which enables the processor to perform the operation of the efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain as described in claim 1.

4. A computer device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus, and the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the efficient scheduling method for multi-job longitudinal federated learning training based on rolling time domain as described in claim 1.

Images