A data-distributed batch scheduling method for graph neural network training tasks

Abstract

The present invention discloses a data-distributed graph neural network training task batch scheduling method, which belongs to the technical field of task scheduling in distributed computing. The graph neural network training task is referred to as a task, the system that executes the graph neural network training task is referred to as a distributed system, and an independently operable computing unit in the distributed system is referred to as a computing node; at the same time, it is assumed that all computing nodes in the distributed system are isomorphic, and each computing node contains at least one accelerator card for graph neural network training; the batch scheduling method specifically includes the following steps: step 1, before the scheduling starts, initialize various parameters; step 2, record the submitted tasks and obtain the task attributes, and add the submitted tasks to the task list; step 3, use a genetic algorithm to schedule tasks. The present invention can effectively improve the utilization rate of accelerator card resources, maximize resource utilization, and reduce energy consumption.

Description

Technical Field

[0001] The present invention belongs to the technical field of task scheduling in distributed computing, and specifically relates to a data-distributed graph neural network training task batch scheduling method. Background Art

[0002] In recent years, graph structured data and graph neural networks have been widely used in social networks, communication networks, transportation networks and other fields. As the scale of graph data continues to grow, distributed training of graph neural networks has become the mainstream solution for graph neural network training for large-scale graph data. Among them, the data distribution method is a main method for realizing distributed training of graph neural networks. When a graph neural network training task is deployed in a distributed system composed of multiple or even a large number of computing nodes, each computing node running the task has a complete graph neural network and a part of the graph data. After completing a training iteration for the graph data it has, it needs to communicate with other computing nodes to exchange necessary data; after completing the data exchange, it enters the next iterative training. In this process, how to schedule the training tasks is a core issue, and it is also a problem that the present invention needs to solve. Summary of the invention

[0003] In order to solve the above problems, the present invention proposes a data-distributed graph neural network training task batch scheduling method. The method aims to improve the resource utilization of the distributed system and regularly schedules the submitted graph neural network training tasks in a batch processing manner.

[0004] The technical solution of the present invention is as follows:

[0005] A data-distributed graph neural network training task batch scheduling method is proposed. The graph neural network training task is referred to as a task, the system that executes the graph neural network training task is referred to as a distributed system, and an independently running computing unit in the distributed system is referred to as a computing node. At the same time, it is assumed that all computing nodes in the distributed system are homogeneous, and each computing node contains at least one accelerator card for graph neural network training. The batch scheduling method specifically includes the following steps:

[0006] Step 1: Initialize various parameters before scheduling begins;

[0007] Step 2: Record the submitted task and obtain the task attributes, and add the submitted task to the task list;

[0008] Step 3: Use genetic algorithm for task scheduling.

[0009] Furthermore, the specific process of step 1 is: Initialize the task list Empty set; set the task scheduling time interval , that is, every Perform a task scheduling; use Indicates the number of accelerator cards in the distributed system, using Indicates The time when the accelerator card starts to execute the task is Indicates The total service time of the accelerator card before the current scheduling; Indicates the current time, and its value increases automatically as time goes by; initially, , , .

[0010] Furthermore, the specific process of step 2 is as follows:

[0011] Step 2.1, initialize the counter , used to record the number of tasks in the task list; determine the current time Whether the pre-set conditions are met, if they are met or , record the submitted task and obtain the task attributes; Express request Divide by The remainder of

[0012] Step 2.2: After receiving the submitted task, Increase by 1 and submit the The task is recorded as ;

[0013] Step 2.3, obtain Number of accelerator cards required , the calculation method is to solve the following minimization problem:

[0014] (1);

[0015] in, for The amount of accelerator card memory required for graph neural network training; for The amount of accelerator card memory required for graph structure data; The memory size that an accelerator card can use for graph neural network training tasks;

[0016] Step 2.4: The graph structure data is divided into subgraphs, and estimate completion Estimated execution time ;Will Add to task list ;

[0017] Step 2.5: Repeat steps 2.2 to 2.4 until and .

[0018] Furthermore, the specific process of step 3 is as follows:

[0019] Step 3.1: Assume that the genetic algorithm uses individuals, among which Individuals are recorded as , each individual represents a task scheduling scheme, and each individual contains Components, which specifically refer to accelerator cards in task scheduling schemes; for No. Components, expressed in In the task scheduling scheme of Run the first Tasks ;

[0020] Step 3.2: For each component of each individual, randomly select a value between 0 and An integer in which the counter Is the list of tasks that currently need to be scheduled The number of tasks in ;

[0021] Step 3.3: For Individual , check whether the following constraints are met:

[0022] (2);

[0023] in, is a statistical function; For the Tasks The number of accelerator cards required; for No. Quantity, Different from ; It means "implies", that is, "When the condition before is true, " The expression after " is also true; , Respectively represent , The accelerator card starts executing the task time;

[0024] Step 3.4: Repeat step 3.3 to check whether all individuals satisfy formula (2). For individuals that do not satisfy formula (2), randomly generate a new individual until there are individuals that satisfy formula (2). So far, these individuals that satisfy formula (2) are added to the set middle;

[0025] Step 3.5: For Individual , calculate its fitness value:

[0026] (3);

[0027] in, For individuals The fitness value of To adopt When scheduling tasks using the scheme, The total service time of an accelerator card is:

[0028] (4);

[0029] in, It is The total service time of each accelerator card before this scheduling; yes No. Components, expressed in In the task scheduling scheme of Run the first Tasks ; Completed Tasks Repeat step 3.5 to calculate the estimated execution time of the set The fitness of all individuals in ;

[0030] Step 3.6: Set The individual with the largest fitness value is assigned to the best individual , whose fitness value is assigned to the best fitness ;

[0031] Step 3.7, use Represents a new set of individuals and takes ;

[0032] Step 3.8, any Two individuals and , , Number different individuals. , from 1 to Any A non-repeating number , , for ; then swap and No. components, generating two new individuals and ,examine and Whether formula (2) is satisfied, add the individuals satisfying formula (2) to the new individual set Repeat step 3.8 until the number of newly generated individuals reaches So far, among which is a predefined constant;

[0033] Step 3.9, any An individual , Number the individual; from 1 to Any A non-repeating number , , for Then, No. The components range from 0 to Generate a new individual using a random number between ,examine Whether formula (2) is satisfied, add the individuals satisfying formula (2) to the new individual set Repeat step 3.9 until the number of newly generated individuals reaches So far, among which is a predefined constant;

[0034] Step 3.10: According to formula (3), calculate The fitness value of the individuals generated in steps 3.8 and 3.9;

[0035] Step 3.11, clear the collection ;Will The individuals in the array are sorted in descending order according to their fitness values, and then the first Put individuals into a collection ;

[0036] Step 3.12, if the set The fitness value of the first individual is greater than , then assign the individual to , assign its fitness value to ;

[0037] Step 3.13, repeat steps 3.7 to 3.12 until No longer changes; then according to the individual Perform task scheduling.

[0038] The beneficial technical effects brought by the present invention are as follows: the present invention collects graph neural network training tasks submitted by users in each cycle through a fixed task scheduling cycle. When performing task scheduling, the goal is to improve the utilization rate of the system's accelerator card resources. Each individual in the algorithm represents a feasible scheduling scheme. Through crossover, mutation and selection, it continuously iterates to seek the optimal solution, and performs task scheduling with the scheme corresponding to the optimal solution. Using this method, the utilization rate of accelerator card resources can be effectively improved, resource utilization can be maximized, and energy consumption can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a structural diagram of the distributed graph neural network training system of the present invention.

[0040] Figure 2 This is a flow chart of the batch processing scheduling method for graph neural network training tasks with data distribution of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0042] The present invention refers to the graph neural network training task as task, the system that executes the graph neural network training task as a distributed system, and an independently running computing unit in the distributed system as a computing node. This method assumes that all computing nodes in the distributed system are homogeneous, that is, the software and hardware configurations of all computing nodes are exactly the same; each computing node contains at least one dedicated accelerator card that can be used for graph neural network training, referred to as an accelerator card.

[0043] like Figure 1 As shown, the distributed system consists of one master node and several computing nodes. The master node is responsible for receiving and distributing the graph neural network training tasks submitted by the user, and each computing node contains the same number of accelerator cards. The task scheduling algorithm of the present invention takes the accelerator card as the smallest unit, that is, the accelerator card that executes a specific task has a complete graph neural network model and a part of the training data.

[0044] like Figure 2 As shown, a data-distributed graph neural network training task batch scheduling method includes the following steps:

[0045] Step 1: Initialize various parameters before scheduling begins; specifically:

[0046] Initialize the task list Empty set; set the task scheduling time interval , that is, every Perform a task scheduling; use Indicates the number of accelerator cards in the distributed system, using Indicates The time when the accelerator card starts to execute the task is Indicates The total service time of the accelerator card before the current scheduling; Indicates the current time, and its value increases automatically as time goes by; initially, , , ;

[0047] Step 2: Record the submitted task, obtain the task attributes, and add the submitted task to the task list. The specific steps are as follows:

[0048] Step 2.1, initialize the counter , used to record the number of tasks in the task list; determine the current time Whether the pre-set conditions are met, if they are met or , record the submitted task and obtain the task attributes; Express request Divide by The remainder of

[0049] Step 2.2: After receiving the submitted task, Increase by 1 and submit the The task is recorded as ;

[0050] Step 2.3, obtain Number of accelerator cards required , the calculation method is to solve the following minimization problem:

[0051] (1);

[0052] in, for The amount of accelerator card memory required for graph neural network training; for The amount of accelerator card memory required for graph structure data; The memory size that an accelerator card can use for graph neural network training tasks.

[0053] Step 2.4: The graph structure data is divided into subgraphs, and estimate completion Estimated execution time .Will Add to task list ;

[0054] Step 2.5: Repeat steps 2.2 to 2.4 until and .

[0055] Step 3: Use genetic algorithm to schedule tasks; the specific steps are as follows:

[0056] Step 3.1: Assume that the genetic algorithm uses individuals, among which Individuals are recorded as , each individual represents a task scheduling scheme, and each individual contains Components, which specifically refer to accelerator cards in task scheduling schemes; for No. Components, expressed in In the task scheduling scheme of Run the first Tasks ;

[0057] For example, suppose There are 6 components, that is, there are 6 accelerator cards; in a distributed system composed of 6 accelerator cards, Individual The meanings are as follows:

[0058] ;

[0059] Step 3.2: For each component of each individual, randomly select a value between 0 and An integer in which the counter Is the list of tasks that currently need to be scheduled The number of tasks in ;

[0060] Step 3.3: For Individual , check whether the following constraints are met:

[0061] (2);

[0062] in, is a statistical function; For the Tasks The number of accelerator cards required; For statistical Individual The median value is The number of components, It's a task Serial number; for No. Quantity, Different from ; It means "implies", that is, "When the condition before is true, " The expression after " is also true; , Respectively represent , The accelerator card starts executing the task time;

[0063] In formula (2), Limited to tasks The number of allocated accelerator cards is equal to ; Limit all accelerator cards assigned to the same task to execute the task The time is the same;

[0064] Step 3.4: Repeat step 3.3 to check whether all individuals satisfy formula (2). For individuals that do not satisfy formula (2), randomly generate a new individual until there are individuals that satisfy formula (2). So far, these individuals that satisfy formula (2) are added to the set middle;

[0065] Step 3.5: For Individual , calculate its fitness value:

[0066] (3);

[0067] in, For individuals The fitness value of To adopt When scheduling tasks using the scheme, The total service time of an accelerator card is:

[0068] (4);

[0069] in, It is The total service time of each accelerator card before this scheduling; yes No. Components, expressed in In the task scheduling scheme of Run the first Tasks ; Completed Tasks Repeat step 3.5 to calculate the estimated execution time of the set The fitness of all individuals in ;

[0070] Step 3.6: Set The individual with the largest fitness value is assigned to the best individual , whose fitness value is assigned to the best fitness ;

[0071] Step 3.7, use Represents a new set of individuals and takes ;

[0072] Step 3.8, any Two individuals and , , Number different individuals. , from 1 to Any A non-repeating number , , for ; then swap and No. components, generating two new individuals and ,examine and Whether formula (2) is satisfied, add the individuals satisfying formula (2) to the new individual set Repeat step 3.8 until the number of newly generated individuals reaches So far, among which is a predefined constant;

[0073] Step 3.9, any An individual , Number the individual; from 1 to Any A non-repeating number , , for Then, No. The components range from 0 to Generate a new individual using a random number between ,examine Whether formula (2) is satisfied, add the individuals satisfying formula (2) to the new individual set Repeat step 3.9 until the number of newly generated individuals reaches So far, among which is a predefined constant;

[0074] Step 3.10: According to formula (3), calculate The fitness value of the individuals generated in steps 3.8 and 3.9;

[0075] Step 3.11, clear the collection ;Will The individuals in the array are sorted in descending order according to their fitness values, and then the first Put individuals into a collection ;

[0076] Step 3.12, if the set The fitness value of the first individual is greater than , then assign the individual to , assign its fitness value to ;

[0077] Step 3.13, repeat steps 3.7 to 3.12 until No longer changes; then according to the individual Perform task scheduling.

[0078] In order to prove the feasibility and superiority of the present invention, the following comparative experiment is given.

[0079] The method proposed in the present invention is compared with the commonly used First Come First Serve (FCFS), Shortest Job First (SJF), and Round Robin (RR) scheduling algorithms.

[0080] Table 1 Comparison of 100 graph neural network training tasks

[0081] .

[0082] Table 2 Comparison of 500 graph neural network training tasks

[0083] .

[0084] Table 3 Comparison of 1000 graph neural network training tasks

[0085] .

[0086] Table 4 Comparison of 1500 graph neural network training tasks

[0087] .

[0088] The experimental comparison results are shown in Tables 1 to 5. The experiment randomly generates 100, 500, 1000, 1500, and 2000 graph neural network training tasks, and sets the number of accelerator cards in the distributed system to 20, 40, 60, 80, and 100.

[0089] Table 5 Comparison of 2000 graph neural network training tasks

[0090] .

[0091] The experimental results in Tables 1 to 5 show that the resource utilization rate of the scheduling algorithm proposed in the present invention is better than other scheduling algorithms in about 90% of the cases, and the advantage becomes more obvious as the number of graph neural network training tasks increases. When the number of graph neural network training tasks is small, the FCFS scheduling algorithm is better than other algorithms, but as the number of graph neural network tasks increases, the limitations of the FCFS scheduling algorithm become more prominent. When the number of tasks increases, the scheduling algorithm of the present invention and the SJF scheduling algorithm will show better performance than the FCFS scheduling algorithm and the RR scheduling algorithm.

[0092] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A data-distributed graph neural network training task batch scheduling method, characterized in that: The graph neural network training task is referred to as a task, the system that executes the graph neural network training task is called a distributed system, and an independently running computing unit in the distributed system is called a computing node. At the same time, it is assumed that all computing nodes in the distributed system are homogeneous, and each computing node contains at least one accelerator card for graph neural network training. The batch scheduling method specifically includes the following steps: Step 1: Initialize various parameters before scheduling begins; Step 2: Record the submitted task and obtain the task attributes, and add the submitted task to the task list; Step 3: Use genetic algorithm to schedule tasks; The specific process of step 3 is as follows: Step 3.1: Assume that the genetic algorithm uses m individuals in total, and the pth individual is denoted as X p =(x p,1 ,x p,2 ,…,x p,q ,…,x p,N ), each individual represents a task scheduling scheme, each individual contains N components, and the component specifically refers to the accelerator card in the task scheduling scheme; x p,q For x p The qth component of p In the task scheduling scheme, the xth task is run on the qth accelerator card. p,q Tasks Step 3.2: For each component of each individual, a random value is taken as an integer between 0 and k, where the counter k is the number of tasks in the task list TL that currently needs to be scheduled; Step 3.3: For the pth individual X p , check whether the following constraints are met: Where count(·) is a statistical function; Acc l For the lth task T l The number of accelerator cards required; For X p No. Quantity, Different from q; → means "implies", that is, when the condition before "→" is true, the expression after "→" is also true; AT q , Respectively represent the qth, The accelerator card starts executing the task time; Step 3.4, repeat step 3.3 to check whether all individuals satisfy formula (2); for individuals that do not satisfy formula (2), randomly generate a new individual until there are m individuals that satisfy formula (2), and add these individuals that satisfy formula (2) to the set XS; Step 3.5: For the pth individual X p , calculate its fitness value: Among them, Fit p For individual X p The fitness value of S p,c To adopt X p When the task is scheduled according to the scheme, the total service time of the cth accelerator card is: Among them, ST c is the total service time of the cth accelerator card before this scheduling; x p,c Yes X p The cth component of p In the task scheduling scheme, the xth task is run on the cth accelerator card. p,c Tasks Completed the xth p,c Tasks Repeat step 3.5 to calculate the fitness of all individuals in the set XS; Step 3.6: Assign the individual with the largest fitness value in the set XS to the best individual X best , and its fitness value is assigned to the best fitness F best ; Step 3.7, use XS′ to represent the new individual set, and take XS′=XS; Step 3.8, select any two individuals X in XS j and j. Number different individuals. Randomly select n1 non-repeating numbers from 1 to N for Then swap X j and No. components, generate two new individuals X j 'and Check X j 'and Whether formula (2) is satisfied, add the individuals satisfying formula (2) to the new individual set XS′; repeat step 3.8 until the number of newly generated individuals reaches (α×m), where 0<α<0.5 is a predefined constant; Step 3.9: Pick any individual from XS Number the individuals; select n2 non-repeating numbers from 1 to N for Then, No. The component takes a random number between 0 and k and generates a new individual examine Whether formula (2) is satisfied, add the individuals satisfying formula (2) to the new individual set XS′; repeat step 3.9 until the number of newly generated individuals reaches (β×m), where 0<β<0.5 is a predefined constant; Step 3.10: Calculate the fitness value of the individuals generated in steps 3.8 and 3.9 in XS′ according to formula (3); Step 3.11, clear the set XS; sort the individuals in XS′ in descending order of fitness value, and then put the first m individuals into the set XS; Step 3.12: If the fitness value of the first individual in the set XS is greater than F best , then assign the individual to X best , assign its fitness value to F best ; Step 3.13, repeat steps 3.7 to 3.12 until F best No longer changes; then according to individual X best Perform task scheduling.

2. The method for batch scheduling of graph neural network training tasks based on data distribution according to claim 1 is characterized in that: The specific process of step 1 is as follows: initialize the task list TL to be an empty set; set the task scheduling time interval Cyc, that is, perform task scheduling once every Cyc; use N to represent the number of accelerator cards in the distributed system, and use AT i Indicates the time when the i-th accelerator card starts executing the task, using ST i represents the total service time of the ith accelerator card before this scheduling; Indicates the current time. Its value increases automatically as time goes by. Initially, AT i =0, ST i =0, 3. The method for batch scheduling of graph neural network training tasks based on data distribution according to claim 2 is characterized in that: The specific process of step 2 is: Step 2.1, initialize counter k = 0 to record the number of tasks in the task list; determine the current time Whether the pre-set conditions are met, if they are met or , record the submitted task and obtain the task attributes; Express request The remainder when divided by Cyc; Step 2.2: After receiving the submitted task, increase k by 1 and record the kth submitted task as T k ; Step 2.3: Get T k Number of accelerator cards required Acc k , the calculation method is to solve the following minimization problem: min Acc k s.t. Acc k >1 Among them, NAM k T k The amount of memory required by the accelerator card for training the graph neural network; GAM k T k The memory size of the accelerator card required for the graph structure data; AM is the memory size that an accelerator card can use for graph neural network training tasks; Step 2.4: T k The graph structure data is divided into Acc k subgraphs, and estimate the completion of T k Estimated execution time ET k ; T k Add to task list TL; Step 2.5: Repeat steps 2.2 to 2.4 until and

Images