Scheduling method and server

Abstract

The invention relates to the technical field of data processing, in particular to a scheduling method and a server. In the embodiment of the invention, the resource utilization rate score of each candidate server node is obtained by comprehensively considering the GPU utilization rate, the CPU utilization rate, the video memory utilization rate, the memory utilization rate, the network bandwidth and the disk IO of different servers in different clusters. The electronic equipment screens each candidate server node by adopting a preset threshold value, and distributes each reasoning task to each screened server node by adopting a preset server node scheduling strategy, so that reasoning task distribution is realized while the server resource utilization rate is improved, and the reasoning efficiency is improved. And the most reasonable server node scheduling scheme can be quickly found out.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a scheduling method and a server. Background Art

[0002] In recent years, with the development of artificial intelligence technology, the corresponding server nodes are also developing continuously. However, the current artificial intelligence server nodes are mostly small models. With the rapid development of large models, server nodes with only small models can no longer meet today's reasoning needs.

[0003] Moreover, in the related technologies, when scheduling server nodes, most of them only consider the GPU and video memory utilization, which is a one-sided consideration. As a result, when large model server nodes complete large model inference tasks, they do not consider the complex changes under multiple constraints, which has great limitations. Summary of the invention

[0004] The present application provides a scheduling method and a server to solve the problem that most of the existing technologies are small model server nodes, one-sided factors are considered when scheduling server nodes, and there are great limitations for large model reasoning tasks.

[0005] In a first aspect, an embodiment of the present application provides a scheduling method, the method comprising:

[0006] Determine the resource utilization score of each candidate server node according to the saved GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each candidate server node; wherein each candidate server node includes a small model server node, a large model server node and a large and small model mixed server node;

[0007] Filtering each server node for scheduling from each candidate server node according to the resource utilization score of each candidate server node and a preset threshold;

[0008] According to a preset server node scheduling strategy, each inference task to be processed is distributed to each server node.

[0009] In a second aspect, an embodiment of the present application further provides a server, the server comprising at least a processor and a memory, the processor being configured to implement the steps of the scheduling method described above when executing a computer program stored in the memory.

[0010] In the embodiment of the present application, the dispatch center obtains the resource utilization score of each candidate server node by comprehensively considering the GPU utilization, CPU utilization, video memory utilization, memory utilization, network bandwidth, and disk IO of different servers in different clusters. The electronic device uses a preset threshold to screen each candidate server node, and uses a preset server node scheduling strategy to distribute each reasoning task to each screened server node, while improving the server resource utilization, realizing reasoning task allocation, and quickly finding the most reasonable server node scheduling solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A schematic diagram of a scheduling process provided in an embodiment of the present application;

[0013] Figure 2 A system architecture diagram provided for an embodiment of the present application;

[0014] Figure 3 A schematic diagram of the interface of the reasoning platform provided in the embodiment of the present application;

[0015] Figure 4a A line graph showing the change in the inertia factor corresponding to formula a provided in an embodiment of the present application as a function of the number of iterations;

[0016] Figure 4b A line graph showing the change in the inertia factor corresponding to formula b provided in an embodiment of the present application as a function of the number of iterations;

[0017] Figure 5 A schematic diagram of the scheduling process provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of the structure of a scheduling device provided in an embodiment of the present application;

[0019] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0021] In order to realize reasoning task allocation, quickly find the most reasonable server node scheduling plan, and improve server node resource utilization, the embodiment of the present application provides a scheduling method and a server.

[0022] In an embodiment of the present application, the electronic device determines the resource utilization score of each candidate server node based on the GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each saved candidate server node; wherein each candidate server node includes a small model server node, a large model server node and a large and small model mixed server node; based on the resource utilization score of each candidate server node and a preset threshold, each server node for scheduling is screened out from each candidate server node; according to the preset server node scheduling strategy, each inference task to be processed is distributed to each server node.

[0023] Figure 1 A schematic diagram of a scheduling process provided in an embodiment of the present application, the process includes:

[0024] S101: Determine the resource utilization score of each candidate server node according to the saved GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each candidate server node; wherein each candidate server node includes a small model server node, a large model server node and a large and small model mixed server node.

[0025] A scheduling method provided in an embodiment of the present application is applied to an electronic device, which may be a PC or a server, and may also be a scheduling center server.

[0026] In the embodiment of the present application, the electronic device stores the GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each server node. In each server node, at least one of a large model and a small model is deployed, that is, the server node in the embodiment of the present application can be a small model server node that only deploys a small model, a large model server node that only deploys a large model, or a large and small model mixed server node that deploys both a small model and a large model.

[0027] In an embodiment of the present application, server nodes of the same type can be deployed in the same cluster. Figure 2 The system architecture diagram provided in the embodiment of the present application is as follows: Figure 2 As shown, the scheduling center can schedule any server node in the small model server node cluster, the large model server node cluster, and the large and small model mixed server node cluster. In addition, each server node in the small model server node cluster is deployed with at least one small model, each server node in the large model server node cluster is deployed with at least one large model, and each server node in the large and small model mixed server node cluster is deployed with at least one large model and one small model.

[0028] Compared with the prior art that only schedules based on GPU and video memory utilization, which considers one-sided factors, in the embodiments of the present application, the resource utilization of each server node is scored according to the saved GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each candidate server node, and the resource utilization score of each candidate server node is determined. The factors are considered more comprehensively, making the scheduling more accurate.

[0029] S102: Filter each server node for scheduling from each candidate server node according to the resource utilization score of each candidate server node and a preset threshold.

[0030] In an embodiment of the present application, the electronic device is also pre-configured with a preset threshold for screening based on the resource utilization score of each candidate server node.

[0031] Specifically, the electronic device determines each candidate server node whose corresponding resource utilization score is not lower than a preset threshold, and determines each candidate server node as each server node for scheduling.

[0032] Generally, the resource utilization score ranges from [0,100], where 100 represents the best and 0 represents the worst. The highest-scoring node is not unique. The preset threshold configured in the electronic device is 60 points. The electronic device considers that a resource utilization score less than 60 points indicates that the reasoning task cannot be completed normally, and removes candidate server nodes with a resource utilization score less than 60 points.

[0033] In addition, in the embodiment of the present application, before determining the resource utilization score of each candidate server node, the electronic device will first perform a pre-selection to filter out each candidate server node from all registered server nodes.

[0034] Specifically, the electronic device traverses all registered server nodes and filters out registered server nodes that do not meet the requirements. For example, registered server nodes with too high resource occupancy rates; registered server nodes that are down. In addition, the electronic device can also pre-select according to the model type corresponding to the reasoning task to be processed. For example, if the reasoning tasks to be processed are all small model reasoning tasks, the registered server nodes that are not deployed with small models will be filtered out; if the reasoning tasks to be processed are all small model reasoning tasks, the registered server nodes that are not deployed with large models will be filtered out, etc.

[0035] S103: Distribute each inference task to be processed to each server node according to a preset server node scheduling strategy.

[0036] In an embodiment of the present application, a server node scheduling strategy is pre-configured in the electronic device, and the electronic device can distribute each inference task to be processed to each server node for processing according to the server node scheduling strategy. When distributing tasks, one inference task can only be distributed to one server node for processing, but one server node can process one or more inference tasks.

[0037] In a possible implementation, the inference tasks to be processed include large model inference tasks and small model inference tasks. During scheduling, large model inference tasks can only be distributed to server nodes deployed with large models, and small model inference tasks can only be distributed to server nodes deployed with small models.

[0038] In the embodiment of the present application, scheduling includes but is not limited to two models: large model reasoning task priority and small model reasoning task priority. Taking large model reasoning task priority as an example, if there are currently z reasoning tasks to be processed, including z1 large model reasoning tasks and z2 small model reasoning tasks, and the server nodes determined by the electronic device for scheduling include x large model server nodes, y small model server nodes, and k large and small model mixed server nodes, the electronic device can be scheduled in the following manner:

[0039] Step 1: According to the priority of z1 large model inference tasks, first distribute them to x large model server nodes;

[0040] a. If the distribution is complete, proceed to step 2;

[0041] b. If the distribution is not complete, continue to distribute to k large and small model mixed server nodes. If the distribution is complete, go to step 2); if the distribution is not complete, enter the queue and wait for the next round of scheduling;

[0042] Step 2: According to the priority of z2 small model inference tasks, first distribute them to y small model server nodes;

[0043] a. If the distribution is completed, this round of scheduling ends;

[0044] b. If the distribution is not complete, check the k large and small model server nodes to see if there are any remaining nodes after the large model inference task. If there are any remaining nodes, continue to distribute them. If there are no remaining nodes, enter the queue and wait for the next round of scheduling.

[0045] In the embodiment of the present application, the dispatch center obtains the resource utilization score of each candidate server node by comprehensively considering the GPU utilization, CPU utilization, video memory utilization, memory utilization, network bandwidth, and disk IO of different servers in different clusters. The electronic device uses a preset threshold to screen each candidate server node, and uses a preset server node scheduling strategy to distribute each reasoning task to each screened server node, while improving the server resource utilization, realizing reasoning task allocation, and quickly finding the most reasonable server node scheduling solution.

[0046] In order to realize the reasoning task allocation, quickly find the most reasonable server node scheduling scheme, and improve the server node resource utilization, based on the above embodiment, in the embodiment of the present application, each reasoning task carries the model type of the model that executes the reasoning task;

[0047] The distributing the inference task to be processed to each server node according to the preset server node scheduling strategy includes:

[0048] Classifying each reasoning task according to the model type corresponding to each reasoning task to obtain a reasoning task group corresponding to each model type;

[0049] According to the model types of the idle models in the large-model mixed server nodes in each of the current server nodes and the model types deployed in the non-large-model mixed server nodes in each of the saved server nodes, each of the server nodes is classified to obtain a server node group corresponding to each model type;

[0050] Each inference task in each inference task group is distributed to a target server node in a server node group of the same model type.

[0051] In an embodiment of the present application, each inference task carries the model type of the model that executes the inference task, and the electronic device also stores the model type of the model deployed in each server node. When the electronic device distributes the inference task, it distributes the inference task to the server node whose deployed model type is consistent with the model type corresponding to the inference task. For example, the electronic device distributes the large model inference task to the server node deployed with the large model, and distributes the small model inference task to the server node deployed with the small model.

[0052] Specifically, the electronic device classifies each reasoning task according to the model type corresponding to each reasoning task, and obtains the reasoning task group corresponding to each model type. Generally, the electronic device constructs a large model reasoning task group and a small model reasoning task group. The electronic device also classifies each server node according to the model type deployed in each saved server node, and obtains the server node group corresponding to each model type. Generally, the electronic device constructs a large model server node group and a small model server node group.

[0053] It should be noted that in the embodiment of the present application, for a large-model and small-model hybrid server node that is deployed with both a large model and a small model, the electronic device can add the large-model and small-model hybrid server node to both the large model server node group and the small model server node group.

[0054] In addition, the electronic device can also count the model types of the currently idle models in the large-model mixed server node, and add the large-model mixed server node to the server node group corresponding to the model type. For example, if the electronic device determines that the large model in the large-model mixed server node is in an idle state, the electronic device adds the large-model mixed server node to the large model server node group; if the electronic device determines that the small model of the large-model mixed server node is in an idle state, the electronic device adds the large-model mixed server node to the small model server node group.

[0055] In the embodiment of the present application, the electronic device distributes each reasoning task in each reasoning task group to the target server node in the server node group of the same model type. That is, the electronic device distributes the reasoning tasks in the large model reasoning task group to the target server node in the large model server node group, and distributes the reasoning tasks in the small model reasoning task group to the target server node in the small model server node group.

[0056] The electronic device may distribute each reasoning task in each reasoning task group to a target server node in a server node group of the same model type in a random distribution manner.

[0057] In order to realize the reasoning task allocation, quickly find the most reasonable server node scheduling scheme, and improve the server node resource utilization, on the basis of the above embodiments, in the embodiment of the present application, each reasoning task also carries a task priority;

[0058] The distributing each reasoning task in each reasoning task group to a server node in a server node group of the same model type comprises:

[0059] According to the task priority corresponding to each reasoning task, the reasoning tasks included in each reasoning task group are sorted to obtain a reasoning task sequence corresponding to each model type;

[0060] According to the resource utilization score corresponding to each server node, the server nodes included in each server node group are sorted to obtain a server node sequence corresponding to each model type;

[0061] Each inference task in each inference task sequence is distributed in sequence to a target server node in a server node sequence of the same model type.

[0062] In an embodiment of the present application, each reasoning task also carries a task priority. When distributing reasoning tasks, the electronic device can also distribute the reasoning tasks with high task priority to the server nodes with high resource utilization scores based on the task priority corresponding to each reasoning task and the resource utilization score corresponding to each server node.

[0063] Specifically, the electronic device sorts the reasoning tasks included in each reasoning task group according to the task priority corresponding to each reasoning task, and obtains the reasoning task sequence corresponding to each model type; the electronic device sorts the server nodes included in each server node group according to the resource utilization score corresponding to each server node, and obtains the server node sequence corresponding to each model type. The electronic device distributes each reasoning task in each reasoning task sequence to the target server node in the server node sequence of the same model type in sequence.

[0064] For example, the large model reasoning task sequence determined by the electronic device is reasoning task 1, reasoning task 3 and reasoning task 2, and the large model server node sequence determined by the electronic device is server node 2, server node 3 and server node 1, then the electronic device distributes reasoning task 1 to server node 2, distributes reasoning task 3 to server node 3, and distributes reasoning task 2 to server node 1.

[0065] In addition, in an embodiment of the present application, the lengths of the reasoning task sequence and the server node sequence corresponding to each model type may be different, such as the length of the reasoning task sequence is greater than the length of the server node sequence, or the length of the reasoning task sequence is less than the length of the server node sequence.

[0066] Among them, if the length of the reasoning task sequence is greater than the length of the server node sequence, each reasoning task in the reasoning task sequence is sent to the target server node at the corresponding position in the server node sequence in turn. When each server node in the server node sequence is occupied, the electronic device suspends the sending of the reasoning task and waits for a server node to complete the reasoning task before sending it again, or the electronic device starts a new round of reasoning task distribution from the first one in the server node sequence. There is no restriction here.

[0067] In order to realize the reasoning task allocation, quickly find the most reasonable server node scheduling scheme, and improve the server node resource utilization, based on the above embodiments, in the embodiment of the present application, each reasoning task in each reasoning task group is distributed to each server node in the server node group of the same model type, including:

[0068] For each model type, determine the number of inference tasks included in the inference task group corresponding to the model type, and the number of server nodes included in the server node group corresponding to the model type; use a particle swarm algorithm to generate a configuration matrix based on the number of inference tasks and the number of server nodes, wherein the number of rows of the configuration matrix is ​​the number of inference tasks, and the number of columns of the configuration matrix is ​​the number of inference nodes; determine the elements in the configuration matrix whose corresponding values ​​are preset values, and determine the target inference tasks corresponding to the rows where the elements are located, and the target server nodes corresponding to the columns where the elements are located; distribute the target inference tasks to the target server nodes.

[0069] In the embodiment of the present application, the electronic device may also use a particle swarm algorithm to distribute the reasoning tasks. When initiating a scheduling task, the user may choose which strategy to use for reasoning task distribution on the reasoning platform.

[0070] Figure 3 The interface diagram of the reasoning platform provided in the embodiment of the present application is shown in FIG. Figure 3 As shown in the figure, users can choose scheduling strategies, including but not limited to: sliding average algorithm, instantaneous weighted algorithm, particle swarm algorithm and improved particle swarm algorithm. In addition, users can also configure corresponding weights for each resource to calculate the resource utilization score of each server node.

[0071] In addition, based on the scheduling platform, users can also click on the "Node Registration" page to register the IP, port, GPU, CPU, memory, network bandwidth, and disk IO of all server nodes, and then click on the "Policy Configuration" tab to select the scheduling strategy and configure the parameters.

[0072] Specifically, for each model type, the electronic device determines the number of inference tasks included in the inference task group corresponding to the model type, and determines the number of server nodes included in the server node group corresponding to the model type. The electronic device can generate a configuration matrix using a particle swarm algorithm based on the number of inference tasks and the number of server nodes. The number of rows in the configuration matrix is ​​the number of inference tasks, and the number of columns in the configuration matrix is ​​the number of inference nodes; determine the element whose corresponding value in the configuration matrix is ​​a preset value, and determine the target inference task corresponding to the row where the element is located, and the target server node corresponding to the column where the element is located; distribute the target inference task to the target server node.

[0073] Generally, in an embodiment of the present application, the value of each element in the configuration matrix is ​​0 or 1, and the preset value is 1. That is, if the value corresponding to any element in the configuration matrix is ​​1, the electronic device sends the target inference task corresponding to the row where the element is located to the target server node corresponding to the column where the element is located.

[0074] For example, the number of inference tasks is n, the number of server nodes is m, and the request set is Requests = {r 1 ,r 2 ,r 3 ,…,r n}, server node set Nodes = {n 1 ,n 2 ,n 3 ,…,n m}, select a server node in Nodes for reasoning for each reasoning task in Requests, and the generated configuration matrix can be:

[0075]

[0076] Among them, d ij The value is 0 or 1, 1 means sending the i-th inference task to the j-th server node for execution, and 0 means not sending it.

[0077] In the particle swarm algorithm, a configuration matrix D is a particle, representing a scheduling scheme.

[0078] In order to realize the reasoning task allocation, quickly find the most reasonable server node scheduling scheme, and improve the server node resource utilization, based on the above embodiments, in the embodiment of the present application, the particle swarm algorithm is used to generate a configuration matrix according to the number of reasoning tasks and the number of server nodes, including:

[0079] Construct a blank matrix with the number of rows equal to the number of the inference tasks and the number of columns equal to the number of the inference nodes;

[0080] According to a preset random function, assigning an initial position to each element in the blank matrix;

[0081] Assigning an initial speed to each of the elements according to the random function and a preconfigured maximum speed value and a preconfigured minimum speed value;

[0082] Perform multiple iterations until the preset end condition is met, where each iteration process includes:

[0083] According to the preset inertia factor, the individual learning factor, the social learning factor and the random function, the speed of each element obtained in the previous iteration is updated;

[0084] According to the updated speed, the position corresponding to each element obtained in the previous iteration is updated.

[0085] In the embodiment of the present application, the electronic device uses a particle swarm algorithm. When generating a configuration matrix according to the number of inference tasks and the number of server nodes, the electronic device constructs a blank matrix with the number of rows equal to the number of inference tasks and the number of columns equal to the number of inference nodes; assigns an initial position to each element in the blank matrix according to a preset random function; assigns an initial speed to each element according to the random function and the pre-configured maximum speed value and minimum speed value; and performs multiple iterations until a preset end condition is met. The electronic device determines the matrix obtained by the final iteration as the configuration matrix.

[0086] In the embodiment of the present application, the distribution of reasoning tasks is a discrete problem, which requires mapping of binary discrete particle swarms. The standard particle swarm algorithm uses the sigmoid function for mapping, and the mapping probability is obtained by mapping, and the probability of the current particle mutating to 1 in this dimension can be calculated. Specifically, it can be calculated by the following formula:

[0087]

[0088] Among them, s(v ij ) represents the probability that the element in row i and column j suddenly changes to 1, v ij It is the velocity corresponding to the element calculated by the standard particle swarm algorithm.

[0089] The update formula of the element's position is:

[0090]

[0091] Among them, random() is a random function with a value range of [0,1]

[0092] However, as the number of iterations increases, the randomness of the mutation using the above formula will gradually increase. When the maximum number of iterations is reached, the randomness of the mutation also reaches the maximum value, which makes the results that need to converge in the later stage more uncertain and it is not easy to converge to the global optimal value.

[0093] Therefore, the formula for determining the probability of element mutation is modified to:

[0094]

[0095] According to the modified formula, the velocity is centered at the origin, symmetrical on both sides, and is an even function. When the velocity is positive, the probability function increases; when the velocity is negative, the probability function decreases. Based on this, the update formula of the element position is modified to:

[0096] When v ij <0,

[0097]

[0098] When v ij >0,

[0099]

[0100] To avoid s(v ij ) is too close to the extreme value 1, which can be obtained by v ij ∈[-v max ,v max ] is restricted. Through the binary mapping relationship, the particle swarm algorithm is applied to the binary discrete multiple reasoning request distribution problem.

[0101] Based on this, in the embodiment of the present application, the electronic device can initialize the velocity of the particle by the following formula:

[0102] v ij =v min +random()×(v max -v min )

[0103] Among them, random() is the value of the random function, v max is the pre-configured maximum speed value, v min is the pre-configured minimum speed value, generally, v max =2,vmin =-2.

[0104] Electronic devices can initialize their positions using the following formula:

[0105]

[0106] Among them, random() is the value of the random function.

[0107] A particle calculates its velocity in a certain dimension through its speed, that is, the distance it moves in that dimension. When the data features of all dimensions are gathered together, the position change of a particle in the solution space is completed.

[0108] Based on this, after the position and velocity of the particles have been initialized, a preset number of iterations are performed according to the particle swarm algorithm. Each iteration process includes:

[0109] The electronic device updates the speed of each element obtained in the previous iteration according to the preset inertia factor, individual learning factor, social learning factor and random function; and updates the corresponding position of each element obtained in the previous iteration according to the updated speed.

[0110] Among them, the electronic device can use the following formula to update the speed:

[0111] v k+1 =ωv k +c 1 ×random1()×(Pbest k -x k )+c 2 ×random2()×(Gbest k -x k )

[0112] Where k∈[1,r] is the number of iterations, r is the total number of iterations, ω is the inertia factor, which determines the weight of the particle inheriting the previous moving trajectory, and its value is generally 0.9; c 1 is the individual learning factor, indicating the influence of its own historical optimal value on the current movement; c 2 is the social learning factor, indicating the attraction of the optimal particle in the group to the current particle, c 1 and c 2 The value of Pbest is generally 2; k It is the best historical position at present; Gbest k is the optimal position of the current group; x kis the position corresponding to the element in the previous iteration; random1() and random2() are two random functions with a value range of [0,1], which are used to enhance the randomness of particle optimization and prevent particles from converging too early, reducing the global optimization ability, and falling into local optimality.

[0113] The electronic device can use the following formula to update the value corresponding to each element:

[0114] x k+1 =x k +v k+1

[0115] Among them, x k is the position corresponding to the element in the previous iteration, v k+1 The updated speed.

[0116] In order to realize the reasoning task allocation, quickly find the most reasonable server node scheduling scheme, and improve the server node resource utilization, based on the above embodiments, in the embodiment of the present application, the process of determining the inertia factor includes:

[0117] Get the pre-configured maximum and minimum inertia factors;

[0118] determining a first difference between the maximum inertia factor and the minimum inertia factor;

[0119] Determine a first ratio of the square value of the number of iterations to the square value of the preset total number of iterations;

[0120] A first product of the first difference and the first ratio is determined, and a second difference between the maximum inertia factor and the first product is determined as the inertia factor.

[0121] In actual application, the larger the inertia factor, the stronger the global search capability of the current particle swarm, and the smaller the inertia factor, the stronger the local search capability of the current particle swarm. In order to ensure that the particle swarm has good global search capability in the early stage of iteration and strong local search capability in the later stage, in a possible implementation, the inertia factor can be updated using the following formula a:

[0122]

[0123] Among them, ω k+1 is the inertia factor corresponding to the current iteration, ω max is the preset maximum inertia factor, usually set to 0.9, ω min is the preset minimum inertia factor, usually 0.4, k is the number of iterations, and r is the total number of iterations.

[0124] Although the inertia factor has a dynamic change feature based on the above formula a, it still has the disadvantage of falling into the local optimal solution due to the linear transformation. Therefore, the inertia factor is slowly reduced in the early iteration and quickly reduced in the later iteration through nonlinear transformation, thereby enhancing the early global optimization ability and the later local convergence ability. With the help of parabola, a more reasonable nonlinear transformation from the maximum inertia factor to the minimum inertia factor is obtained. Therefore, the above formula a is improved to obtain the following formula b:

[0125]

[0126] Figure 4a A line graph showing the change in the inertia factor corresponding to formula a provided in the embodiment of the present application as a function of the number of iterations, Figure 4b The line graph of the inertia factor corresponding to the formula b provided in the embodiment of the present application as a function of the number of iterations. Figure 4a and Figure 4b It can be seen that although the inertia factor in both formulas is reduced from 0.9 to 0.4, the inertia factor in formula a is a linear transformation, which easily causes the particle swarm algorithm to fall into a local optimal solution. The inertia factor in formula b decreases slowly in the early iterations and decreases rapidly in the later iterations, thereby enhancing the early global optimization ability and the later local convergence ability.

[0127] In order to realize the reasoning task allocation, quickly find the most reasonable server node scheduling scheme, and improve the server node resource utilization, based on the above embodiments, in the embodiment of the present application, the determination process of the individual learning factor and the social learning factor includes:

[0128] Determine a first ratio of the square value of the number of iterations to the square value of a preset total number of iterations, and determine a third difference between the preset value and the ratio;

[0129] Determine a second product of a preset target learning factor corresponding to the current iteration and the third difference as the individual learning factor;

[0130] A third product of the target learning factor and the first ratio is determined as the social learning factor.

[0131] In an embodiment of the present application, individual learning factors and social learning factors can also be optimized so that in the early iterations, the individual learning factors account for a larger proportion, thereby enhancing the global optimization capability. In the later iterations, the proportion of social learning factors is increased to ensure rapid convergence after the global optimal range is determined.

[0132] Specifically, the electronic device is pre-configured with a target learning factor for updating the individual learning factor and the social learning factor. The electronic device determines the ratio of the square value of the number of iterations to the square value of the preset total number of iterations, and determines the third difference between the preset value and the first ratio; the electronic device determines the second product of the target learning factor corresponding to the preset current iteration and the third difference as the individual learning factor; and determines the third product of the target learning factor and the first ratio as the social learning factor.

[0133] In a possible implementation, the electronic device determines the individual learning factor by the following formula:

[0134]

[0135] Among them, c 1 is the individual learning factor, c is the target learning factor, k is the number of iterations, and r is the total number of iterations.

[0136] The electronic device determines the social learning factor using the following formula:

[0137]

[0138] Among them, c 2 is the social learning factor, c is the target learning factor, k is the number of iterations, and r is the total number of iterations.

[0139] In order to realize the reasoning task allocation, quickly find the most reasonable server node scheduling scheme, and improve the server node resource utilization, based on the above embodiments, in the embodiment of the present application, the process of determining the target learning factor includes:

[0140] According to the preset objective function, determine the target fitness corresponding to the current iteration;

[0141] Obtain the maximum fitness and average fitness determined before the current iteration, and obtain the preset maximum learning factor, minimum learning factor and curve smoothing control factor;

[0142] If the target fitness is less than the average fitness, determining the maximum fitness as the target learning factor;

[0143] If the target fitness is not less than the average fitness, determine a fourth difference between the target fitness and the average fitness, and a fifth difference between the maximum fitness and the average fitness; determine a second ratio of the fourth difference to the fifth difference, and determine a fourth product of the second ratio and the curve smoothing control factor; perform a tangent operation on the fourth product to obtain a tangent value; determine a sixth difference between the maximum learning factor and the minimum learning factor; determine a fifth product of the sixth difference and the tangent value; and determine the sum value of the fifth product and the minimum learning factor as the target learning factor.

[0144] In an embodiment of the present application, the scheduling algorithm takes into account the GPU, CPU, memory, network bandwidth, and disk I / O. Before scheduling, the electronic device needs to obtain the total amount of resources {C gpu , C cpu , C mem , C net , C disk} of each server node, and the resource request values of each inference task are {R gpu , R cpu , R mem , R net , R disk}. The electronic device determines the target fitness of the current iteration through the objective function, so as to screen and optimize and compare all solutions.

[0145] In a possible implementation manner, the objective function is:

[0146]

[0147] where U represents the resource utilization rate, j is the server node number, and U avg represents the average resource utilization rate of the server node, and Each inference request can be sent to at most one server node for running at the same time, so Secondly, the total amount of resources required for each inference request shall not exceed the total amount of resources owned by the current server node, that is, ∑d ij ×R < C, where i is the number of the inference task deployed to the jth server node.

[0148] Specifically, in an embodiment of the present application, the electronic device determines the target fitness corresponding to the current iteration according to a preset objective function. The electronic device obtains the maximum fitness and average fitness determined before the current iteration, and obtains the preset maximum learning factor, minimum learning factor and curve smoothing control factor. If the electronic device determines that the target fitness is less than the average fitness, the maximum fitness is determined as the target learning factor; if the electronic device determines that the target fitness is not less than the average fitness, the target learning factor is determined based on the target fitness, maximum fitness, average fitness, maximum learning factor, minimum learning factor and curve smoothing control factor.

[0149] In one possible implementation, the electronic device determines a fourth difference between the target fitness and the average fitness, and a fifth difference between the maximum fitness and the average fitness; determines a second ratio of the fourth difference to the fifth difference, and determines a fourth product of the second ratio and the curve smoothing control factor; performs a tangent operation on the fourth product to obtain a tangent value; determines a sixth difference between the maximum learning factor and the minimum learning factor; determines a fifth product of the sixth difference and the tangent value; and determines the sum of the fifth product and the minimum learning factor as the target learning factor.

[0150] Among them, the electronic device can determine the target learning factor through the following formula:

[0151]

[0152] Where c is the target learning factor; c min is the minimum learning factor, which is generally set to 1; c max is the maximum learning factor, which is generally 3; F is the target fitness; F avg is the average fitness; F max is the maximum fitness; θ is the curve smoothing control factor, which is generally taken as 3.

[0153] In summary, in the embodiment of the present application, the improved particle swarm algorithm solves the problem that the particle swarm algorithm is prone to fall into local optimality by optimizing the inertia factor, individual learning factor, and social learning factor, thereby completing balanced scheduling and obtaining the optimal resource allocation plan. Electronic devices can distribute reasoning tasks based on the improved particle swarm algorithm, with better global search capabilities in the early stage and faster local convergence speed in the later stage, and can quickly find the most reasonable server node resource scheduling plan to improve server node resource utilization.

[0154] In order to realize the inference task allocation, quickly find the most reasonable server node scheduling scheme, and improve the resource utilization of server nodes, based on the above embodiments, in the embodiment of the present application, the resource utilization score of each server node is determined according to the saved GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each server node, including:

[0155] Get the pre-configured GPU weight, CPU weight, memory weight, network bandwidth weight, and disk IO weight;

[0156] According to the GPU weight, CPU weight, memory weight, network bandwidth weight and disk IO weight, a weighted sum is performed on the GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each server node to determine a resource utilization score for each server node.

[0157] In an embodiment of the present application, the electronic device is pre-configured with GPU weight, CPU weight, memory weight, network bandwidth weight and disk IO weight. The electronic device can score each server node according to the GPU weight, CPU weight, memory weight, network bandwidth weight and disk IO weight.

[0158] Specifically, the electronic device performs weighted summation of the GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each server node according to the GPU weight, CPU weight, memory weight, network bandwidth weight and disk IO weight to determine the resource utilization score of each server node.

[0159] In a possible implementation, the electronic device may determine the resource utilization score corresponding to each server node based on the following formula:

[0160] Score = a 1 U gpu +a 2 U cpu +a 3 U mem +a 4 U net +a 5 U disk

[0161] Among them, Score is the resource utilization score, a 1 is the GPU weight, a 2 is the CPU weight, a 3 is the memory weight, a 4 is the network bandwidth weight, a 5 is the disk IO weight, Ugpu Indicates GPU utilization, U cpu Indicates CPU utilization, U mem Indicates memory utilization, U net Indicates the network bandwidth utilization, U disk Indicates disk IO utilization.

[0162] Figure 5 The scheduling process diagram provided in the embodiment of the present application is as follows: Figure 5 As shown, the process includes: the dispatch center receives batch reasoning tasks, and performs pre-selection, optimization, statistics and dispatching decisions.

[0163] The pre-selection includes: traversing all server nodes and filtering out server nodes that do not meet the requirements, such as server nodes with excessively high resource usage; server nodes that are down; if all are small model reasoning tasks, filter out large model server nodes; if all are large model reasoning tasks, filter out small model server nodes, etc.

[0164] The optimization includes: scoring the candidate server nodes selected through the pre-selection stage, with the scoring range being [0, 100], 100 representing the best, 0 representing the worst, and the highest-scoring node is not unique. Nodes with a score of less than 60 are eliminated, which means that the reasoning task cannot be completed normally.

[0165] The statistics include: counting the number of currently idle model types in each filtered server node.

[0166] Decision scheduling includes: distributing each inference task to be processed to each selected server node according to a preset server node scheduling strategy.

[0167] Based on the above embodiments, Figure 6 A schematic diagram of a scheduling device structure provided in an embodiment of the present application, the device includes:

[0168] Processing module 601 is used to determine the resource utilization score of each candidate server node according to the GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each candidate server node saved; wherein each candidate server node includes a small model server node, a large model server node and a large and small model mixed server node; according to the resource utilization score of each candidate server node and a preset threshold, each server node for scheduling is screened out from each candidate server node;

[0169] The scheduling module 602 is used to distribute each inference task to be processed to each server node according to a preset server node scheduling strategy.

[0170] In a possible implementation manner, each reasoning task carries a model type of a model that executes the reasoning task;

[0171] The scheduling module 602 is specifically used to classify each reasoning task according to the model type corresponding to each reasoning task, and obtain the reasoning task group corresponding to each model type; classify each server node according to the model type of the idle model in the large and small model mixed server node in each server node, and the saved model type deployed in the non-large and small model mixed server node in each server node, and obtain the server node group corresponding to each model type; distribute each reasoning task in each reasoning task group to the target server node in the server node group of the same model type.

[0172] In a possible implementation manner, each reasoning task also carries a task priority;

[0173] The scheduling module 602 is specifically used to sort the reasoning tasks contained in each reasoning task group according to the task priority corresponding to each reasoning task, so as to obtain the reasoning task sequence corresponding to each model type; sort the server nodes contained in each server node group according to the resource utilization score corresponding to each server node, so as to obtain the server node sequence corresponding to each model type; and distribute each reasoning task in each reasoning task sequence to the target server node in the server node sequence of the same model type in turn.

[0174] In a possible implementation, the scheduling module 602 is specifically used to determine, for each model type, the number of inference tasks included in the inference task group corresponding to the model type, and the number of server nodes included in the server node group corresponding to the model type; using a particle swarm algorithm, generate a configuration matrix based on the number of inference tasks and the number of server nodes, wherein the number of rows of the configuration matrix is ​​the number of inference tasks, and the number of columns of the configuration matrix is ​​the number of inference nodes; determine the elements in the configuration matrix whose corresponding values ​​are preset values, and determine the target inference task corresponding to the row where the element is located, and the target server node corresponding to the column where the element is located; and distribute the target inference task to the target server node.

[0175] In one possible implementation, the scheduling module 602 is specifically used to construct a blank matrix with the number of rows being the number of the reasoning tasks and the number of columns being the number of the reasoning nodes; assigning an initial position to each element in the blank matrix according to a preset random function; assigning an initial speed to each element according to the random function and a preconfigured maximum speed value and a minimum speed value; performing multiple iterations until a preset end condition is met, wherein each iteration process includes: updating the speed of each element obtained in the previous iteration according to a preset inertia factor, an individual learning factor, a social learning factor and the random function; and updating the position corresponding to each element obtained in the previous iteration according to the updated speed.

[0176] In a possible implementation, the scheduling module 602 is also used to obtain a preconfigured maximum inertia factor and a minimum inertia factor; determine a first difference between the maximum inertia factor and the minimum inertia factor; determine a first ratio of the square of the number of iterations to the square of a preset total number of iterations; determine a first product of the first difference and the first ratio, and determine a second difference between the maximum inertia factor and the first product as the inertia factor.

[0177] In a possible implementation, the scheduling module 602 is also used to determine a first ratio of the square value of the number of iterations to the square value of the preset total number of iterations, and to determine a third difference between the preset value and the first ratio; to determine the second product of the preset target learning factor corresponding to the current iteration and the third difference as the individual learning factor; and to determine the third product of the target learning factor and the first ratio as the social learning factor.

[0178] In a possible implementation, the scheduling module 602 is also used to determine the target fitness corresponding to the current iteration according to a preset objective function; obtain the maximum fitness and average fitness determined before the current iteration, and obtain the preset maximum learning factor, minimum learning factor and curve smoothing control factor; if the target fitness is less than the average fitness, the maximum fitness is determined as the target learning factor; if the target fitness is not less than the average fitness, the fourth difference between the target fitness and the average fitness, and the fifth difference between the maximum fitness and the average fitness are determined; the second ratio of the fourth difference to the fifth difference is determined, and the fourth product of the second ratio and the curve smoothing control factor is determined; a tangent operation is performed on the fourth product to obtain a tangent value; the sixth difference between the maximum learning factor and the minimum learning factor is determined; the fifth product of the sixth difference and the tangent value is determined; and the sum of the fifth product and the minimum learning factor is determined as the target learning factor.

[0179] In a possible implementation, the processing module 601 is specifically used to obtain pre-configured GPU weight, CPU weight, memory weight, network bandwidth weight and disk IO weight; based on the GPU weight, CPU weight, memory weight, network bandwidth weight and disk IO weight, the GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each server node are weighted summed to determine the resource utilization score of each server node.

[0180] On the basis of the above embodiments, an embodiment of the present invention further provides a server, which includes at least a processor and a memory, and the processor is used to implement the scheduling method steps provided in the above embodiments when executing.

[0181] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the scheduling method, the implementation of the above-mentioned computer-readable storage medium can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0182] Based on the above embodiments, the present application also provides an electronic device, Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, it includes: a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704;

[0183] The memory 703 stores a computer program. When the program is executed by the processor 701, the processor 701 executes the steps of the scheduling method provided in the above embodiments.

[0184] Since the principle of solving the problem by the above electronic device is similar to that of the scheduling method, the implementation of the above electronic device can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0185] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 702 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0186] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0187] Based on the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the scheduling method steps provided in the above embodiments when executing.

[0188] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the scheduling method, the implementation of the above-mentioned computer-readable storage medium can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0189] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0190] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A scheduling method, characterized in that: The method comprises: Determine the resource utilization score of each candidate server node according to the saved GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each candidate server node; wherein each candidate server node includes a small model server node, a large model server node and a large and small model mixed server node; Filtering each server node for scheduling from each candidate server node according to the resource utilization score of each candidate server node and a preset threshold; According to a preset server node scheduling strategy, each inference task to be processed is distributed to each server node.

2. The method according to claim 1, characterized in that: Each reasoning task carries a model type of a model that executes the reasoning task; The distributing the inference task to be processed to each server node according to the preset server node scheduling strategy includes: Classifying each reasoning task according to the model type corresponding to each reasoning task to obtain a reasoning task group corresponding to each model type; According to the model types of idle models in the large-model mixed server nodes in each of the current server nodes and the model types deployed in the non-large-model mixed server nodes in each of the saved server nodes, each of the server nodes is classified to obtain a server node group corresponding to each model type; Each inference task in each inference task group is distributed to a target server node in a server node group of the same model type.

3. The method according to claim 2, characterized in that Each reasoning task also carries a task priority; The distributing each reasoning task in each reasoning task group to a server node in a server node group of the same model type comprises: According to the task priority corresponding to each reasoning task, the reasoning tasks included in each reasoning task group are sorted to obtain a reasoning task sequence corresponding to each model type; According to the resource utilization score corresponding to each server node, the server nodes included in each server node group are sorted to obtain a server node sequence corresponding to each model type; Each inference task in each inference task sequence is distributed in sequence to a target server node in a server node sequence of the same model type.

4. The method according to claim 2, characterized in that: The distributing each reasoning task in each reasoning task group to each server node in the server node group of the same model type comprises: For each model type, determine the number of inference tasks included in the inference task group corresponding to the model type, and the number of server nodes included in the server node group corresponding to the model type; use a particle swarm algorithm to generate a configuration matrix based on the number of inference tasks and the number of server nodes, wherein the number of rows of the configuration matrix is ​​the number of inference tasks, and the number of columns of the configuration matrix is ​​the number of inference nodes; determine the elements in the configuration matrix whose corresponding values ​​are preset values, and determine the target inference tasks corresponding to the rows where the elements are located, and the target server nodes corresponding to the columns where the elements are located; distribute the target inference tasks to the target server nodes.

5. The method according to claim 4, characterized in that The particle swarm algorithm is used to generate a configuration matrix according to the number of inference tasks and the number of server nodes, including: Construct a blank matrix with the number of rows equal to the number of the inference tasks and the number of columns equal to the number of the inference nodes; According to a preset random function, assigning an initial position to each element in the blank matrix; Assigning an initial speed to each of the elements according to the random function and a preconfigured maximum speed value and a preconfigured minimum speed value; Perform multiple iterations until the preset end condition is met, wherein each iteration process includes: According to the preset inertia factor, the individual learning factor, the social learning factor and the random function, the speed of each element obtained in the previous iteration is updated; According to the updated speed, the position corresponding to each element obtained in the previous iteration is updated.

6. The method according to claim 5, characterized in that The process of determining the inertia factor includes: Get the pre-configured maximum and minimum inertia factors; determining a first difference between the maximum inertia factor and the minimum inertia factor; Determine a first ratio of the square value of the number of iterations to the square value of the preset total number of iterations; A first product of the first difference and the first ratio is determined, and a second difference between the maximum inertia factor and the first product is determined as the inertia factor.

7. The method according to claim 5, characterized in that The process of determining the individual learning factor and the social learning factor includes: Determine a first ratio of the square value of the number of iterations to the square value of a preset total number of iterations, and determine a third difference between the preset value and the first ratio; Determine a second product of a preset target learning factor corresponding to the current iteration and the third difference as the individual learning factor; A third product of the target learning factor and the first ratio is determined as the social learning factor.

8. The method according to claim 7, characterized in that The process of determining the target learning factor includes: According to the preset objective function, determine the target fitness corresponding to the current iteration; Obtain the maximum fitness and average fitness determined before the current iteration, and obtain the preset maximum learning factor, minimum learning factor and curve smoothing control factor; If the target fitness is less than the average fitness, determining the maximum fitness as the target learning factor; If the target fitness is not less than the average fitness, determine the fourth difference between the target fitness and the average fitness, and the fifth difference between the maximum fitness and the average fitness; determine the second ratio of the fourth difference to the fifth difference, and determine the fourth product of the second ratio and the curve smoothing control factor; perform a tangent operation on the fourth product to obtain a tangent value; determine the sixth difference between the maximum learning factor and the minimum learning factor; determine the fifth product of the sixth difference and the tangent value; and determine the sum of the fifth product and the minimum learning factor as the target learning factor.

9. The method according to claim 1, characterized in that: Determining the resource utilization score of each server node according to the saved GPU utilization, CPU utilization, memory utilization, network bandwidth utilization, and disk IO utilization corresponding to each server node includes: Get the pre-configured GPU weight, CPU weight, memory weight, network bandwidth weight, and disk IO weight; According to the GPU weight, CPU weight, memory weight, network bandwidth weight and disk IO weight, a weighted sum is performed on the GPU utilization, CPU utilization, memory utilization, network bandwidth utilization and disk IO utilization corresponding to each server node to determine a resource utilization score for each server node.

10. A server, characterized in that: The server comprises at least a processor and a memory, and the processor is used to implement the steps of the scheduling method according to any one of claims 1 to 9 when executing a computer program stored in the memory.

Images