Large-scale model inference scheduling method and system in multi-node heterogeneous environment

Abstract

This invention relates to the field of multi-node task data processing technology, specifically to a method and system for scheduling large-scale model inference in a multi-node heterogeneous environment. The method extracts stage vectors for each computing node under two stages of the inference task, further determining power consumption characteristic differences and excess latency. Two independent predictors are trained based on these two features. A directed graph of state transitions is constructed for each computing node, with edge weights determined by the stage vector differences between state nodes and the additional risk obtained from the predictors. After determining the node corresponding to the task to be inferred in the directed graph of state transitions, the optimal path for each computing node for that task is determined, thereby selecting the optimal execution computing node. This invention avoids hardware performance instability caused by blindly selecting execution computing nodes, thus preventing the task from being affected.

Description

Technical Field

[0001] This invention relates to the field of multi-node task data processing technology, specifically to a method and system for scheduling large-scale model inference in a multi-node heterogeneous environment. Background Technology

[0002] Large language model inference services are increasingly being deployed in multi-node heterogeneous computing environments composed of different types of computing devices. Existing scheduling methods allocate inference tasks mainly based on the static specifications of the devices (such as model and memory capacity) and real-time macro load indicators (such as average utilization). Such methods treat inference tasks as a homogeneous computational load. However, the inherent load characteristics of large model inference are not constant, but exhibit significant two-stage changes throughout its lifecycle: first, a highly parallel, computationally intensive prompt word processing (Prefill) stage, followed by a high-frequency, low-intensity memory-access-intensive token generation (Decode) stage.

[0003] Existing scheduling methods ignore the phased changes in these internal load characteristics and fail to recognize the differences and asymmetries in the response of different hardware architectures to these changes. Specifically, when a node's task ends at a low-power Decode phase and immediately switches to the high-power Prefill phase of the next task (i.e., the "load ramping process"), the transient response capability of its power supply network is tested; conversely, the load ramping process from Prefill to Decode places different demands on the system's power offloading stability. Existing methods cannot quantify and predict the performance stability of computing nodes under such asymmetric state transitions, and therefore cannot anticipate and avoid performance spikes caused by hardware response misalignment (such as voltage drops and clock frequency fluctuations), resulting in long-tailed delays in service response time. Summary of the Invention

[0004] To address the issue that existing technologies fail to effectively consider the hardware performance stability of computing nodes during the transitions between stages within and across inference tasks when selecting computing nodes for inference tasks, the present invention aims to provide a method and system for scheduling large-scale model inference in a multi-node heterogeneous environment. The specific technical solution adopted is as follows:

[0005] This invention proposes a method for scheduling large-scale model inference in a multi-node heterogeneous environment, the method comprising:

[0006] For each computing node, the power consumption and load of the prompt word processing stage and the token generation stage in each inference task are statistically analyzed to obtain the stage vector of each stage; the power consumption feature difference between the two stages in each inference task, as well as the excess latency of the inference task are obtained.

[0007] Using the stage vector of each inference task as input and the power consumption feature difference as label, we train an in-task load derating risk predictor; using the two stage vectors between the token generation stage and the prompt word processing stage as input and the excess latency corresponding to the prompt word processing stage as label, we train a cross-task load derating latency predictor.

[0008] For each computing node, the inferred task is used as a state node, and a directed graph of state transitions is constructed. The edge weight of each bidirectional edge between adjacent state nodes in the directed graph of state transitions is obtained from the stage vector difference between the state nodes and the additional risk. The additional risk is obtained by inputting the feature vectors of the two state nodes into the in-task load reduction risk predictor or the cross-task load increase delay predictor.

[0009] For each computing node, the task to be inferred is matched with the tasks that have already been inferred. Based on the state node corresponding to the matched tasks and the initial state of the computing node, the optimal path in the directed graph of state transition is determined. Based on the optimal path and the hardware state of each computing node, the optimal execution computing node for the task to be inferred is selected.

[0010] Furthermore, the method for obtaining the power consumption characteristic differences includes:

[0011] In a reasoning task, the peak power consumption during the prompt word processing stage is taken as the first power consumption feature, and the average power consumption during the token generation stage is taken as the second power consumption feature. The absolute value of the difference between the first power consumption feature and the second power consumption feature is taken as the numerator, and the duration of the reasoning task is taken as the denominator to obtain the power consumption feature difference.

[0012] Furthermore, the method for obtaining the excess delay includes:

[0013] For each inferenced task, predict the expected latency under ideal conditions based on the total load of the two stages. Use the difference between the actual latency and the expected latency as the numerator and the expected latency as the denominator to obtain the excess latency.

[0014] Furthermore, the state node includes a biased prompt word processing node and a biased token generation node; for a state node, if the computational load of the prompt word processing stage is greater than the memory access load of the token generation stage, then the state node is a biased prompt word processing node; otherwise, it is a token generation node.

[0015] Furthermore, the edge weight is a weighted sum of the stage vector difference and the additional risk according to preset weights.

[0016] Furthermore, the method for obtaining the additional risk includes:

[0017] If the bidirectional edge is a biased prompt word processing node pointing to a biased token generation node, then the stage vector of the prompt word processing stage of the biased prompt word processing node and the stage vector of the token generation stage of the biased token generation node are merged and used as the input of the in-task load reduction risk predictor, and the output result is the additional risk.

[0018] If the bidirectional edge is a biased token generation node pointing to a biased prompt word processing node, then the stage vector of the token generation stage of the biased token generation node and the stage vector of the prompt word processing stage of the biased prompt word processing node are merged and used as the input of the cross-task load delay predictor, and the output result is the additional risk.

[0019] If the two state nodes corresponding to the bidirectional edge are of the same type, then the additional risk is set to 0.

[0020] Furthermore, the method for obtaining the initial state of the computing node includes:

[0021] If the computing node is in a task execution state, the load information corresponding to the most recently completed token generation stage of the computing node is used as the initial state.

[0022] If the computing node is in an idle state, the stable load information in the stable standby state will be used as the initial state.

[0023] Furthermore, the method for obtaining the optimal path includes:

[0024] The total computational load of the task to be inferred is matched with all biased prompt word processing nodes in the state transition directed graph to obtain the prompt word processing stage anchor point; the total memory access load of the task to be inferred is matched with all biased token generation nodes in the state transition directed graph to obtain the token generation stage anchor point; the initial state of the computation node is matched with the biased token generation node in the state transition directed graph to obtain the initial state anchor point.

[0025] In the directed graph of state transition, a path optimization algorithm is used to obtain the first sub-optimal path between the initial state anchor point and the prompt word processing stage anchor point, and to obtain the second sub-optimal path between the prompt word processing stage anchor point and the token generation stage anchor point; the first sub-optimal path and the second sub-optimal path are merged as the optimal path.

[0026] Furthermore, the method for selecting the optimal execution computing node includes:

[0027] The path loss is obtained by summing the weights of all edges corresponding to the optimal path; an unhealthy score is obtained based on the current hardware degradation state of the computing node; the path loss and the unhealthy score are normalized and then weighted and summed with preset weights to obtain the execution loss of the computing node; the computing node with the smallest execution loss is selected as the optimal execution computing node for the inference task.

[0028] This invention proposes a large model inference scheduling system in a multi-node heterogeneous environment, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the large model inference scheduling method in a multi-node heterogeneous environment.

[0029] The present invention has the following beneficial effects:

[0030] This invention extracts the stage vectors of the two stages of the inference task and further analyzes the inference task to determine the differences in power consumption characteristics and excess latency. These two characteristics represent the adjustment pressure faced by the computing nodes during the deload and load-in processes, respectively. Therefore, this invention further trains two independent predictors based on these two characteristics. Considering that processors have different internal hardware bottlenecks when dealing with sudden increases and decreases in power consumption, this invention uses two independent predictors, trained independently, to predict the computing node pressure during various state transitions. Furthermore, a directed graph of state transitions is constructed for each computing node. This graph describes the computing node pressure loss from one state node to another through the paths between state nodes. Therefore, the edge weights are determined by both the stage vector differences between state nodes and the additional risks obtained from the predictors. After determining the node corresponding to the task to be inferred in the directed graph of state transitions, the optimal path for each computing node for that task can be determined. Then, combined with the hardware state, it can be determined whether the current computing node is suitable for executing the task to be inferred. This invention identifies the optimal execution computing node for the task to be inferred by analyzing the state transitions of the computing node and the load characteristics of the task. This avoids hardware performance instability caused by blindly selecting the computing node, which in turn affects task execution. Attached Figure Description

[0031] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 The flowchart illustrates a large model inference scheduling method in a multi-node heterogeneous environment, as provided in one embodiment of the present invention. Detailed Implementation

[0033] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a large-model inference scheduling method and system in a multi-node heterogeneous environment proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0035] The following description, in conjunction with the accompanying drawings, details the specific scheme of a large model inference scheduling method and system provided by the present invention in a multi-node heterogeneous environment.

[0036] Please see Figure 1 The diagram illustrates a flowchart of a large-model inference scheduling method in a multi-node heterogeneous environment according to an embodiment of the present invention. The method includes:

[0037] Step S1: For each computing node, calculate the power consumption and load of the prompt word processing stage and the token generation stage in each inference task, and obtain the stage vector of each stage; obtain the power consumption feature difference between the two stages in each inference task, as well as the excess latency of the inference task.

[0038] It should be noted that the embodiments of the present invention aim to select the optimal execution computing node suitable for the inference task from multiple computing nodes. The analysis method for each computing node is the same, requiring analysis based on the computing node's own historical data and subsequent model and graph structure construction. Therefore, it will not be described in detail. The embodiments of the present invention will only use one computing node as an example in the following description. Furthermore, the inference tasks selected in the embodiments of the present invention are all inference tasks recently executed by the computing node. The characteristics reflected in the execution stage of these inference tasks are the current hardware processing state of the computing node.

[0039] For large-scale model inference tasks, power consumption during execution is not uniform throughout the task cycle. High instantaneous power consumption occurs during the prompt word processing phase, while lower sustained power consumption occurs during the token generation phase. Therefore, directly analyzing the macroscopic power consumption metrics across the entire task cycle masks the different pressures exerted on the computing node hardware by the two phases. Hardware components such as the power supply network and cooling system are affected differently by the power consumption of the inference task. Therefore, this embodiment of the invention requires the separate integration and statistical analysis of various characteristics from the two different phases of an inference task to determine the phase vector for each phase. The key information in the phase vector includes the power consumption during the inference task processing and the load required for the corresponding phase. The load in the prompt word processing phase is the computational load, while the load in the token generation phase is the memory access load.

[0040] It should be noted that, since each of the two stages in the reasoning task corresponds to an instantaneous power consumption at each moment within the time range, this embodiment of the invention obtains two instantaneous power consumption sequences throughout the entire task cycle of the reasoning task, representing the instantaneous power consumption sequence of the prompt word processing stage and the instantaneous power consumption sequence of the token generation stage, respectively. Based on the two instantaneous power consumption sequences and the characteristics of their respective stages, the power consumption characteristics of each stage can be obtained.

[0041] To further analyze the adjustment capability of computing nodes between stage transitions, this embodiment of the invention first obtains the power consumption characteristic difference between the two stages in the inferred task. Since the inferred task is executed in the order of first executing the prompt word processing stage and then executing the token generation stage, that is, the inferred task itself has already performed a load reduction process. Therefore, the power consumption characteristic difference between the two stages represents the load reduction characteristic of the computing node when executing the inferred task. Secondly, this embodiment of the invention obtains the excess latency of the inferred task. The excess latency represents whether the inferred task encountered performance loss due to hardware factors such as reaching power consumption critical conditions or temperature critical conditions during execution, resulting in untimely response of the computing node. This excess latency is determined by the hardware state of the computing node. The excess latency can be used to quantify the load increase characteristics of the computing node during the cross-task process, that is, the load increase process.

[0042] Preferably, in this embodiment of the invention, considering that the prompt word processing stage is a high instantaneous power consumption stage and the token generation stage is a low power consumption continuous stage, the method for obtaining the power consumption characteristic difference includes:

[0043] In a reasoning task, the peak power consumption during the prompt word processing phase is used as the first power consumption feature. The first power consumption feature quantifies the maximum instantaneous power requirement that the computing node needs to meet when dealing with large-scale parallel computing. The average power consumption during the token generation phase is used as the second power consumption feature. The second power consumption feature quantifies the stable operating power consumption of the system when the computing node is dealing with high-frequency continuous memory access load.

[0044] It should be noted that, because the power consumption characteristics of each stage are extracted in the embodiments of the present invention, the stage vector for each stage is a two-dimensional vector composed of the power consumption characteristics and the required load. For example, the stage vector of the prompt word processing stage is {first power consumption characteristic, total load calculated}, and the stage vector of the token generation stage is {second power consumption characteristic, total memory access load}.

[0045] This invention further considers that during the load reduction process, the rapid decrease in power demands the adjustment capability of the computing node, and this rapid decrease may lead to voltage oscillations, impacting the stability of the power supply system. Therefore, to quantify the severity of the load reduction process, this invention uses the absolute value of the difference between the first power consumption characteristic and the second power consumption characteristic as the numerator, and the duration of the inference task as the denominator to obtain the power consumption characteristic difference. That is, the power consumption characteristic difference is a power consumption decrease rate, characterizing the rate of change of power when switching from peak computing mode to continuous memory access mode.

[0046] Preferably, in this embodiment of the invention, considering that the actual latency of the original execution of a task cannot clearly characterize the impact of the computing node's hardware state on the execution of the inferred task, this embodiment compares it with the expected latency. For each inferred task, the expected latency under ideal conditions is predicted based on the total load of the two stages. The difference between the actual latency and the expected latency is used as the numerator, and the expected latency is used as the denominator to obtain the excess latency. The larger the normalized excess latency obtained by this fraction, the greater the actual latency is than the expected latency under ideal conditions, indicating that the instantaneous hardware conditions of the computing node are poor.

[0047] It should be noted that the method for obtaining the expected latency is a technical means well known to those skilled in the art, and can be directly predicted from the total computing load and total memory access load of the two stages, and will not be elaborated further.

[0048] Finally, for step S1, each recently processed inference task by the computing node corresponds to two stages of power consumption information, load information, power consumption characteristic differences, and excess latency—these are execution characteristics. It should be noted that these execution characteristics, due to their respective dimensions and data ranges, have large numerical scales between different execution characteristics. Direct use in subsequent processing would affect the final result. Therefore, these execution characteristics should be standardized in their respective dimensions. That is, the numerical set of execution characteristics for each dimension is obtained from the database. This invention uses Z-standardization to process the execution characteristics of each dimension, transforming all execution characteristics into a dimensionless distribution with a mean of 0 and a standard deviation of 1, providing an unbiased data foundation for subsequent steps. Specific standardization steps are techniques well-known to those skilled in the art. In other embodiments of this invention, methods such as range standardization can also be used for processing; specific details will not be elaborated or limited. In subsequent processing, this embodiment directly uses the standardized data for processing, without further specific explanation.

[0049] Step S2: Using the stage vector of each inference task as input and the power consumption feature difference as label, train an intra-task load derating risk predictor; using the two stage vectors between the token generation stage and the prompt word processing stage as input and the excess latency corresponding to the prompt word processing stage as label, train a cross-task load derating latency predictor.

[0050] In this embodiment of the invention, the processor's internal hardware bottlenecks are completely different when dealing with the load increase and load decrease processes. The load increase process tests the transient response capability of the power supply network, while the latter tests the stability of power offloading. Therefore, for these two processes, in order to construct the directed graph of subsequent state transitions, this embodiment of the invention needs to construct two independent predictors to quantify the risks of the two asymmetric transitions respectively.

[0051] Since a single inferred task is already a complete deload process, to quantify the impact of state transitions within the task, we can use the statistical data from each inferred task. We concatenate the stage vectors corresponding to the prompt processing and token generation stages as input data, and use the power consumption characteristics differences of the inferred tasks as supervision labels for training. Based on the training data, we train a gradient boosting regression model to obtain an in-task deload risk predictor.

[0052] For the overloading process, it is a cross-task process, namely, from the token generation stage of the previous inference task to the prompt word processing stage of the current inference task. Therefore, the temporal vectors of the two stages between the token generation stage and the prompt word processing stage are concatenated as input, and the excess latency corresponding to the prompt word processing stage is used as the label. The excess latency directly quantifies the extent to which the execution of the subsequent prompt word processing stage is negatively affected, thus reflecting the actual pressure faced by the computing node under this state transition. Similarly, a gradient boosting regression model is trained to obtain the cross-task overloading latency predictor.

[0053] By training two predictors, the concatenation results of vectors at different stages can be used as inputs to obtain the corresponding power consumption feature differences or the computational node pressure under the stage transition corresponding to the excess latency as input data.

[0054] It should be noted that the embodiments of the present invention can set an update or reconstruction mechanism for the predictor, that is, to count all inference tasks counted in real time step S1. When the number of counted tasks exceeds a preset threshold, the update or reconstruction mechanism is activated to retrain the predictor so that the predictor can be aligned with the real-time hardware conditions of the corresponding computing node and ensure the reference value of the output results.

[0055] Step S3: For each computing node, the inferred task is used as a state node to construct a directed graph of state transitions; the edge weight of each bidirectional edge between adjacent state nodes in the directed graph of node state transitions is obtained from the stage vector difference between the state nodes and the additional risk; the additional risk is obtained by inputting the feature vectors of the two state nodes into the in-task load reduction risk predictor or the cross-task load increase delay predictor.

[0056] To illustrate the computational pressure faced by computing nodes during load increases or decreases under different conditions, this embodiment of the invention constructs a directed graph of state transitions, using inferred tasks as state nodes. Because the inferred tasks identified in step S1 are fixed-form stage transition tasks, directly matching the database identified in step S1 with the task to be inferred could result in no match or an unsuitable inferred task. Therefore, this embodiment constructs a directed graph of state transitions, where each inferred task is a state node, and adjacent state nodes are connected by a bidirectional edge, representing a path from the first state node to the second state node, or vice versa. Each edge has two weights, each corresponding to one direction of the bidirectional edge, representing the computational pressure generated during the transition from one state node to another. By constructing the directed graph of state transitions, the computational pressure faced by each state node to any other state node can be obtained; that is, by finding the path between two nodes through path optimization, the computational pressure faced by the computing node during this state transition can be determined.

[0057] In this embodiment of the invention, the K-nearest neighbor algorithm is first used to identify the K nearest neighboring state nodes for each state node, establishing an initial adjacency relationship. K can be set according to the number of inference tasks to be statistically analyzed. The K-nearest neighbor algorithm is a well-known technique. In this embodiment, the Euclidean distance between the stage vectors of two state nodes at the same stage is calculated, and the average Euclidean distance is selected as the node distance. That is, for any state node, the K nearest neighboring state nodes are its closest neighboring state nodes. Corresponding analysis is performed on each state node to obtain the adjacency relationship of all state nodes, thus obtaining a preliminary directed graph of state transitions.

[0058] This embodiment of the invention further sets edge weights between adjacent state nodes, introducing additional risks in addition to the stage vector differences between state nodes. The additional risks are obtained by inputting the feature vectors of the two state nodes into an intra-task load reduction risk predictor or a cross-task load increase delay predictor. That is, for any two adjacent state nodes, if the process from the first state node to the second state node is determined to be a load reduction process, the corresponding stage vector is input into the intra-task load reduction risk predictor, and the output result is the additional risk; if the process is determined to be a load increase process, the corresponding stage vector is input into the cross-task load increase delay predictor, and the output result is the additional risk.

[0059] Preferably, in this embodiment of the invention, since a state node represents an entire reasoned task, which includes both the prompt word processing stage and the token generation stage, in order to reflect the switching between different states in the directed graph of state transitions, each state node can be considered as a state node dominated by a certain stage based on the load bias corresponding to it. That is, state nodes are divided into prompt word processing biased nodes and token generation biased nodes. For a state node, if the computational load of the prompt word processing stage is greater than the memory access load of the token generation stage, then the state node is a prompt word processing biased node; otherwise, it is a token generation node. By classifying state nodes, it is convenient to qualitatively characterize the state transition type of an edge in the directed graph of state transitions. That is, if an edge is a prompt word processing biased node pointing to a token generation biased node, it indicates that the edge corresponds to a load reduction process; conversely, if an edge is a token generation biased node pointing to a prompt word processing biased node, it indicates that the edge corresponds to a load increase process.

[0060] Preferably, in this embodiment of the invention, the method for obtaining additional risks includes:

[0061] If the bidirectional edge points from a biased prompt word processing node to a biased token generation node, then the stage vector of the prompt word processing stage of the biased prompt word processing node and the stage vector of the token generation stage of the biased token generation node are merged and used as the input of the in-task load reduction risk predictor, and the output result is the additional risk.

[0062] If the bidirectional edge is a biased token generation node pointing to a biased prompt word processing node, then the stage vector of the token generation stage of the biased token generation node and the stage vector of the prompt word processing stage of the biased prompt word processing node are merged and used as the input of the cross-task load delay predictor, and the output result is the additional risk.

[0063] If the two state nodes corresponding to the bidirectional edge are of the same type, then the additional risk is set to 0.

[0064] Preferably, in this embodiment of the invention, a weighted fusion method is used to fuse the stage vector difference and the additional risk to obtain the edge weight, that is, the edge weight is the weighted sum of the stage vector difference and the additional risk according to preset weights. In this embodiment of the invention, the weights of both the stage vector difference and the additional risk are set to 0.5, that is, the edge weight can be regarded as the average value of the stage vector difference and the additional risk. It should be noted that, in one embodiment of the invention, the stage vector difference is the node distance corresponding to the state node in the K-nearest neighbor algorithm.

[0065] Step S4: For each computing node, match the task to be inferred with the tasks that have already been inferred. Based on the state node corresponding to the matched tasks that have already been inferred and the initial state of the computing node, determine the optimal path in the directed graph of state transitions. Based on the optimal path and the hardware state of each computing node, select the optimal execution computing node for the task to be inferred.

[0066] Based on the analysis and model construction described above, each computation node can obtain a corresponding directed graph of state transitions. For a newly arrived, yet-to-be-executed task to be reasoned about, since it does not belong to the previously counted reasoned tasks, a direct path query cannot be performed in the directed graph of state transitions. Therefore, it is necessary to match the task to be reasoned about with the already reasoned tasks. Based on the state nodes corresponding to the matched already reasoned tasks and the initial state of the computation node, the optimal path in the directed graph of state transitions is determined. The optimal path includes three important anchor points: the first is the initial state of the computation node; the second is the state node matched by the task to be reasoned about, representing the prompt word processing stage; and the third is the state node matched by the task to be reasoned about, representing the token generation stage. That is, the optimal path starts from the first anchor point, first performs path optimization to the second anchor point, and then continues path optimization to determine the path between the second and third anchor points, thus completing the construction of the optimal path.

[0067] The path optimization algorithm used in this embodiment of the invention is the Floyd-Warshall algorithm. The specific algorithm process is a well-known technique to those skilled in the art and will not be described in detail here.

[0068] Preferably, in this embodiment of the invention, the method for obtaining the initial state of a computing node includes:

[0069] If the computing node is in a task execution state, the load information corresponding to the most recently completed token generation stage of the computing node is used as the initial state.

[0070] If the computing node is in an idle state, the stable load information in the stable standby state will be used as the initial state.

[0071] Preferably, in this embodiment of the invention, the method for obtaining the optimal path includes:

[0072] The total computational load of the task to be inferred is matched with all biased prompt word processing nodes in the state transition directed graph to obtain the prompt word processing stage anchor point; the total memory access load of the task to be inferred is matched with all biased token generation nodes in the state transition directed graph to obtain the token generation stage anchor point; the initial state of the computation node is matched with the biased token generation node in the state transition directed graph to obtain the initial state anchor point.

[0073] It should be noted that in the three matching processes mentioned above, the matching is based on load information. Taking the first matching process as an example, the difference between the total computational load of the task to be inferred and the total computational load corresponding to all biased prompt word processing nodes is calculated. The biased prompt word processing node with the smallest difference is selected as the anchor point for the prompt word processing stage. If there are multiple nodes with the smallest difference, any one can be selected as the anchor point for the prompt word processing stage. The other two matching processes are similar and will not be described in detail.

[0074] In the directed graph of state transitions, a path optimization algorithm is used to obtain a first sub-optimal path between the initial state anchor point and the prompt word processing stage anchor point, representing the loading process of task startup; a second sub-optimal path is obtained between the prompt word processing stage anchor point and the token generation stage anchor point, representing the unloading process during task execution; the first and second sub-optimal paths are merged to form the optimal path. It should be noted that because the edges in the directed graph of state transitions in this embodiment are bidirectional edges, and the two state nodes in the obtained optimal path have a unique direction, the corresponding edge weight should be the edge weight corresponding to that direction.

[0075] After obtaining the optimal path, the larger the edge weight corresponding to the optimal path, the greater the computational pressure faced by the task to be inferred under that computing node. Therefore, we can further determine the computational pressure of each computing node for the task to be inferred by combining the current hardware status of the computing nodes, and then select the optimal computing node to execute.

[0076] Preferably, in this embodiment of the invention, the method for selecting the optimal execution computing node includes:

[0077] The path loss is obtained by summing the weights of all edges corresponding to the optimal path; an unhealthy score is obtained based on the current hardware degradation state of the computing node; the path loss and the unhealthy score are normalized and then weighted and summed with preset weights to obtain the execution loss of the computing node; the computing node with the smallest execution loss is selected as the optimal execution computing node for the inference task.

[0078] In this embodiment of the invention, the method for obtaining the unhealthy score is as follows: the temperature index of the computing node is selected as the hardware status index, and the ratio of the temperature to the hardware critical temperature is used as the unhealthy score. That is, the closer the temperature of the computing node is to the specified maximum value, the closer the computing node status is to the limit state, and the larger the unhealthy score is. It should be noted that in other embodiments of the invention, other hardware statuses such as the current power consumption can also be selected to obtain the unhealthy score, and all can be analyzed based on critical conditions, which will not be elaborated further.

[0079] In this embodiment of the invention, the weights of the normalized path loss and the unhealthy score are both set to 0.5, and the execution loss of the computing node is obtained by weighted summation.

[0080] In summary, this invention extracts stage vectors for each computing node under the two stages of the inference task, further determining the differences in power consumption characteristics and excess latency. Two independent predictors are trained based on these two features. A directed graph of state transitions is constructed for each computing node, with edge weights determined by the stage vector differences between state nodes and the additional risk obtained from the predictors. After determining the node corresponding to the task to be inferred in the directed graph of state transitions, the optimal path for each computing node for that task can be determined. Furthermore, combined with the hardware state, it can be determined whether the current computing node is suitable for executing the task. This invention finds the optimal computing node for executing the task from the state transitions of the computing node and the load characteristics of the task to be inferred, avoiding hardware performance instability caused by blindly selecting computing nodes, which could affect task execution.

[0081] Based on the same inventive concept, this invention also proposes a large model inference scheduling system in a multi-node heterogeneous environment, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the steps of the large model inference scheduling method in a multi-node heterogeneous environment described above.

[0082] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0083] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A large model inference scheduling method in a multi-node heterogeneous environment, characterized in that, The method comprises: For each computing node, statistics the power consumption and load of the prompt word processing stage and the token generation stage in each reasoned task to obtain a stage vector of each stage; obtain the power consumption feature difference between the two stages in each reasoned task, and the excess delay of the reasoned task; The stage vector of each reasoned task is taken as input, and the power consumption feature difference is taken as label to train an intra-task load reduction risk predictor; the two stage vectors between the token generation stage and the prompt word processing stage in time sequence are taken as input, and the excess delay corresponding to the prompt word processing stage is taken as label to train a cross-task load increase delay predictor; For each computing node, the reasoned task is taken as a state node to construct a state transition directed graph; the edge weight of each bidirectional edge between adjacent state nodes in the node state transition directed graph is obtained by the stage vector difference and the additional risk between the state nodes; the additional risk is obtained by inputting the feature vectors of the two state nodes into the intra-task load reduction risk predictor or the cross-task load increase delay predictor; For each computing node, the to-be-reasoned task is matched with the reasoned task, and the optimal path in the state transition directed graph is determined according to the state node corresponding to the matched reasoned task and the initial state of the computing node; the optimal execution computing node of the to-be-reasoned task is screened out according to the optimal path and the hardware state of each computing node; The state node comprises a prompt word processing node and a token generation node; for a state node, if the computing load of the prompt word processing stage is greater than the memory load of the token generation stage, the state node is a prompt word processing node; otherwise, it is a token generation node; The optimal path comprises: The total computing load of the to-be-reasoned task is matched with all the prompt word processing nodes in the state transition directed graph to obtain a prompt word processing stage anchor point; the total memory load of the to-be-reasoned task is matched with all the token generation nodes in the state directed graph to obtain a token generation stage anchor point; the initial state of the computing node is matched with the token generation nodes in the state transition directed graph to obtain an initial state anchor point; In the state transition directed graph, a first sub-optimal path between the initial state anchor point and the prompt word processing stage anchor point is obtained by using a path optimization algorithm, and a second sub-optimal path between the prompt word processing stage anchor point and the token generation stage anchor point is obtained; the first sub-optimal path and the second sub-optimal path are combined as the optimal path.

2. The method of claim 1, wherein, The power consumption feature difference comprises: In a reasoned task, the peak power consumption in the prompt word processing stage is taken as a first power consumption feature, and the average power consumption in the token generation stage is taken as a second power consumption feature; the absolute value of the difference between the first power consumption feature and the second power consumption feature is taken as the numerator, and the time length of the reasoned task is taken as the denominator to obtain the power consumption feature difference.

3. The method of claim 1, wherein, The excess delay comprises: For each reasoned task, according to the total load of the two stages, the expected delay in the ideal state is predicted, the difference between the actual delay and the expected delay is taken as the numerator, and the expected delay is taken as the denominator to obtain the excess delay.

4. The method of claim 1, wherein, The edge weight is a weighted sum value of the stage vector difference and the additional risk according to a preset weight.

5. The method of claim 1, wherein, The method for obtaining the additional risk comprises: If the bidirectional edge is a bias prompt word processing node pointing to a bias token generation node, the stage vector of the prompt word processing stage of the bias prompt word processing node and the stage vector of the token generation stage of the bias token generation node are merged as the input of the intra-task load reduction risk predictor, and the output result is the additional risk; If the bidirectional edge is a bias token generation node pointing to a bias prompt word processing node, the stage vector of the token generation stage of the bias token generation node and the stage vector of the prompt word processing stage of the bias prompt word processing node are merged as the input of the cross-task load increase delay predictor, and the output result is the additional risk; If the two state nodes corresponding to the bidirectional edge are the same type of state nodes, the additional risk is set to 0.

6. The method of claim 1, wherein, The method for obtaining the initial state of the computing node comprises: If the computing node is in an execution task state, the load information corresponding to the token generation stage recently completed by the computing node is taken as the initial state; If the computing node is in an idle state, the stable load information in the stable standby state is taken as the initial state.

7. The method of claim 1, wherein, The method for screening the optimal execution computing node comprises: An accumulated sum of all edge weights corresponding to the optimal path is taken as a path loss, an unhealthy score is obtained according to the current hardware degradation state of the computing node, after the path loss and the unhealthy score are normalized respectively, a weighted sum is obtained by using a preset weight, an execution loss of the computing node is obtained, and a computing node with the minimum execution loss is selected as the optimal execution computing node of the reasoned task.

8. A large model inference scheduling system in a multi-node heterogeneous environment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, The processor executes the computer program to realize the steps of the large model inference scheduling method in a multi-node heterogeneous environment according to any one of claims 1-7.

Images