Cross-gpu parallel inference method and system for large language model

Abstract

The present application relates to the technical field of data processing, in particular to a cross-GPU parallel inference method and system for large language models. The method performs information statistics and feature extraction on the historical micro-batches that have been executed, determines the performance deviation between the expected processing time and the calculation execution time generated during the execution of each batch. A multi-stage offset matrix is constructed combining a plurality of consecutive historical micro-batches, and the fluctuation pattern and fluctuation intensity under the historical micro-batches are determined. The method of predicting risk is used to determine the risk of the inference system under each historical micro-batch relative to the future micro-batch. According to the basic information and risk information of the recent historical micro-batches, time simulation is carried out to realize accurate pipeline emptying prediction. Through data extraction and feature analysis of the historical micro-batches, the present application determines the cooperative fluctuation pattern of the GPU and then performs effective risk prediction, realizing effective execution time prediction of the pre-filled micro-batches.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a cross-GPU parallel inference method and system for large language models. Background Technology

[0002] Large language models commonly employ pipelined parallelism when processing long sequence inference requests, deploying the model layer in a distributed pipeline composed of multiple GPUs. This process is divided into two stages: prefill and decoding. Current technologies, when switching from the prefill to the decoding stage, must wait for the last micro-batch to pass through all pipeline stages to ensure data consistency; this process is called "pipeline emptying."

[0003] Current methods typically rely on simple predictions of historical average execution times to determine the completion time of draining. However, these predictions often treat individual GPU work units in the pipeline as independent sources of random performance fluctuations, neglecting the coupling relationships between them due to shared hardware resources (such as the power supply network and PCIe communication bus within the same server node). Load fluctuations in these shared resources can trigger coordinated performance fluctuations across multiple GPUs, forming a systemic, rather than isolated, performance change. Existing methods lack the ability to diagnose such coordinated fluctuations, failing to identify their inherent patterns (e.g., whether the fluctuations are concentrated at the front or back of the pipeline), and even less able to predict their development trends. Therefore, predictions based on isolated historical data or passive waiting inevitably lead to misjudgments of the draining completion time, creating a "transition delay" on the order of tens of milliseconds between physical draining completion and the logical state switch. During this period, system computing resources are idle, increasing the generation time of the first token and reducing system throughput. Summary of the Invention

[0004] To address the technical problem that existing technologies fail to effectively identify GPU-coordinated fluctuation patterns, leading to inaccurate prediction results and hindering inference tasks, this invention aims to provide a cross-GPU parallel inference method and system for large language models. The specific technical solution adopted is as follows:

[0005] This invention proposes a cross-GPU parallel inference method for large language models, the method comprising:

[0006] For each stage under the historical micro-batch, obtain the queue waiting time and computation execution time for each stage;

[0007] The expected processing time is obtained based on the base load processing time of each stage and the load of historical micro-batches. The base load processing time is updated according to the load size of historical micro-batches and the computation execution time. The performance deviation between the expected processing time and the computation execution time is obtained.

[0008] The performance deviation of all stages under multiple consecutive historical micro-batches is statistically analyzed to obtain a multi-stage performance deviation matrix. The fluctuation pattern and fluctuation intensity of the historical micro-batches are obtained based on the multi-stage performance deviation matrix.

[0009] Based on the fluctuation pattern and fluctuation intensity, predict the predicted intensity of future micro-batches from historical micro-batches; based on the predicted intensity, the queue waiting time of the downstream stage, the calculation execution time of the downstream stage, and the performance deviation of each stage, obtain the risk index of each stage under each historical micro-batch.

[0010] For the allocation time of the pre-filled micro-batch, the processing time is simulated based on the basic load processing time of each stage in the most recent historical micro-batch, and the time is corrected by combining the risk index of each stage to determine the synchronization target time of the pre-filled micro-batch; based on the synchronization target time, the control command is fed back during the real-time processing stage of the pre-filled micro-batch.

[0011] Furthermore, the method for updating the basic load processing time includes:

[0012] For each stage, the ratio of the computation execution time to the load of the micro-batch is used as the unit load processing time. The unit load processing time and the basic load processing time of the same stage under the previous historical micro-batch are weighted and summed according to a preset weight to obtain the basic load processing time.

[0013] Furthermore, the performance deviation is the difference between the computation execution time and the expected processing time.

[0014] Furthermore, obtaining the fluctuation patterns and fluctuation intensities of historical micro-batches includes:

[0015] Singular value decomposition is performed on the multi-stage performance deviation matrix to obtain the column vectors of the left singular vector matrix and the singular values ​​of the diagonal matrix; the column vectors are matched with the template vectors of all known fluctuation patterns, and the best matching known fluctuation pattern is taken as the fluctuation pattern of the historical micro-batch, and the singular value is the fluctuation intensity.

[0016] Furthermore, the known fluctuation patterns include front-end bottleneck pattern, end-end bottleneck pattern, and globally consistent pattern.

[0017] Furthermore, the method for obtaining the predicted intensity includes:

[0018] The fluctuation intensity of the target historical micro-batch and the previous historical micro-batch with the same fluctuation pattern are combined into a fluctuation intensity sequence. The fluctuation intensity sequence is then processed using a Kalman filter to obtain the predicted intensity.

[0019] Furthermore, the method for obtaining the risk index includes:

[0020] The predicted intensity is used as the system threat exposure; the buffer capacity of the downstream stage is obtained based on the queue waiting time and the computation execution time of the downstream stage; the system threat exposure is weighted according to the performance deviation of each stage to obtain the weighted system threat exposure; and the risk index of each stage is obtained based on the weighted system threat exposure and the buffer capacity of the downstream stage.

[0021] Furthermore, the buffer capacity is the ratio of the queue waiting time in the downstream stage to the computation execution time.

[0022] Further, determining the synchronization target time for the pre-filled micro-batch includes:

[0023] The baseline execution time for each stage in the most recent historical micro-batch is obtained by multiplying the base load processing time of the pre-filled micro-batch. The risk index for each stage in the most recent historical micro-batch is normalized, and the normalized risk index is multiplied by the baseline execution time to obtain the time correction amount.

[0024] If the fluctuation pattern of the most recent historical micro-batch belongs to the preset high-risk pattern, then the sum of the time correction amount and the baseline execution time is used as the target execution time for each batch; otherwise, the baseline execution time is directly used as the target execution time.

[0025] The overall target execution time is obtained by summing the target execution times of all batches in the pre-filled micro-batch, and the synchronization target time is determined based on the overall target execution time.

[0026] The present invention also proposes a cross-GPU parallel inference system for large language models, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the steps of the cross-GPU parallel inference method for large language models described above.

[0027] The present invention has the following beneficial effects:

[0028] This invention first performs information statistics and feature extraction on historical micro-batches that have already been executed. For each batch, targeted analysis is conducted to determine the performance deviation between the expected processing time and the computation execution time. This performance deviation is caused by GPU co-fluctuations in the inference system during the execution of historical micro-batches. Therefore, to effectively identify the fluctuation pattern, this invention constructs a multi-stage offset matrix by combining multiple consecutive historical micro-batches, integrating and quantizing the offset information into a matrix form. The fluctuation pattern and intensity under historical micro-batches can then be determined based on the offset distribution in the matrix. After accurately determining the fluctuation pattern, a risk prediction method can be used to determine the risk of the inference system's state relative to future micro-batches under each historical micro-batchens. Therefore, this invention performs predictive analysis on a stage-by-stage basis, combining the characteristics of downstream stages to determine the risk index of each stage. For real-time pre-filled micro-batches, time simulation can be performed based on the basic information and risk information of the most recent historical micro-batches to determine the synchronization target time, thereby achieving accurate pipeline emptying prediction. Attached Figure Description

[0029] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 The flowchart illustrates a cross-GPU parallel inference method for large language models, as provided in one embodiment of the present invention. Detailed Implementation

[0031] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a cross-GPU parallel inference method and system for large language models proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0033] The following description, in conjunction with the accompanying drawings, details the specific scheme of a cross-GPU parallel inference method and system for large language models provided by this invention.

[0034] Please see Figure 1 The diagram illustrates a flowchart of a cross-GPU parallel inference method for large language models provided by an embodiment of the present invention, the method comprising:

[0035] Step S1: For each stage under the historical micro-batch, obtain the queue waiting time and computation execution time for each stage.

[0036] The processing latency of a micro-batch at a certain stage mainly consists of two situations: first, queuing due to the computing resources being occupied at that stage; and second, idle waiting due to the upstream stage's data not yet arriving. The original scheduler logs and GPU execution events exist only as discrete timestamps, which cannot directly reveal the specific duration of these two types of waiting. Therefore, in order to achieve latency analysis for each stage, this embodiment of the invention first needs to determine the queue waiting time and computation execution time for each stage.

[0037] In this embodiment of the invention, a monitoring module is deployed in the central scheduler. This module monitors in real time the specific timestamps of the k-th micro-batch in the i-th stage in a system with a pipeline depth (number of stages) of P: 1. Task issuance time, which is the system time when the scheduler puts the task description of micro-batch k into the input work queue of stage i; 2. Calculation start time, which is the system time when the work unit of stage i takes micro-batch k out of its input queue and starts to execute the calculation; 3. Calculation completion time, which is the system time when the work unit of stage i completes all the calculations of micro-batch k and puts the output data into its output queue. Queue waiting time is the time difference between the start of computation and the time the task is issued. It represents the length of time that micro-batch k waits to be processed in the input queue of stage i. This value directly reflects the workload backlog of stage i. Its continuous increase indicates that the processing capacity of stage i may not be able to match the data supply rate of the upstream. Computation execution time is the time difference between the start of computation and the completion of computation. It represents the actual time consumed by stage i to process micro-batch k without queuing. It includes data transmission and model calculation and is the basic data for evaluating the computational throughput of stage i itself.

[0038] It should be noted that the embodiments of the present invention aim to statistically analyze the data of each stage of completed historical micro-batches in order to predict the execution status of unprocessed pre-filled micro-batches. Therefore, the data acquisition object of the embodiments of the present invention is multiple consecutive historical micro-batches preceding the pre-filled micro-batches.

[0039] Step S2: Obtain the expected processing time based on the base load processing time of each stage and the load of historical micro-batches. The base load processing time is updated according to the load size of historical micro-batches and the computation execution time; obtain the performance deviation between the expected processing time and the computation execution time.

[0040] For each stage, the inference system has a basic processing capacity, which determines the basic processing time required for the micro-batch load within that stage, thus determining the expected processing time for each stage. In this embodiment of the invention, to construct a dynamic performance baseline, the basic load processing time is updated based on the micro-batch load size and the computation execution time of each stage. The expected processing time for each stage can then be obtained based on the updated basic load processing time and the load of historical micro-batches. Since historical micro-batches represent historical data from completed inference tasks, the performance deviation between the expected processing time and the computation execution time can be obtained. This performance deviation represents the deviation from the expected result during actual task execution, and it can be used to determine whether the actual performance is lower or higher than expected.

[0041] Preferably, in this embodiment of the invention, the method for updating the basic load processing time includes:

[0042] For each stage, the ratio of the computation execution time to the load of the micro-batch is used as the unit load processing time. The unit load processing time and the basic load processing time of the same stage under the previous historical micro-batch are weighted and summed according to a preset weight to obtain the basic load processing time.

[0043] In this embodiment of the invention, the weight of the base load processing time in the same stage under the previous historical micro-batch is set to 0.1, and the weight of the unit load processing time is set to 0.9, that is, the sum of the weights of the two objects is a positive integer 1. By weighted summation, the information of the base load processing time in the same stage under the previous historical micro-batch can be effectively integrated with the unit load processing time in the current stage, so as to achieve dynamic adjustment of the performance baseline.

[0044] It should be noted that since the basic load processing time represents the time required to execute a unit load, the expected time for the corresponding stage can be obtained by directly multiplying the basic load processing time by the load of the historical micro-batch.

[0045] Preferably, in this embodiment of the invention, the performance deviation is the difference between the computation execution time and the expected processing time. That is, the performance deviation has positive and negative values; a positive value represents that the actual performance is lower than expected, and a negative value represents that the actual performance is higher than expected.

[0046] Step S3: Calculate the performance deviation of all stages under multiple consecutive historical micro-batches to obtain a multi-stage performance deviation matrix, and obtain the fluctuation pattern and fluctuation intensity of the historical micro-batches based on the multi-stage performance deviation matrix.

[0047] Since performance deviations across multiple GPU stages in a pipeline are often correlated due to shared resources, analyzing these deviations in isolation would fail to reveal this system-level coupling. Therefore, this embodiment of the invention further statistically analyzes the performance deviations of all stages across multiple consecutive historical micro-batches to obtain a multi-stage performance deviation matrix. That is, the size of this matrix is ​​P×M, where P is the number of stages and M is the number of consecutive historical micro-batches.

[0048] It should be noted that, since the embodiments of the present invention are intended to analyze pre-filled micro-batches, the first 100 consecutive historical micro-batches of the pre-filled micro-batches are selected as statistical objects to construct a multi-stage performance deviation matrix.

[0049] The multi-stage performance deviation matrix can describe the changes in performance deviation at the same stage in different historical micro-batches, and can also characterize the changes between all stages in different historical micro-batches. Therefore, the fluctuation pattern and fluctuation intensity of historical micro-batches can be obtained based on the multi-stage performance deviation matrix.

[0050] Preferably, in this embodiment of the invention, a matrix decomposition method is used to identify and separate the dominant cooperative performance fluctuation pattern reflected by the matrix, that is, to obtain the fluctuation pattern and fluctuation intensity of historical micro-batch, including:

[0051] Singular value decomposition is performed on the multi-stage performance deviation matrix to obtain the column vectors of the left singular vector matrix and the singular values ​​of the diagonal matrix; where the column vectors represent the different spatial distributions of performance deviation in P stages, and thus the vectors can be used as feature vectors for comparative analysis.

[0052] The column vector is matched with the template vector of all known fluctuation patterns, and the best matching known fluctuation pattern is taken as the fluctuation pattern of the historical micro-batch. The singular value is the fluctuation intensity.

[0053] In this embodiment of the invention, the known fluctuation patterns include front-end bottleneck patterns, end-end bottleneck patterns, and globally consistent patterns. For the front-end bottleneck pattern, a template vector can be constructed, in which the preceding values ​​are relatively obvious, such as [0.8 0.2 0 …… 0]; similarly, the template vector for the end-end bottleneck pattern can be [0 0 …… 0.2 0.8]; and the template vector for the globally consistent pattern can be [1 1 1 1 …… 1]. The specific construction of the template vector can be specifically set, and will not be elaborated here. In this embodiment of the invention, after normalizing the elements in the column vector, the cosine similarity between the normalized column vector and the template vector is calculated, and the known fluctuation pattern with the largest cosine pixel is selected as the fluctuation pattern of the historical micro-batch. The normalization method can adopt range standardization. The normalization method and the cosine similarity and other basic mathematical algorithms are technical means well known to those skilled in the art, and will not be elaborated here.

[0054] Step S4: Predict the predicted intensity of future micro-batches based on the volatility pattern and volatility intensity of historical micro-batches; obtain the risk index for each stage under each historical micro-batch based on the predicted intensity, the queue waiting time of the downstream stage, the calculation execution time of the downstream stage, and the performance deviation of each stage.

[0055] After determining the fluctuation pattern of the current inference system, for the previous historical micro-batch (i.e., the most recent historical micro-batch), a risk prediction method can be used to determine the performance evolution trend exhibited by that historical micro-batch. Therefore, embodiments of the present invention can predict the prediction intensity of future micro-batches based on specific fluctuation patterns and fluctuation intensities. That is, for the previous historical micro-batch, the future micro-batch is the pre-filled micro-batch.

[0056] The predicted intensity reflects the computational pressure that pre-filled micro-batches may experience during execution. A higher intensity indicates a greater risk to the system, potentially leading to local performance degradation and systemic threats in the future. Furthermore, for each stage, if the downstream stage has a long queue waiting time but a fast computation execution time, it indicates a large downstream buffer capacity that can offset some of the risk. Therefore, the risk index for each stage under each historical micro-batch can be obtained based on these characteristics.

[0057] Preferably, in this embodiment of the invention, the method for obtaining the predicted intensity includes:

[0058] The fluctuation intensities of the target historical micro-batch and the preceding historical micro-batch with the same fluctuation pattern are combined to form a fluctuation intensity sequence. This sequence is then processed using a Kalman filter to obtain the predicted intensity. It should be noted that the target historical micro-batch is the historical micro-batch preceding the pre-filled micro-batch, and the preceding historical micro-batch with the same fluctuation pattern can be found among the 100 historical micro-batches constructed from the multi-stage performance deviation matrix.

[0059] Preferably, in this embodiment of the invention, the method for obtaining the risk index includes:

[0060] The predicted intensity is used as the system threat exposure; the buffer capacity of the downstream stage is obtained based on the queue waiting time and the computation execution time of the downstream stage; the system threat exposure is weighted according to the performance deviation of each stage to obtain the weighted system threat exposure; and the risk index of each stage is obtained based on the weighted system threat exposure and the buffer capacity of the downstream stage.

[0061] As a specific example, the risk index in this embodiment of the invention is expressed by the formula:

[0062] ;in The risk index for the k-th stage of the k-th micro-batch. Let max[] be the system threat exposure of the (k+1)th micro-batch, and max[] be the maximum value filtering function. Let be the deviation of the k-th stage in the k-th micro-batch. This indicates that the maximum value is selected between 0 and the deviation amount. In other words, in this embodiment of the invention, only the case where the performance is lower than expected is considered. The lower the performance is than expected, the greater the weight of the system threat exposure, and the larger the denominator of the formula. This represents the buffer capacity of the (i+1)th stage of the kth micro-batch, i.e., the downstream stage. A smaller buffer capacity corresponds to a higher risk index for the i-th stage. The risk index deeply integrates system pattern diagnosis, future trend prediction, local real-time status, and downstream buffer capacity.

[0063] In this embodiment of the invention, the buffer capacity is the ratio of the queue waiting time in the downstream stage to the computation execution time.

[0064] Step S5: For the time to be allocated in the pre-filled micro-batch, simulate the processing time based on the basic load processing time of each stage in the most recent historical micro-batch, and perform time correction in combination with the risk index of each stage to determine the synchronization target time of the pre-filled micro-batch; based on the synchronization target time, provide feedback of control commands in the real-time processing stage of the pre-filled micro-batch.

[0065] Based on the processing results of the above steps, at the allocation time of the pre-filled micro-batch, a time simulation of the pre-filled micro-batch can be performed. The specific simulation parameters use the base load processing time of each stage in the most recent historical micro-batch, and time correction can be performed by combining the risk index of each stage to finally determine the synchronization target time of the pre-filled micro-batch. Then, control commands can be broadcast to all P pipeline stages' GPU work units based on the target synchronization time.

[0066] Preferably, in this embodiment of the invention, determining the target synchronization time of the pre-filled micro-batch includes:

[0067] The baseline execution time for each stage in the most recent historical micro-batch is obtained by multiplying the base load processing time of the pre-filled micro-batch by the base load. The risk index for each stage in the most recent historical micro-batch is normalized, and the normalized risk index is multiplied by the baseline execution time to obtain the time correction amount.

[0068] If the recent historical micro-batch fluctuation pattern belongs to a preset high-risk mode, then the sum of the time correction amount and the baseline execution time is used as the target execution time for each batch; otherwise, the baseline execution time is directly used as the target execution time. In this embodiment of the invention, the high-risk mode is set as the end bottleneck mode and the front bottleneck mode. These two modes indicate that the risk index of the front or end stage of the pipeline is significantly higher, and time correction is required based on the time correction amount.

[0069] The overall target execution time is obtained by summing the target execution times of all batches in the pre-filled micro-batch, and the synchronization target time is determined based on the overall target execution time.

[0070] In this embodiment of the invention, the control command is a timing command containing a precise timestamp, such as "at the time when the system clock reaches the synchronization target, atomically abort the current pre-filled execution context and activate the computational task in the decoding phase." The execution of this instruction can be achieved through a timed triggering mechanism in the GPU programming interface, ensuring that all distributed work units perform state switching at the same time, thereby eliminating transition delays caused by waiting and misjudgment.

[0071] In summary, this invention performs information statistics and feature extraction on historical micro-batches that have already been executed. For each batch, targeted analysis is conducted to determine the performance deviation between the expected processing time and the computation execution time. A multi-stage offset matrix is ​​constructed by combining multiple consecutive historical micro-batches, and the fluctuation pattern and intensity under historical micro-batches are determined based on the offset distribution in the matrix. A risk prediction method is used to determine the risk of the inference system's state relative to future micro-batches under each historical micro-batchens. For real-time pre-filled micro-batches, time simulation can be performed based on the basic information and risk information of the most recent historical micro-batches to determine the synchronization target time, thereby achieving accurate pipeline emptying prediction. This invention, through data extraction and feature analysis of historical micro-batches, determines the GPU's cooperative fluctuation pattern and then performs effective risk prediction, achieving effective execution time prediction for pre-filled micro-batches.

[0072] Based on the same inventive concept, this invention also proposes a cross-GPU parallel inference system for large language models, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the steps of the cross-GPU parallel inference method for large language models described above.

[0073] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0074] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A cross-GPU parallel inference method for large language models, characterized in that, The method includes: For each stage under the historical micro-batch, obtain the queue waiting time and computation execution time for each stage; The expected processing time is obtained based on the base load processing time of each stage and the load of historical micro-batches. The base load processing time is updated according to the load size of historical micro-batches and the computation execution time. The performance deviation between the expected processing time and the computation execution time is obtained. The performance deviation of all stages under multiple consecutive historical micro-batches is statistically analyzed to obtain a multi-stage performance deviation matrix. The fluctuation pattern and fluctuation intensity of the historical micro-batches are obtained based on the multi-stage performance deviation matrix. Based on the fluctuation pattern and fluctuation intensity, predict the predicted intensity of future micro-batches from historical micro-batches; based on the predicted intensity, the queue waiting time of the downstream stage, the calculation execution time of the downstream stage, and the performance deviation of each stage, obtain the risk index of each stage under each historical micro-batch. For the allocation time of the pre-filled micro-batch, the processing time is simulated using the basic load processing time of each stage in the most recent historical micro-batch, and time correction is performed by combining the risk index of each stage to determine the synchronization target time of the pre-filled micro-batch; based on the synchronization target time, control commands are fed back during the real-time processing stage of the pre-filled micro-batch; the method for updating the basic load processing time includes: For each stage, the ratio of the computation execution time to the load of the micro-batch is used as the unit load processing time. The unit load processing time and the basic load processing time of the same stage under the previous historical micro-batch are weighted and summed according to a preset weight to obtain the basic load processing time. The performance deviation is the difference between the calculation execution time and the expected processing time; The acquisition of historical micro-batch fluctuation patterns and fluctuation intensities includes: Singular value decomposition is performed on the multi-stage performance deviation matrix to obtain the column vectors of the left singular vector matrix and the singular values ​​of the diagonal matrix; the column vectors are matched with the template vectors of all known fluctuation patterns, and the best matching known fluctuation pattern is taken as the fluctuation pattern of the historical micro-batch, and the singular value is the fluctuation intensity. The determination of the synchronization target time for the pre-filled micro-batch includes: The baseline execution time for each stage in the most recent historical micro-batch is obtained by multiplying the base load processing time of the pre-filled micro-batch. The risk index for each stage in the most recent historical micro-batch is normalized, and the normalized risk index is multiplied by the baseline execution time to obtain the time correction amount. If the fluctuation pattern of the most recent historical micro-batch belongs to the preset high-risk pattern, then the sum of the time correction amount and the baseline execution time is used as the target execution time for each batch; otherwise, the baseline execution time is directly used as the target execution time. The overall target execution time is obtained by summing the target execution times of all batches in the pre-filled micro-batch, and the synchronization target time is determined based on the overall target execution time.

2. The cross-GPU parallel inference method for large language models according to claim 1, characterized in that, The known fluctuation patterns include front-end bottleneck pattern, end-end bottleneck pattern, and globally consistent pattern.

3. The cross-GPU parallel inference method for large language models according to claim 1, characterized in that, The method for obtaining the predicted intensity includes: The fluctuation intensity of the target historical micro-batch and the previous historical micro-batch with the same fluctuation pattern are combined into a fluctuation intensity sequence. The fluctuation intensity sequence is then processed using a Kalman filter to obtain the predicted intensity.

4. The cross-GPU parallel inference method for large language models according to claim 1, characterized in that, The methods for obtaining the risk index include: The predicted intensity is used as the system threat exposure; the buffer capacity of the downstream stage is obtained based on the queue waiting time and the computation execution time of the downstream stage; the system threat exposure is weighted according to the performance deviation of each stage to obtain the weighted system threat exposure; and the risk index of each stage is obtained based on the weighted system threat exposure and the buffer capacity of the downstream stage.

5. The cross-GPU parallel inference method for large language models according to claim 4, characterized in that, The buffer capacity is the ratio of the queue waiting time in the downstream stage to the computation execution time.

6. A cross-GPU parallel inference system for large language models, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the cross-GPU parallel inference method for large language models as described in any one of claims 1 to 5.

Images