Sharding scheduling method, apparatus, device, medium, and product

Abstract

This application provides a sharding scheduling method, apparatus, device, medium, and product. The method includes: acquiring the operator dimension features and hardware parameters of an operator to be executed; the hardware parameters at least include memory access alignment boundaries; under the constraints of the memory access alignment boundaries, determining the target sharding strategy for the operator to be executed for a target request based on the operator dimension features; and processing the target request using multiple graphics processors according to the target sharding strategy. This method aims to establish a collaborative relationship between operator form, load changes, and hardware constraints, thereby improving the adaptability and execution coordination of sharding scheduling in distributed inference processes.

Description

Technical Field

[0001] This application relates to the field of distributed reasoning technology for artificial intelligence models, and in particular to a sharding scheduling method, apparatus, device, medium, and product. Background Technology

[0002] Current distributed inference for large-scale language models typically employs tensor parallelism to distribute model weights across multiple graphics processing units (GPUs), enabling multiple GPUs to collaboratively complete linear operator computations and result aggregation.

[0003] However, existing solutions mostly adopt a static block method with a fixed number of fragments. During the row-parallel stage, the results need to be aggregated after the local matrix calculation, which can easily create gaps in computation waiting for communication. Moreover, this method is difficult to adapt to different operator dimensions and batch and sequence length changes, which will limit the memory access efficiency of operators, communication masking effect and inference stability.

[0004] Therefore, how to balance different operators and load changes in distributed inference, improve the efficiency of computation and communication collaboration, and maintain execution stability has become a technical problem that needs to be solved. Summary of the Invention

[0005] This application provides a sharding scheduling method, apparatus, device, medium, and product to solve the aforementioned technical problems. This solution addresses scenarios where different operator dimensions and hardware execution conditions differ during distributed inference. It adaptively determines a matching sharding scheduling strategy for the target request (the operator to be executed) and drives operator execution accordingly. This establishes a collaborative relationship between operator form, load changes, and hardware constraints, improving the adaptability and execution coordination of sharding scheduling during distributed inference.

[0006] In a first aspect, embodiments of this application provide a fragmented scheduling method, including:

[0007] Obtain the operator dimension features and hardware parameters of the operator to be executed; the hardware parameters include at least the memory access alignment boundary.

[0008] Under the constraint of memory access alignment boundary, the target sharding strategy of the operator to be executed for the target request is determined based on the operator dimension features;

[0009] According to the target fragmentation strategy, multiple graphics processors are used to process the target request.

[0010] In one possible embodiment, under the constraint of memory access alignment boundaries, the target sharding strategy for the target request of the operator to be executed is determined based on operator dimensional features, including:

[0011] Based on the operator dimension features and the dynamic features of the target request, the request type of the target request is determined;

[0012] If the request type of the target request is the target type, the overlapping evaluation result of each shard in different candidate sharding strategies is determined based on dynamic features;

[0013] Based on the memory access alignment boundary, the candidate fragmentation strategy is obtained by verifying the overlap evaluation results to indicate that the computational cost of fragmentation covers the communication cost, and the target fragmentation strategy is obtained.

[0014] In one possible embodiment, the request type of the target request is determined based on the operator dimension features and the dynamic features of the target request, including:

[0015] Based on the operator dimensionality and dynamic characteristics, the computing power density of the operator to be executed for the target request is calculated;

[0016] The request type of the target request is determined based on the computing power density and the pre-configured density threshold;

[0017] Specifically, if the computing power density is greater than the density threshold, the request type is the first type; otherwise, the request type is the second type. The first type indicates that the request traffic is greater than the second type.

[0018] In one possible embodiment, the target type is a first type, and based on dynamic features, the overlap evaluation result of each fragment in different candidate fragmentation strategies is determined, including:

[0019] Based on dynamic characteristics, the computational and communication costs corresponding to each fragment in the candidate fragmentation strategy are determined.

[0020] Based on the computational load and the communication load, determine the ratio of the delay corresponding to the computational load to the delay corresponding to the communication load.

[0021] Based on the ratio and a pre-configured ratio threshold, the overlap evaluation result of the operator to be executed is determined;

[0022] Specifically, if the ratio reaches the ratio threshold, the overlap evaluation result is that the computational workload of the current fragment covers the communication workload; otherwise, the overlap evaluation result is that the computational workload of the current fragment cannot cover the communication workload.

[0023] In one possible embodiment, based on memory access alignment boundaries, the candidate sharding strategy, which indicates that the computational cost of sharding covers the communication cost, is verified according to the overlap evaluation results, to obtain the target sharding strategy, including:

[0024] For candidate fragmentation strategies whose computational complexity covers communication volume as indicated by the overlap evaluation results, the candidate fragmentation strategy that reaches the memory access alignment boundary of the fragmentation byte tensor is taken as the target fragmentation strategy.

[0025] In one possible embodiment, the method further includes:

[0026] If the byte tensors of fragments with multiple candidate fragmentation strategies reach the memory access alignment boundary, the target fragmentation strategy is determined based on the predicted processing latency of the target request corresponding to different candidate fragmentation strategies.

[0027] In one possible embodiment, the method further includes:

[0028] If there is no candidate sharding strategy whose byte tensor reaches the memory access alignment boundary, execute the sharding rollback strategy to iteratively reduce the number of shards indicated by the candidate sharding strategy until the byte tensor of the shards reaches the memory access alignment boundary, and obtain the target sharding strategy.

[0029] In one possible embodiment, before processing the target request using multiple graphics processors according to the target fragmentation strategy, the method further includes:

[0030] The operator to be executed is parsed to obtain the execution topology graph corresponding to the operator; the execution topology graph contains multiple dependent nodes and dependency edges connecting the multiple dependent nodes;

[0031] Based on multiple dependent nodes, construct computation branches and communication branches; each computation branch and communication branch contains multiple dependent nodes.

[0032] Construct cross-flow dependency edges between the nth dependent node of the computation branch and the (n-1)th dependent node of the communication branch to obtain the deconstructed topology graph; where n is an integer greater than or equal to 1.

[0033] In one possible embodiment, after parsing the operator to be executed and obtaining the execution topology graph corresponding to the operator to be executed, the process further includes:

[0034] If there is a directed cycle formed by multiple dependent nodes and multiple dependent edges in the execution topology graph, traverse the dependent edges in the directed cycle, and remove the dependent edge if the output of the upstream dependent node of the dependent edge is not used as the input of the downstream dependent node.

[0035] In one possible embodiment, according to the target fragmentation strategy, multiple graphics processors are used to process the target request, including:

[0036] According to the target fragmentation strategy, multiple graphics processors are used to process the target request in multiple stages according to the deconstructed topology graph; the multiple stages include at least the Prefill stage and the Decode stage.

[0037] In one possible embodiment, according to the target fragmentation strategy, multiple graphics processors are used to process the target request in multiple stages according to the deconstructed topology graph, including:

[0038] Following the sequence of multiple stages, the topology graph is modified based on the preset topology graph corresponding to the first stage; the modifications include at least one of node changes, edge changes, and parameter changes.

[0039] The graphics processor is controlled to use the modified structure map for the first stage of processing;

[0040] Continuing in the order of arrangement, based on the preset topology graph corresponding to the m-th stage, the modified deconstruction topology graph used in the (m-1)-th stage is changed again... until the processing of multiple stages; where m is an integer greater than or equal to 1.

[0041] Secondly, embodiments of this application provide a segmentation scheduling device, comprising:

[0042] The acquisition module is used to acquire the operator dimension features and hardware parameters of the operator to be executed; the hardware parameters include at least the memory access alignment boundary;

[0043] The determination module is used to determine the target sharding strategy of the operator to be executed for the target request based on the operator dimension features under the constraint of memory access alignment boundary.

[0044] The processing module is used to process target requests using multiple graphics processors according to the target fragmentation strategy.

[0045] Thirdly, embodiments of this application provide a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method provided above.

[0046] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method provided above.

[0047] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method provided above.

[0048] The sharding scheduling method, apparatus, device, medium, and product provided in this application obtain the operator dimension features of the operator to be executed and hardware parameters including at least the memory access alignment boundary, and determine the target sharding strategy of the operator to be executed for the target request based on the operator dimension features under the constraint of the memory access alignment boundary. This enables the sharding method to be better adapted to the operator dimension, request load, and hardware memory access requirements, thereby improving memory access efficiency and operator execution stability, and improving the collaborative effect of computing and communication. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] Figure 1 A schematic diagram comparing the ideal total time and actual total time of computation and communication in a distributed inference system provided in this application under different numbers of shards;

[0051] Figure 2 A flowchart illustrating a segmentation scheduling method provided in this application;

[0052] Figure 3 A schematic diagram of the decision-making process for determining the target fragmentation strategy in a fragmentation scheduling method provided in this application;

[0053] Figure 4 This application provides a schematic diagram of the structure of a segmented scheduling device;

[0054] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application.

[0055] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0056] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0057] Distributed inference for large-scale language models falls under the fields of parallel and intelligent computing, and is commonly used in scenarios involving ultra-long text understanding, long context generation, and concurrent processing of multiple requests. These scenarios typically rely on inference systems composed of multiple GPUs, distributing model weights across different computing devices using tensor parallelism to collaboratively complete the computation of linear operators and the aggregation of results.

[0058] In existing distributed inference schemes, linear operators are typically statically divided into blocks with a fixed number of shards, and the results are aggregated after each GPU completes its local matrix calculation, thereby achieving collaborative inference in the row-parallel stage. The basic idea of ​​this type of scheme is to pre-define a unified sharding method, so that different operators perform block calculations and subsequent communication processing under the same rules.

[0059] However, static partitioning with a fixed number of fragments is difficult to adapt to the dimensional differences of different operators, and it is also difficult to adapt to dynamic changes in load conditions such as batch size and sequence length. When the fragments are too fine, the local computation scale may approach or even fall below the hardware memory access alignment requirements, which will lead to a decrease in the memory access efficiency of the operators; when the fragments are insufficient, it is difficult to take into account the processing overhead required for subsequent result aggregation, which will also limit the overall collaborative efficiency.

[0060] As an example, such as Figure 1 As shown, without sharding, the Chunk Num is 1, and the ideal total time for GPU to perform computation and communication is 2T. If there are 2 shards and the Chunk Num is 2, the overlap between the GPU's computation time and the communication time of another GPU (i.e., another shard) increases (by default, different shards correspond to different GPUs). The dimension and time of a single-shard computation task continuously shrink. In this case, the ideal total time for two GPUs to perform computation and communication is 1.5T, while the actual total time is 1.75T, saving 0.25T of time compared to without sharding.

[0061] However, if the number of shards is increased to 4 and the chunk number is 4, the actual total time will rebound from 1.75T when the number of shards is 2 to 2.25T because the increase in the time spent by the GPU to perform computation exceeds the amount of time saved by the overlap between computation and communication. The actual time spent by the four GPUs to perform computation and communication exceeds the actual total time spent without sharding.

[0062] Furthermore, under long sequences or high-concurrency requests, the differences in the scale of input and output features of different operators will amplify the above problems, causing a decrease in the processing efficiency of some operators, an increase in the overall inference latency fluctuation, and affecting the stability of the distributed inference process.

[0063] Therefore, choosing a more suitable sharding method under different operators and different load conditions has become an important issue in improving the performance of distributed inference.

[0064] Therefore, how to determine a more suitable sharding method based on the actual characteristics of the operator to be executed and the hardware constraints is a technical problem that urgently needs to be solved.

[0065] To address the aforementioned issues, a sharding scheduling method is provided. When a target request arrives, the operator dimension features and hardware parameters of the operator to be executed are first obtained. Under the constraint of memory access alignment boundaries, the target sharding strategy corresponding to the operator is determined based on the operator dimension features. Then, the operator to be executed is controlled to process the target request according to the target sharding strategy, thereby improving the processing adaptability and execution stability under different operators and different load scenarios. This method can be deployed in a distributed inference system composed of multiple GPU cards (i.e., multiple GPU devices) that collaborate through high-speed interconnection.

[0066] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0067] In one embodiment, a sharding scheduling method is provided. This embodiment uses the application of this sharding scheduling method to the aforementioned distributed inference system as an example for illustration. Figure 2 As shown, the fragmentation scheduling method includes:

[0068] Step 202: Obtain the operator dimension features and hardware parameters of the operator to be executed; the hardware parameters include at least the memory access alignment boundary.

[0069] In this embodiment, after the target request enters the inference queue, the operator dimension features are first extracted from the input tensor description information, intermediate activation tensor description information, or model structure metadata corresponding to the operator to be executed.

[0070] Operator dimension features refer to the set of static attribute parameters inherent to the operator to be executed, including fixed features such as operator channel dimension, sequence length, batch processing scale, computational complexity, tensor size, etc., which determine the computational power requirements and shardability of the operator itself.

[0071] Hardware parameters describe the inherent configuration parameters of a GPU device. Hardware parameters are the hard constraints that enable the GPU device to perform operator operations and access data memory.

[0072] In this application, the hardware parameters include at least the memory access alignment boundary. The memory access alignment boundary refers to the smallest compliant byte unit for hardware memory read and write operations of the GPU device. The GPU device can efficiently read and write data only when the byte size of the tensor fragment is an integer multiple of this memory access alignment boundary; unaligned fragments will incur zero-padding overhead, bandwidth waste, and performance degradation.

[0073] Memory access alignment boundaries are used to define the minimum alignment conditions that data access after fragmentation must meet. Specifically, memory access alignment boundaries can be expressed as alignment thresholds in bytes, such as 16 bytes, 32 bytes, 64 bytes, or 128 bytes.

[0074] Step 204: Under the constraint of memory access alignment boundary, determine the target sharding strategy of the operator to be executed for the target request based on the operator dimension features.

[0075] The target fragmentation strategy is used to indicate the fragmentation method that the operator to be executed will ultimately adopt, including parameters such as the number of valid fragments, tensor splitting rules, and single fragment size.

[0076] In this embodiment, candidate sharding strategies can be generated based on operator dimension features. For example, the minimum number of shards can be 1, and the maximum number of shards can be the total number of GPU devices, resulting in multiple candidate sharding strategies.

[0077] The target request is used to represent a model inference service request initiated by a user, carrying complete input tensor data.

[0078] As an example, the local computational overhead, data transfer overhead, or overall execution overhead of candidate solutions can be estimated, and the candidate sharding strategy with the better execution cost can be selected as the target sharding strategy.

[0079] Step 206: According to the target fragmentation strategy, use multiple graphics processors to process the target request.

[0080] In this embodiment, after determining the target sharding strategy, the number of shards indicated by the target sharding strategy can be used to map each shard to the corresponding GPU device, so that each GPU device can perform local computation. Then, after the local computation is completed, the result aggregation is triggered.

[0081] Based on the above analysis, this embodiment enables the fragmentation judgment result to be directly applied to the reasoning process of the target request, and keeps the access method after fragmentation consistent with the hardware parameters.

[0082] The aforementioned sharding scheduling method incorporates operator dimensional features and memory access alignment boundaries into the sharding decision-making process. During execution, it organizes local computation and result aggregation according to a predetermined number of shards, shard dimensions, and device mapping relationships, enabling different operators to employ corresponding sharding methods under varying dimensional scales and hardware constraints. This approach suppresses execution fluctuations caused by fixed sharding rules under different operators or target request conditions, maintaining processing adaptability and execution stability in distributed inference.

[0083] In some alternative embodiments, step 204 includes:

[0084] Based on the operator dimension features and the dynamic features of the target request, the request type of the target request is determined;

[0085] If the request type of the target request is the target type, the overlapping evaluation result of each shard in different candidate sharding strategies is determined based on dynamic features;

[0086] Based on the memory access alignment boundary, the candidate fragmentation strategy is obtained by verifying the overlap evaluation results to indicate that the computational cost of fragmentation covers the communication cost, and the target fragmentation strategy is obtained.

[0087] In this embodiment, after receiving a target request, the dynamic characteristics of the target request are first jointly matched with the operator dimension characteristics of the operator to be executed to determine whether the current target request belongs to the target type.

[0088] If the determination result is a target type, then for the preset multiple candidate sharding strategies, the computational and communication volume corresponding to each shard is calculated based on the dynamic characteristics of the target request, and an overlap evaluation result is generated accordingly.

[0089] The overlap evaluation result can be expressed as whether each fragment satisfies the condition that the computation delay is greater than or equal to the communication delay, or whether it satisfies the ratio threshold condition obtained by converting computational quantity to communication quantity.

[0090] After obtaining the overlap evaluation results, each candidate fragmentation strategy is further verified based on the memory access alignment boundary to determine whether there are any candidate fragmentation strategies that can satisfy the requirement that computational complexity covers communication complexity and that the fragment tensor size meets the alignment constraints. Specifically, for candidate fragmentation strategies whose fragmented tensor length, number of bytes, or number of elements does not meet the memory access alignment boundary, even if their overlap evaluation results satisfy the requirement that computational complexity covers communication complexity, they will not be output as the target fragmentation strategy.

[0091] The above-described sharding scheduling method makes the selection of target sharding strategy no longer dependent on a fixed number of shards, but rather based on operator dimension features and dynamic features for targeted screening. Furthermore, it performs alignment verification on candidate strategies that meet the conditions of computation coverage and communication, making the final sharding result more in line with the execution load of the current target request and maintaining consistency in memory access granularity, thereby improving the execution stability and adaptability in distributed inference.

[0092] In some optional embodiments, the step of determining the request type of the target request based on operator dimension features and the dynamic features of the target request includes:

[0093] Based on the operator dimensionality and dynamic characteristics, the computing power density of the operator to be executed for the target request is calculated;

[0094] The request type of the target request is determined based on the computing power density and the pre-configured density threshold;

[0095] Specifically, if the computing power density is greater than the density threshold, the request type is the first type; otherwise, the request type is the second type. The first type indicates that the request traffic is greater than the second type.

[0096] Dynamic features are used to indicate dynamic parameters such as the amount of data requested by the target request, the number of concurrent requests, the frequency of real-time calls, the priority of request execution, the number of operator calls required for a single request, and the scale of real-time data processing.

[0097] Computational density indicates the total amount of computing resources required by an operator to execute for a given target request, per unit of processing dimension, per unit of time, or per unit of data volume. Computational density is used to quantify the intensity of a target request's consumption of system computing power.

[0098] The first type refers to high-load, high-volume request types, which means that the current target request corresponds to a high intensity of operator computing power consumption, a large overall request traffic scale, and a large resource consumption, putting greater pressure on the system's computing power and operating load.

[0099] The second type refers to low-load, low-volume request types, which means that the current target request has low operator computing power consumption intensity, small overall request traffic scale, low resource consumption, and less pressure on system computing power and operating load.

[0100] In this embodiment, the computing power density value can be obtained, for example, through dimension product, dimension ratio, or weighted combination. Specifically, it can be normalized according to the operator dimension scale and dynamic load changes, so that features of different dimensions can be compared under the same scale.

[0101] When the computing power density is greater than the density threshold, the target request can be identified as the first type; when the computing power density is not greater than the density threshold, it is identified as the second type.

[0102] The above determination results can be used as input for subsequent sharding strategy filtering, enabling the system to adopt different strategy branches for different request types.

[0103] In this embodiment, after obtaining the target request, the complete operator dimension features of all operators to be executed corresponding to the current target request can be obtained first. Fixed attribute data such as the operator's preset computational complexity, baseline computing power consumption, and resource usage specifications can be retrieved. All dynamic features of the current target request can be collected, including real-time running data such as the data processing volume of this request, the number of concurrent calls, and the real-time request frequency.

[0104] Subsequently, the collected static feature data of the operator dimension and the dynamic feature data of the target request are correlated, matched, and fused. During the calculation, the fixed computing power consumption benchmark of the operator to be executed is used as the base, and the dynamic scale parameter of the target request is combined for weighted correction to offset the shortcomings of a single static parameter that cannot adapt to the real-time request scenario. Finally, the actual computing power consumption intensity of the operator to be executed for this specific target request is accurately calculated, that is, the computing power density.

[0105] Furthermore, the computing power density is compared with the pre-configured density threshold value. When the real-time calculated computing power density is greater than the density threshold, it can be determined that the resource consumption intensity and traffic scale of the current target request exceed the system's normal standards, and the corresponding request type is the first type.

[0106] When the real-time calculated computing power density value is less than or equal to the pre-configured density threshold, it is determined that the resource consumption intensity and traffic scale of the current target request are within the normal carrying capacity of the system, and the corresponding request type is the second type.

[0107] The aforementioned sharding scheduling method uses both operator dimensional features and the dynamic features of the target request for computing power density calculation, and classifies request types based on density thresholds. This ensures that the request type matches the current target request load, providing a stable basis for subsequent sharding strategy selection. Since the first and second types correspond to different request traffic, the scheduling process can differentiate between different load intensities, thereby improving the matching degree between the sharding strategy and the target request state, and reducing adaptation deviations caused by static partitioning.

[0108] In some optional embodiments, the target type is a first type, and the step of determining the overlap evaluation result of each fragment in different candidate fragmentation strategies based on dynamic features includes:

[0109] Based on dynamic characteristics, the computational and communication costs corresponding to each fragment in the candidate fragmentation strategy are determined.

[0110] Based on the computational load and the communication load, determine the ratio of the delay corresponding to the computational load to the delay corresponding to the communication load.

[0111] Based on the ratio and a pre-configured ratio threshold, the overlap evaluation result of the operator to be executed is determined;

[0112] Specifically, if the ratio reaches the ratio threshold, the overlap evaluation result is that the computational cost of the fragments covers the communication cost; otherwise, the overlap evaluation result is that the computational cost of the fragments cannot cover the communication cost.

[0113] In this embodiment, for each shard unit, the computational and communication volumes of the shard can be calculated based on its local computational dimension and subsequent communication boundaries. Then, based on the computing power and communication parameters of the corresponding GPU device, two types of latency are obtained, and their ratio is calculated.

[0114] When the ratio is not less than the ratio threshold, it is determined that the computational cost of the fragment can cover its communication cost, indicating that the fragment can hide some communication overhead during the computation process. When the ratio is less than the ratio threshold, it is determined that the computational cost of the fragment cannot cover its communication cost. Furthermore, based on this, the overlap evaluation results for each fragment under each candidate fragmentation strategy are output, and the candidate fragmentation strategies where computational cost covers communication cost are used as subsequent verification objects.

[0115] The aforementioned sharding scheduling method quantifies the sharding overlap capability of different candidate sharding strategies by considering the dynamic load changes of the first type of target request. This ensures that the selection of sharding strategies is consistent with the actual computation and communication characteristics, thereby making the target sharding strategy more suitable for the execution characteristics of the current target request and improving the execution stability of the operator in distributed inference.

[0116] In some optional embodiments, the step of verifying the overlap evaluation results indicating that the computational cost of fragmentation covers the communication cost of candidate fragmentation strategies, based on memory access alignment boundaries, to obtain the target fragmentation strategy includes:

[0117] For candidate fragmentation strategies whose computational complexity covers communication volume as indicated by the overlap evaluation results, the candidate fragmentation strategy that reaches the memory access alignment boundary of the fragmentation byte tensor is taken as the target fragmentation strategy.

[0118] In this embodiment, for candidate sharding strategies where computational complexity exceeds communication throughput, memory access alignment boundaries are further verified. The size of the shard tensor corresponding to each candidate sharding strategy is converted into bytes and compared modulo-based with a preset alignment granularity. If the number of bytes is divisible by the memory access alignment boundary, or if the sharding relationship of the current sharding strategy is maintained after padding, the candidate sharding strategy is considered to have reached the memory access alignment boundary. Subsequently, the candidate sharding strategy can be written into the set of target sharding strategies and output as the final target sharding strategy for the operator to be executed.

[0119] The aforementioned fragmentation scheduling method, assuming the overlap evaluation results meet the requirements, further constrains candidate fragmentation strategies with memory access alignment boundaries. This ensures that the final output fragmentation strategy simultaneously satisfies computation coverage and hardware access alignment, thereby maintaining consistency between local computation and memory access after fragmentation and reducing access degradation caused by unaligned fragmentation. This allows the target fragmentation strategy to better align with the current operator load and hardware characteristics, improving execution stability in distributed inference.

[0120] In some optional embodiments, the fragmentation scheduling method further includes:

[0121] If the byte tensors of fragments with multiple candidate fragmentation strategies reach the memory access alignment boundary, the target fragmentation strategy is determined based on the predicted processing latency of the target request corresponding to different candidate fragmentation strategies.

[0122] In this embodiment, when multiple candidate sharding strategies all satisfy the condition that the byte tensor of the sharding reaches the memory access alignment boundary, these candidate sharding strategies can all be used as comparable objects, and the latency of each candidate sharding strategy can be estimated by combining the operator dimension characteristics, communication overhead characteristics, and parallel computing overhead characteristics of the target request.

[0123] This latency estimation can be accomplished through historical operational statistics, cost models, or pre-configured latency prediction models, outputting the predicted processing latency value corresponding to each candidate sharding strategy.

[0124] Subsequently, the latency of multiple prediction processes can be compared, and the candidate sharding strategy with the minimum latency can be determined as the target sharding strategy.

[0125] Since multiple candidate sharding strategies have passed memory access alignment boundary verification, the focus of the comparison here shifts to the difference in processing time of different strategies under the target request, and the most suitable target sharding strategy is determined accordingly.

[0126] The above-mentioned sharding scheduling method, when multiple candidate sharding strategies all meet the memory access alignment boundary, can further filter based on the predicted processing latency, so that the final selected target sharding strategy matches the actual processing load of the target request. Thus, the sharding decision is no longer limited to the alignment condition itself, but is finally determined by combining the latency estimation result.

[0127] In some optional embodiments, the fragmentation scheduling method further includes:

[0128] If there is no candidate sharding strategy whose byte tensor reaches the memory access alignment boundary, execute the sharding rollback strategy to iteratively reduce the number of shards indicated by the candidate sharding strategy until the byte tensor of the shards reaches the memory access alignment boundary, and obtain the target sharding strategy.

[0129] The fragment rollback strategy refers to an iterative correction rule that actively reduces the number of fragments and increases the byte capacity of a single fragment sub-tensor when the candidate fragment strategy cannot meet the memory access alignment boundary requirements. As an example, the fragment rollback strategy can adopt a halving reduction logic, that is, in each iteration, the number of fragments K is updated to half of the original number of fragments.

[0130] In this embodiment, an initial maximum candidate sharding strategy can be generated first based on the tensor dimensions of the current request (BS, S, Cin, Cout), the operator dimension features, and the real-time dynamic load features, and the initial number of candidate shards (K) can be determined. At the same time, hardware parameters, namely the memory access alignment boundary, are read as a unified judgment standard for subsequent verification and judgment.

[0131] Subsequently, based on the number of fragments K carried by the current candidate fragmentation strategy, the complete byte tensor is equally divided into K sub-fragment byte tensors, and the total memory byte length corresponding to a single sub-fragment is calculated. The byte length of a single fragment is then checked against the pre-stored memory access alignment boundary to determine whether the current fragment sub-tensor byte block can exactly match the hardware memory access alignment boundary.

[0132] If the verification result shows that the byte tensor of the current fragment fully reaches the memory access alignment boundary, there is no need to enter the rollback process, and the candidate fragmentation strategy is directly determined as the target fragmentation strategy; if the verification determines that there is no set of fragment byte tensors based on the current fragment number that can match the memory access alignment boundary, the fragmentation rollback strategy is immediately started, and the iterative correction process is entered.

[0133] The sharding rollback strategy adopts an iterative logic of halving reduction. The operation of a single iteration is as follows: the number of candidate shards K that do not meet the alignment constraints in this round is halved to generate a new number of candidate shards K=K / 2, forming a new candidate sharding strategy to be verified.

[0134] After reducing the number of shards, immediately recalculate the length of the byte tensor corresponding to each sub-shard under the new sharding scheme based on the updated number of shards, and perform memory access alignment boundary matching verification again to complete one round of iteration.

[0135] If the total length of the fragment byte tensor obtained after a certain round of candidate fragmentation strategy is divisible by the hardware memory access alignment boundary, that is, the fragment byte tensor reaches the memory access alignment boundary; when the termination condition is met, the fragmentation rollback iteration operation is stopped immediately.

[0136] The aforementioned partitioned scheduling method can retain the maximum number of parallel partitions while meeting hardware specifications, maximizing the throughput of parallel computing, fully exploring the upper limit of hardware computing power, and avoiding idle parallel resources caused by directly selecting small partitions. Secondly, the binary iterative backoff converges quickly. Compared with sequential decrementing verification, it significantly reduces the number of memory access alignment verification loops, reduces the judgment and calculation overhead in the scheduling phase, and shortens the scheduling time before instruction issuance.

[0137] like Figure 3As shown, in this embodiment, after obtaining the target request, the dynamic features corresponding to the current target request can be obtained first, including batch size (BS), sequence length (SeqLen), number of input channels (Channel In, Cin), and number of output channels (Channel In, Cin). The system uses BS, SeqLen, Cin, Cout, and the operator dimension features of the operator to be executed to perform a first-level load scale prediction. If the target request is of the second type, no sharding is performed, and the GPU device processes the target request directly in serial mode. If the target request is of the first type, the system further evaluates the overlap of each shard in different candidate sharding strategies to perform a second-level overlap strategy and alignment constraint. If there is a candidate sharding strategy whose overlap evaluation result is that the computational amount of the sharding covers the communication amount and the byte tensor of the sharding reaches the memory access alignment boundary, then the target sharding strategy is obtained. If there is no overlap evaluation result that is that the computational amount of the sharding covers the communication amount and / or the byte tensor of the sharding does not reach the memory access alignment boundary, then the system further performs a third-level optimization and execution to perform sharding rollback. In each round, the number of shards in the candidate sharding strategies is reduced by half to iteratively obtain the final target sharding strategy.

[0138] In some optional embodiments, prior to step 206, the following steps are also included:

[0139] The operator to be executed is parsed to obtain the execution topology graph corresponding to the operator; the execution topology graph contains multiple dependent nodes and dependency edges connecting the multiple dependent nodes;

[0140] Based on multiple dependent nodes, construct computation branches and communication branches; each computation branch and communication branch contains multiple dependent nodes.

[0141] Construct cross-flow dependency edges between the nth dependent node of the computation branch and the (n-1)th dependent node of the communication branch to obtain the deconstructed topology graph; where n is an integer greater than or equal to 1.

[0142] Dependency nodes are used to indicate a single independent operation within the operator to be executed, covering various basic tasks such as parameter reading, tensor computation, data writing, and sharding synchronization. Each node has independent execution attributes and has fixed execution dependencies.

[0143] Dependency edges are used to indicate logical connection lines linking different dependent nodes, representing data and temporal dependencies between dependent nodes. Dependency edges are used to constrain the execution order of various independent operations within the operator to be executed. Only after the upstream dependent node has completed execution and output valid data can the downstream dependent node be triggered to execute.

[0144] The computation branch is a task chain that is extracted and constructed from the original execution topology graph. It brings together all core computing power dependent nodes and is specifically responsible for core computing power tasks such as tensor operations, feature transformations, and matrix calculations.

[0145] The communication branch is also a task link built by separating it from the original execution topology. It gathers all data interaction-related dependent nodes, does not participate in any core computing power calculation, and is specifically responsible for auxiliary transmission tasks such as sharded data synchronization, cross-sharded data transfer, memory data reading and writing, and device communication interaction.

[0146] Cross-stream dependency edges refer to a new type of logical dependency relationship across computing branches and communication branches. They are different from ordinary dependency edges within branches and are used to connect the timing constraints and data linkage between two independent branches, enabling the interleaved and collaborative execution of computing tasks and communication tasks.

[0147] In this embodiment, by performing a deep traversal of all the internal execution logic, operation process, and data interaction links of the operator to be executed, all the fragmented execution tasks inside the operator to be executed can be decomposed into independent dependency nodes one by one.

[0148] Simultaneously, based on the data transmission logic and execution order of the task, all dependent nodes are sequentially connected through dependency edges, thereby completely replicating the original execution logic, task dependencies, and data flow path of the operator to be executed, forming a complete, accurate, and comprehensive original execution topology.

[0149] Furthermore, by classifying, filtering, and aggregating all dependent nodes, a precise division is achieved based on two core task types: computation and communication. All computational nodes responsible for core tensor computation, feature processing, and parameter calculation are chained together according to their original execution order to form a continuous and complete computation branch. All communication nodes responsible for data transmission, shard synchronization, and memory interaction are chained together according to their execution sequence to form an independent communication branch. Each branch possesses a complete node sequence, with clear functional boundaries, and is initially independent and unconstrained by one another.

[0150] Then, using the node order number as the matching rule, any positive integer n greater than or equal to 1 is selected. The dependent node ranked nth in the computation branch is then linked across links with the dependent node ranked n minus one in the communication branch, actively building a cross-flow dependency edge. This cross-flow dependency edge establishes timing constraints between the computation and communication branches, allowing the preceding tasks of the communication branch to connect with the following tasks of the computation branch in advance, breaking the completely independent operation of the two branches.

[0151] The aforementioned sharding scheduling method splits the original dependencies of the operators to be executed into computational and communication branches, and reorganizes the execution order through cross-flow dependency edges. This allows subsequent processing of target requests according to the target sharding strategy to directly complete collaborative execution based on the deconstructed topology graph, thereby achieving decoupling and isolation between computational power and data transmission tasks, breaking free from the execution constraints of a single serial link. By leveraging the cross-flow dependency construction between the nth computation node and the (n-1)th communication node, a pipelined interleaved execution logic can be formed, allowing the preceding communication tasks to preload data in advance, providing data support for subsequent computational tasks and completely avoiding the resource waste problem of computational tasks waiting for data and communication tasks running idly. This optimization method fully aligns with hardware memory access and sharding scheduling rules, maximizing the computational power advantages of the sharding parallel strategy and improving the overall execution throughput and running efficiency of the operators. Simultaneously, the structured topology reconstruction logic rules are unified and highly stable, adaptable to tensor requests of different dimensions and various sharding parameters, effectively reducing system scheduling overhead and ensuring the high efficiency and stability of the system operation in high-traffic inference scenarios.

[0152] In some optional embodiments, after parsing the operator to be executed and obtaining the execution topology graph corresponding to the operator to be executed, the method further includes:

[0153] If there is a directed cycle formed by multiple dependent nodes and multiple dependent edges in the execution topology graph, traverse the dependent edges in the directed cycle, and remove the dependent edge if the output of the upstream dependent node of the dependent edge is not used as the input of the downstream dependent node.

[0154] A directed cycle is a closed loop structure in an execution topology graph, formed by connecting multiple dependent nodes and directed dependency edges sequentially. Directed cycles can cause deadlocks in task execution logic, preventing dependent nodes from executing in the correct order. They are abnormal structures that need to be removed from the execution topology graph.

[0155] In this embodiment, after obtaining the execution topology graph, a global traversal test can be performed on the generated original execution topology graph to determine whether there is a directed cycle structure formed by multiple dependent nodes and multiple dependent edges within the execution topology graph. Specifically, during the compilation and parsing process, the original topology of the operator to be executed is prone to generating false closed directed cycles due to nested operator logic and repeated dependency bindings. Such structures will directly block topology task scheduling, causing nodes to fail to start execution normally; therefore, they must be detected and identified in advance.

[0156] If a directed cycle is confirmed to exist in the topology graph, all dependent edges contained in the directed cycle can be traversed one by one, and the data association verification can be performed on the upstream and downstream nodes corresponding to each dependent edge. The steps of data association verification include: determining whether the actual output data of the upstream dependent node corresponding to the dependent edge actually participates in the computational input of the downstream dependent node, and whether there is actual data flow support.

[0157] If a dependency edge is found to exist only in logical binding, and the output data of the upstream node is not actually used as the input content of the downstream node, this dependency edge is a spurious and redundant dependency edge without data support. It has no actual constraint value and is only used to incorrectly form a directed cycle. In this case, the invalid dependency edge can be removed directly, thereby breaking the abnormal directed cycle in the execution topology graph and eliminating the logical deadlock problem.

[0158] Valid dependency edges with real data dependencies are retained, resulting in a standardized execution topology graph that is acyclic, logically accurate, and normally schedulable, providing a compliant topology foundation for subsequent branch construction and cross-stream dependency setup.

[0159] The aforementioned sharding scheduling method, by verifying the true data dependencies between upstream and downstream dependent nodes, accurately dismantles spurious dependency edges that only have logical bindings but no data flow. This can break the abnormal closed-loop structure of the topology without disrupting valid computational logic and normal data dependencies, thus eliminating scheduling deadlock problems at their root. This sharding scheduling method can accurately distinguish between valid and redundant dependency edges, thereby precisely preserving the node constraints required for actual computation and avoiding topology logic failures caused by blind deletion or modification.

[0160] In some alternative embodiments, step 206 includes:

[0161] According to the target fragmentation strategy, multiple graphics processors are used to process the target request in multiple stages according to the deconstructed topology graph; the multiple stages include at least the Prefill stage and the Decode stage.

[0162] The Prefill stage refers to the pre-encoding stage of large-scale language model inference, also known as the pre-filling stage. It is mainly responsible for processing the complete prompt word sequence of the target request at one time and completing large-scale tensor calculations in batches. It is characterized by high computing power density, strong parallelism, and large data volume, and is a high computing power consumption stage.

[0163] The Decode stage refers to the post-decoding generation stage of large-scale language model inference. It is used to generate a single output token iteratively. It involves small computational load per operation, frequent memory access interactions, and reliance on KV cache read / write. It is a lightweight and high-frequency serial iterative operation stage.

[0164] In this embodiment, the generated target sharding strategy and standardized deconstruction topology are first read. The target sharding strategy is used as the basis for data partitioning, and the deconstruction topology is used as the basis for task execution timing and dependency constraints, so as to unify and standardize the operation rules of all shards.

[0165] Subsequently, the complete target request tensor data is equally divided into multiple independent fragment tasks according to the alignment specifications of the target fragmentation strategy. At the same time, relying on the cross-flow dependencies of computation and communication branches in the deconstructed topology graph, pipelined execution logic is configured for each fragment to ensure that communication preloading and computation are carried out alternately.

[0166] Subsequently, according to the inference business logic, the computation of each shard is scheduled to be executed in an orderly manner in two core stages. The first stage is the Prefill stage, which performs large-scale parallel computation on the complete user input sequence, fully utilizes the parallel capabilities of all shards, and relies on the parallel scheduling advantages of deconstructing the topology graph to execute multi-shard tensor operations simultaneously. It completes high-density computing tasks such as input feature encoding, tensor mapping, and KV cache initialization in batches, maximizing the computing power throughput advantage of multi-shard parallelism and quickly completing the preprocessing of large volumes of data.

[0167] The second stage is the Decode generation stage, which involves iterative generation of tokens one by one. This stage involves small data processing volumes per transaction and focuses on cache read / write and lightweight computation. The system dynamically adapts the sharding scheduling logic according to the lightweight load characteristics, weakens the ability of large-scale parallel sharding, avoids the communication overhead of small-volume sharding, and relies on the stable temporal dependencies of the deconstructed topology graph to iteratively complete each step of output feature calculation, cache update, and result output in an orderly manner, ensuring a stable and low-latency iterative generation process.

[0168] The entire process achieves differentiated processing by binding sharding strategies and optimizing the topology, enabling efficient parallel processing in high-computing scenarios for Prefill and stable iteration in lightweight scenarios for Decode. It avoids problems such as topology deadlock, memory access mismatch, and sharding resource waste throughout the process, and completes all operator processing of the target request in a complete, stable and efficient manner.

[0169] The aforementioned sharding scheduling method can achieve refined and adaptive operator scheduling optimization for the differentiated computing power characteristics of the two major stages of large-scale language model inference, Prefill and Decode, effectively solving the problems of poor adaptability and low resource utilization of a single execution mode. In the high-computing, high-volume Prefill stage, it can fully leverage the parallel computing capabilities of multiple shards, improving computational and communication coordination efficiency through an optimized topology pipeline structure, significantly increasing batch tensor processing speed and overall inference throughput. In the lightweight, high-iteration Decode stage, it dynamically adapts the scheduling logic, proactively avoiding the communication overhead and synchronization loss caused by multiple shards, ensuring low latency and high stability in the inference generation process. Simultaneously, relying on acyclic and precise deconstruction of the topology graph to constrain task dependencies, it completely avoids scheduling blockages and execution errors caused by topology logic anomalies, thereby effectively balancing inference performance and system stability, and significantly improving the overall operational efficiency of the operators.

[0170] In some optional embodiments, the step of using multiple graphics processors to process the target request in multiple stages according to the deconstructed topology graph, based on the target fragmentation strategy, includes:

[0171] Following the sequence of multiple stages, the topology graph is modified based on the preset topology graph corresponding to the first stage; the modifications include at least one of node changes, edge changes, and parameter changes.

[0172] The graphics processor is controlled to use the modified structure diagram for the first stage of processing;

[0173] Continuing in the order of arrangement, based on the preset topology graph corresponding to the m-th stage, the modified deconstruction topology graph used in the (m-1)-th stage is changed again... until the processing of multiple stages; where m is an integer greater than or equal to 1.

[0174] Pre-configured topology maps refer to exclusive standard topology templates that are pre-configured and adapted to the computing power characteristics and task requirements of each independent operation stage. The pre-configured topology maps of different stages have different configurations in terms of the number of nodes, dependencies, and operating parameters.

[0175] The stage arrangement order refers to the fixed sequence of execution stages, such as following the stage arrangement order of Prefill first and Decode later.

[0176] Topology change refers to the process of making targeted adjustments and optimizations to the topology used in the previous stage based on the pre-set topology template of the current stage.

[0177] Node change refers to the process of adding, deleting, or replacing dependent nodes in the topology graph, and adjusting the computational and communication task units required during the adjustment phase.

[0178] Edge modification refers to the process of adding, deleting, or modifying dependent edges or cross-flow dependent edges to adjust the temporal constraints and data dependencies between nodes.

[0179] Parameter change refers to the process of modifying operational parameters such as node operation parameters, sharding scheduling parameters, and memory access configuration parameters.

[0180] In this embodiment, a preset topology graph corresponding to the first stage can be retrieved first. Based on this, the initial deconstructed topology graph can be modified and adjusted in a targeted manner. The modification method can be flexibly selected from any one or more of node modification, edge modification, and parameter modification according to the stage requirements, to adapt to the computing power characteristics, data scale, and communication requirements of the first stage.

[0181] After the topology change is completed, a new topology is generated to adapt to the first stage (i.e., the Prefill stage). Based on the changed topology map, the GPU device executes the complete data processing and operator operation tasks of the first stage in strict accordance with the target sharding strategy.

[0182] Subsequently, after the first stage is completed, the preset topology map corresponding to the second stage is retrieved according to the preset stage order. Based on the topology structure that has been changed and taken effect in the previous stage, dynamic changes and optimizations of nodes, edges or parameters are carried out again to adapt to the task characteristics of the current stage (i.e., the Decode stage) in real time.

[0183] By continuously iterating through the update and hardware execution process, the topology adaptive optimization and graphics processor computing power scheduling are completed in stages until all inference stages are completed, thus ultimately achieving full-process sharded computation processing of the complete target request.

[0184] The aforementioned sharding scheduling method employs a phased, dynamic, iterative update of the topology structure, which addresses the issue of fixed topologies being unable to adapt to the diverse scenarios of multi-stage inference for large models, significantly improving the overall execution efficiency and scheduling adaptability of GPU clusters. This method relies on iterative optimization using dedicated pre-defined topology templates for each stage, making incremental adjustments based on the topology effective in the previous stage. This preserves the continuity of topology execution logic while avoiding the scheduling overhead of rebuilding the topology each time. Through flexible dynamic changes to nodes, dependent edges, and runtime parameters, it can accurately adapt to the different computing power, memory access, and communication requirements of high-performance parallel scenarios like Prefill and lightweight iterative Decode, ensuring that each inference stage is in an optimal execution state. Compared to a globally unified topology operation mode, this dynamic adaptation mechanism effectively avoids the problems of wasted computing power, communication redundancy, or memory access mismatch that occur with a single topology structure at different stages. Simultaneously, the standardized, phased iterative process is logically rigorous and highly controllable, stably cooperating with sharding strategies and GPU parallel scheduling, significantly improving overall inference throughput and hardware resource utilization while ensuring system stability.

[0185] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0186] Based on the same inventive concept, this application also provides a fragmentation scheduling apparatus for implementing the fragmentation scheduling method described above. The solution provided by this fragmentation scheduling apparatus is similar to the implementation scheme described in the fragmentation scheduling method above. Therefore, the specific limitations in one or more apparatus embodiments provided below can be found in the limitations of the fragmentation scheduling method described above, and will not be repeated here.

[0187] In one embodiment, such as Figure 4 As shown, a fragmentation scheduling device 400 is provided, comprising:

[0188] The acquisition module 402 is used to acquire the operator dimension features and hardware parameters of the operator to be executed; the hardware parameters include at least the memory access alignment boundary;

[0189] The determination module 404 is used to determine the target sharding strategy of the operator to be executed for the target request based on the operator dimension features under the constraint of memory access alignment boundary.

[0190] Processing module 406 is used to process target requests using multiple graphics processors according to the target fragmentation strategy.

[0191] In some optional embodiments, the determining module 404 is further configured to:

[0192] Based on the operator dimension features and the dynamic features of the target request, the request type of the target request is determined;

[0193] If the request type of the target request is the target type, the overlapping evaluation result of each shard in different candidate sharding strategies is determined based on dynamic features;

[0194] Based on the memory access alignment boundary, the candidate fragmentation strategy is obtained by verifying the overlap evaluation results to indicate that the computational cost of fragmentation covers the communication cost, and the target fragmentation strategy is obtained.

[0195] In some optional embodiments, the determining module 404 is further configured to:

[0196] Based on the operator dimensionality and dynamic characteristics, the computing power density of the operator to be executed for the target request is calculated;

[0197] The request type of the target request is determined based on the computing power density and the pre-configured density threshold;

[0198] Specifically, if the computing power density is greater than the density threshold, the request type is the first type; otherwise, the request type is the second type. The first type indicates that the request traffic is greater than the second type.

[0199] In some optional embodiments, the target type is a first type, and the determining module 404 is further configured to:

[0200] Based on dynamic characteristics, the computational and communication costs corresponding to each fragment in the candidate fragmentation strategy are determined.

[0201] Based on the computational load and the communication load, determine the ratio of the delay corresponding to the computational load to the delay corresponding to the communication load.

[0202] Based on the ratio and a pre-configured ratio threshold, the overlap evaluation result of the operator to be executed is determined;

[0203] Specifically, if the ratio reaches the ratio threshold, the overlap evaluation result is that the computational cost of the fragments covers the communication cost; otherwise, the overlap evaluation result is that the computational cost of the fragments cannot cover the communication cost.

[0204] In some optional embodiments, the determining module 404 is further configured to:

[0205] For candidate fragmentation strategies whose computational complexity covers communication volume as indicated by the overlap evaluation results, the candidate fragmentation strategy that reaches the memory access alignment boundary of the fragmentation byte tensor is taken as the target fragmentation strategy.

[0206] In some optional embodiments, the determining module 404 is further configured to:

[0207] If the byte tensors of fragments with multiple candidate fragmentation strategies reach the memory access alignment boundary, the target fragmentation strategy is determined based on the predicted processing latency of the target request corresponding to different candidate fragmentation strategies.

[0208] In some optional embodiments, the determining module 404 is further configured to:

[0209] If there is no candidate sharding strategy whose byte tensor reaches the memory access alignment boundary, execute the sharding rollback strategy to iteratively reduce the number of shards indicated by the candidate sharding strategy until the byte tensor of the shards reaches the memory access alignment boundary, and obtain the target sharding strategy.

[0210] In some optional embodiments, the processing module 406 is further configured to:

[0211] The operator to be executed is parsed to obtain the execution topology graph corresponding to the operator; the execution topology graph contains multiple dependent nodes and dependency edges connecting the multiple dependent nodes;

[0212] Based on multiple dependent nodes, construct computation branches and communication branches; each computation branch and communication branch contains multiple dependent nodes.

[0213] Construct cross-flow dependency edges between the nth dependent node of the computation branch and the (n-1)th dependent node of the communication branch to obtain the deconstructed topology graph; where n is an integer greater than or equal to 1.

[0214] In some optional embodiments, the processing module 406 is further configured to:

[0215] If there is a directed cycle formed by multiple dependent nodes and multiple dependent edges in the execution topology graph, traverse the dependent edges in the directed cycle, and remove the dependent edge if the output of the upstream dependent node of the dependent edge is not used as the input of the downstream dependent node.

[0216] In some optional embodiments, the processing module 406 is further configured to:

[0217] According to the target fragmentation strategy, multiple graphics processors are used to process the target request in multiple stages according to the deconstructed topology graph; the multiple stages include at least the Prefill stage and the Decode stage.

[0218] In some optional embodiments, the determining module 404 is further configured to:

[0219] Following the sequence of multiple stages, the topology graph is modified based on the preset topology graph corresponding to the first stage; the modifications include at least one of node changes, edge changes, and parameter changes.

[0220] The graphics processor is controlled to use the modified structure diagram for the first stage of processing;

[0221] Continuing in the order of arrangement, based on the preset topology graph corresponding to the m-th stage, the modified deconstruction topology graph used in the (m-1)-th stage is changed again... until the processing of multiple stages; where m is an integer greater than or equal to 1.

[0222] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0223] Figure 5A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 500 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 500 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0224] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0225] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0226] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0227] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0228] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0229] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0230] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0231] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0232] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0233] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0234] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0235] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0236] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0237] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0238] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A segmented scheduling method, characterized in that, include: Obtain the operator dimension features and hardware parameters of the operator to be executed; the hardware parameters include at least the memory access alignment boundary. Under the constraints of the memory access alignment boundary, the target sharding strategy of the operator to be executed for the target request is determined based on the operator dimension features; According to the target fragmentation strategy, multiple graphics processors are used to process the target request.

2. The method according to claim 1, characterized in that, Under the constraint of the memory access alignment boundary, determining the target sharding strategy of the operator to be executed for the target request based on the operator dimension features includes: Based on the operator dimension features and the dynamic features of the target request, the request type of the target request is determined; If the request type of the target request is a target type, based on the dynamic characteristics, determine the overlap evaluation result of each shard in different candidate sharding strategies; Based on the memory access alignment boundary, the candidate sharding strategy that indicates the computational cost of sharding covers the communication cost is verified by verifying the overlap evaluation result, and the target sharding strategy is obtained.

3. The method according to claim 2, characterized in that, Determining the request type of the target request based on the operator dimension features and the dynamic features of the target request includes: Based on the operator dimensional features and the dynamic features, the computing power density of the operator to be executed for the target request is calculated; Based on the computing power density and the pre-configured density threshold, the request type of the target request is determined; Wherein, if the computing power density is greater than the density threshold, the request type is the first type; otherwise, the request type is the second type; the request traffic indicated by the first type is greater than that of the second type.

4. The method according to claim 3, characterized in that, The target type is the first type, and the step of determining the overlap evaluation result of each fragment in different candidate fragmentation strategies based on the dynamic features includes: Based on the dynamic characteristics, the computational and communication costs corresponding to each shard in the candidate sharding strategy are determined. Based on the computational load and the communication load, determine the ratio of the delay corresponding to the computational load to the delay corresponding to the communication load; Based on the ratio and a pre-configured ratio threshold, the overlap evaluation result of the operator to be executed is determined; Wherein, if the ratio reaches the ratio threshold, the overlap evaluation result is that the computational cost of the fragment covers the communication cost; otherwise, the overlap evaluation result is that the computational cost of the fragment cannot cover the communication cost.

5. The method according to claim 4, characterized in that, The process of verifying the candidate sharding strategy based on the memory access alignment boundary and the overlap evaluation result indicating that the computational cost of sharding covers the communication cost, and obtaining the target sharding strategy, includes: For candidate sharding strategies whose computational complexity covers communication volume as indicated by the overlap evaluation results, the candidate sharding strategy whose byte tensor of the shard reaches the memory access alignment boundary is taken as the target sharding strategy.

6. The method according to claim 5, characterized in that, The method further includes: If the byte tensors of fragments from multiple candidate fragmentation strategies reach the memory access alignment boundary, the target fragmentation strategy is determined based on the predicted processing latency of the target request corresponding to different candidate fragmentation strategies.

7. The method according to claim 5, characterized in that, The method further includes: If there is no candidate sharding strategy whose byte tensor reaches the memory access alignment boundary, execute the sharding rollback strategy to iteratively reduce the number of shards indicated by the candidate sharding strategy until the byte tensor of the shards reaches the memory access alignment boundary, and obtain the target sharding strategy.

8. The method according to claim 1, characterized in that, Before processing the target request using multiple graphics processors according to the target sharding strategy, the method further includes: The operator to be executed is parsed to obtain the execution topology graph corresponding to the operator to be executed; the execution topology graph contains multiple dependent nodes and dependent edges connecting the multiple dependent nodes; Based on the multiple dependent nodes, a computing branch and a communication branch are constructed; each computing branch and the communication branch contains multiple dependent nodes. A cross-flow dependency edge is constructed between the nth dependent node of the computation branch and the (n-1)th dependent node of the communication branch to obtain a deconstructed topology graph; where n is an integer greater than or equal to 1.

9. The method according to claim 8, characterized in that, After parsing the operator to be executed to obtain the execution topology graph corresponding to the operator to be executed, the process further includes: If there is a directed cycle formed by multiple dependent nodes and multiple dependent edges in the execution topology graph, traverse the dependent edges in the directed cycle, and remove the dependent edge if the output of the upstream dependent node of the dependent edge is not used as the input of the downstream dependent node.

10. The method according to claim 8 or 9, characterized in that, The step of processing the target request using multiple graphics processors according to the target sharding strategy includes: According to the target sharding strategy, multiple graphics processors are used to process the target request in multiple stages according to the deconstructed topology graph; the multiple stages include at least a Prefill stage and a Decode stage.

11. The method according to claim 10, characterized in that, The step of processing the target request in multiple stages according to the target fragmentation strategy and the deconstructed topology graph using multiple graphics processors includes: According to the arrangement order of the multiple stages, the deconstructed topology graph is modified based on the preset topology graph corresponding to the first stage; wherein the modification includes at least one of node modification, edge modification, and parameter modification; The graphics processor is controlled to use the modified structure map to perform the first stage of processing; Continuing in the aforementioned order, based on the preset topology graph corresponding to the m-th stage, the modified deconstruction topology graph used in the (m-1)-th stage is changed again... until the processing of multiple stages; where m is an integer greater than or equal to 1.

12. A segmented scheduling device, characterized in that, include: The acquisition module is used to acquire the operator dimension features and hardware parameters of the operator to be executed; the hardware parameters include at least the memory access alignment boundary; The determination module is used to determine the target sharding strategy of the operator to be executed for the target request based on the operator dimension features under the constraints of the memory access alignment boundary. The processing module is used to process the target request using multiple graphics processors according to the target sharding strategy.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-11.

15. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-11.

Images