Model reasoning parallel strategy determination method and device, equipment and storage medium

Abstract

The invention discloses a model reasoning parallel strategy determination method, device and equipment and a storage medium, and the method comprises the steps: based on obtained hybrid expert model parameters, reasoning deployment requirement parameters and reasoning equipment parameters, utilizing a video memory constraint function and a computing power constraint function to determine a model reasoning parallel strategy; fitting and calculating the minimum number of AI cards required by parallel reasoning of the hybrid expert model; generating a plurality of candidate parallel policies based on the number of the AI cards and a preset parallel policy constraint rule; estimating the reasoning total time delay of each candidate parallel strategy based on the constructed time delay estimation model; and sequentially carrying out progressive screening on the reasoning total time delay of each candidate parallel strategy through a set time delay constraint condition and a load balance degree constraint condition so as to determine an optimal reasoning parallel strategy. The optimal parallel strategy of model reasoning can be automatically screened, the service deployment process can be accelerated, the service handling efficiency can be improved, the hardware resource utilization rate can be remarkably improved, and meanwhile a large amount of time cost and power cost can be saved.

Description

Technical Field

[0001] This invention relates to the field of distributed computing technology for artificial intelligence, and in particular to a method, apparatus, device, and storage medium for determining parallel strategies for model inference. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning models have achieved remarkable results in various fields. Hybrid expert models (MoE), as a special type of deep learning model, can significantly improve the performance and scalability of a model by combining multiple expert models. However, the deployment of MoE models for inference faces several challenges, mainly in the following aspects: 1. Inaccurate resource estimation: Traditional resource estimation methods often ignore the dynamic occupancy of KV cache, resulting in large errors in memory estimation and an inability to accurately predict the actual number of AI cards required. 2. Low efficiency in strategy selection: Existing parallel inference deployment schemes for MoE models require traversing all parallel combinations (dp / tp / sp / moe_tp / moe_ep) through actual testing after the cluster is powered on, which consumes a lot of time and electricity, resulting in low efficiency. 3. Uneven expert load: Fixed expert grouping strategies ignore the differences in expert popularity distribution, leading to low utilization of AI cards with less popular experts and serious resource waste.

[0003] Therefore, how to automatically select the best parallel strategy to improve the efficiency and resource utilization of MoE model inference deployment is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] The main objective of this invention is to provide a method, apparatus, device, and storage medium for determining parallel strategies for model inference, which can automatically select the best parallel strategy for model inference, accelerate business deployment, improve business efficiency, significantly improve hardware resource utilization, and save users a lot of time and electricity costs.

[0005] In a first aspect, this application provides a method for determining a parallel strategy for model inference, wherein the method includes the following steps: Based on the acquired parameters of the hybrid expert model, inference deployment requirements, and inference device, the minimum number of AI cards required for parallel inference of the hybrid expert model is calculated by using the memory constraint function and the computing power constraint function. Based on the number of AI cards and the preset inference parallel strategy constraints, multiple candidate parallel strategies are generated. The total inference latency of each candidate parallel strategy is estimated based on the constructed latency prediction model. The total inference latency of each candidate parallel strategy is progressively screened by setting latency constraints and load balancing constraints in order to determine the optimal inference parallel strategy.

[0006] In conjunction with the first aspect above, as an optional implementation, the latency constraint includes: the estimated total inference latency of each candidate parallel strategy is less than the latency upper limit; The load balancing constraint includes: the ratio of the standard deviation to the average deviation of the load of all server devices is less than a threshold. Based on the aforementioned latency constraints, the total inference latency of each candidate parallel strategy is filtered to confirm the first set of candidate parallel strategies that meet the latency constraints. The first set of candidate parallel strategies is filtered based on the load balancing constraint to determine the target inference parallel strategy.

[0007] In conjunction with the first aspect mentioned above, as an optional implementation method, the obtained hybrid expert model parameters include: the number of model parameters P, the total number of experts E, and the number of bytes corresponding to the weight precision B; The parameters required for deploying hybrid expert model inference include: input sequence length L, target throughput. and latency limit ; The acquired inference device parameters include the memory capacity of a single AI card. Peak computing power of a single AI card Communication bandwidth (BW) between AI cards; Calculate the minimum number of devices using memory constraint functions The memory constraint function is: ,in, This is the memory utilization factor; Calculate the minimum number of devices using computing power constraint functions The computing power constraint function is as follows: ,in, C is the multi-card parallel loss factor, and C is the computational cost per token coefficient. Based on the calculation results of the memory constraint function and the computing power constraint function, the minimum value is selected as the minimum number of AI cards required for final inference parallelism. ,in, .

[0008] In conjunction with the first aspect above, as an optional implementation method, the preset inference parallel strategy constraint rules include: tensor parallel constraint rules, which include: determining that the tensor parallel communication overhead is within the hardware bandwidth tolerance range; The sequence parallel constraint rules include: ensuring that the sequence is correctly and evenly divided into different inference devices and that the inference device resources occupied by the sequence parallelism are less than or equal to the number of devices. The expert intra-parallelism constraint rules include: expert intra-parallelism is a subset of tensor parallelism; Data parallelism constraint rules include: the result of data parallel computation is an integer greater than or equal to 1; The expert parallelism constraint rules include: the result of expert parallel computation is an integer greater than or equal to 1, the number of expert parallel groups does not exceed the total number of experts in the hybrid expert model itself, and the ratio of the total number of experts to the number of parallel experts is equal to zero. Based on the preset inference parallel strategy constraints and the number of AI cards, the values ​​of tensor parallelism tp, sequence parallelism sp, expert intra-parallelism moe_ep, data parallelism dp, and expert parallelism moe_tp are determined to generate multiple candidate parallel strategies. Each candidate parallel strategy consists of a set of specific values ​​of dp, tp, sp, moe_tp, and moe_ep.

[0009] In conjunction with the first aspect mentioned above, as an optional implementation method, according to the formula: dp = / (tp sp), calculate the parallel values ​​of the data, where tp is the calibrated tensor parallel values, sp is the calibrated sequence parallel values, and N is the number of AI cards; According to the formula: moe_tp = / (moe_ep sp), calculates the value of expert parallelism, where moe_ep is the calibrated value of expert intra-parallelism.

[0010] In conjunction with the first aspect mentioned above, as an optional implementation method, according to the formula: The prediction model estimates the total inference latency for each candidate parallel strategy, where... For AI card peak computing power, To calculate the efficiency factor, This represents the total computational cost of AI. For the data parallelism candidate strategy space, For the sequence parallel candidate policy space, As a load balancing factor, For link utilization, For parallel strategies, For the number of parameters in a hybrid expert model, This represents the number of bytes corresponding to the weight precision. For AI card communication bandwidth. This refers to the number of AI cards.

[0011] In conjunction with the first aspect mentioned above, as an optional implementation method, according to the formula: Calculate the load on all server devices, among which, Calculate the intensity of the sequence. Strength calculations for experts This is the communication overhead coefficient. For sequence length, This represents the total number of experts.

[0012] Secondly, this application provides a model inference parallel policy determination apparatus, the apparatus comprising: The computing module is used to fit the minimum number of AI cards required for parallel inference of the hybrid expert model based on the acquired parameters of the hybrid expert model, parameters of inference deployment requirements, and parameters of inference devices, using memory constraint functions and computing power constraint functions. The generation module is used to generate multiple candidate parallel strategies based on the number of AI cards and preset inference parallel strategy constraints. The prediction module is used to predict the total inference latency of each candidate parallel strategy based on the constructed latency prediction model. The determination module is used to progressively filter the total inference latency of each candidate parallel strategy by sequentially using set latency constraints and load balancing constraints, so as to determine the optimal inference parallel strategy.

[0013] Thirdly, this application also provides an electronic device, the electronic device comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described in any one of the first aspects.

[0014] Fourthly, this application also provides a computer-readable storage medium storing computer program instructions that, when executed by a computer, cause the computer to perform the method described in any of the first aspects.

[0015] This application provides a method, apparatus, device, and storage medium for determining a parallel inference strategy for a model. The method includes the following steps: based on acquired hybrid expert model parameters, inference deployment requirement parameters, and inference device parameters, using memory constraint functions and computing power constraint functions, calculating the minimum number of AI cards required for parallel inference of the hybrid expert model; generating multiple candidate parallel strategies based on the number of AI cards and preset inference parallel strategy constraint rules; estimating the total inference latency of each candidate parallel strategy based on a constructed latency prediction model; and progressively filtering the total inference latency of each candidate parallel strategy through set latency constraints and load balancing constraints to determine the optimal inference parallel strategy. This application can automatically select the optimal parallel strategy for model inference, accelerate business deployment, improve business efficiency, significantly improve hardware resource utilization, and save users significant time and electricity costs.

[0016] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] Figure 1 This is a flowchart of a method for determining a parallel strategy for model inference provided in an embodiment of this application; Figure 2 This is a schematic diagram of a model inference parallel strategy determination device provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the fitting calculation of the number of AI cards provided in the embodiments of this application; Figure 4 This is a schematic diagram illustrating the optimal parallel strategy selection provided in the embodiments of this application; Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application; Figure 6 This is a schematic diagram of a computer-readable program medium provided in an embodiment of this application. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0020] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings represent functional entities and do not necessarily correspond to physically or logically independent entities.

[0021] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0022] Reference Figure 1 , Figure 1 The diagram shown is a flowchart of a parallel strategy determination method for model inference provided by the present invention. Figure 1 As shown, the method includes the following steps: Step S101: Based on the acquired hybrid expert model parameters, inference deployment requirement parameters, and inference device parameters, use the memory constraint function and computing power constraint function to fit and calculate the minimum number of AI cards required for parallel inference of the hybrid expert model.

[0023] Specifically, the parameters of the hybrid expert model obtained include: the number of model parameters P, the total number of experts E, and the number of bytes corresponding to the weight precision B; The parameters required for deploying hybrid expert model inference include: input sequence length L, target throughput. and latency limit ; The acquired inference device parameters include the memory capacity of a single AI card. Peak computing power of a single AI card Communication bandwidth (BW) between AI cards; Calculate the minimum number of devices using memory constraint functions The memory constraint function is: ,in, This is the memory utilization factor; Calculate the minimum number of devices using computing power constraint functions The computing power constraint function is as follows: ,in, C is the multi-card parallel loss factor, and C is the computational cost per token coefficient. Based on the calculation results of the memory constraint function and the computing power constraint function, the minimum value is selected as the minimum number of AI cards required for final inference parallelism. ,in, .

[0024] For ease of understanding, an example is provided to obtain the parameters of the hybrid expert model (moe), including the number of model parameters P (175B), the number of experts E (8), the number of bytes of weight precision B (FP16, B=2), and other information.

[0025] Obtain the parameters required for deploying the moe model inference, including the input sequence length L (2048) and the target throughput. and latency limit (200ms).

[0026] Obtain inference device parameters, including the memory capacity of a single AI card in the inference device. Peak computing power of a single AI card (312 TFLOPS) and AI card inter-card communication bandwidth BW (600 GB / s).

[0027] Calculate the minimum number of devices using the memory constraint function: ,in, This is a memory utilization factor that quantifies the memory usage of activation values ​​and the key-value cache. The memory utilization factor needs further explanation. Range optimization: 0.95 (industrial safety value).

[0028] Calculate the minimum number of devices using the computing power constraint function: ,in, Here, C represents the multi-card parallel loss factor, and C is the computational cost per token. In the MOE architecture model, C = k × L (input sequence length), where k ≈ 1.3-1.8. It should be noted that the communication loss factor β is dynamically compensated. .

[0029] Based on the memory constraint function and the computing power constraint function, determine the number of AI cards required for the final inference parallelism. , .

[0030] In one embodiment, the number of inference devices required for parallel inference of the moe model is determined: , The number of AI cards required for the final inference is 8, which means that each inference device is equipped with 8 AI cards.

[0031] Step S102: Based on the number of AI cards and the preset inference parallel strategy constraints, generate multiple candidate parallel strategies.

[0032] Specifically, the preset inference parallel strategy constraint rules include: tensor parallel constraint rules, which include: determining that the tensor parallel communication overhead is within the hardware bandwidth tolerance range; The sequence parallel constraint rules include: ensuring that the sequence is correctly and evenly divided into different inference devices and that the inference device resources occupied by the sequence parallelism are less than or equal to the number of devices. The expert intra-parallelism constraint rules include: expert intra-parallelism is a subset of tensor parallelism; Data parallelism constraint rules include: the result of data parallel computation is an integer greater than or equal to 1; The expert parallelism constraint rules include: the result of expert parallel computation is an integer greater than or equal to 1, the number of expert parallel groups does not exceed the total number of experts in the hybrid expert model itself, and the ratio of the total number of experts to the number of parallel experts is equal to zero. Based on the preset inference parallel strategy constraints and the number of AI cards, the values ​​of tensor parallelism tp, sequence parallelism sp, expert intra-parallelism moe_ep, data parallelism dp, and expert parallelism moe_tp are determined to generate multiple candidate parallel strategies. Each candidate parallel strategy consists of a set of specific values ​​of dp, tp, sp, moe_tp, and moe_ep.

[0033] It is understandable that a five-dimensional candidate policy space is generated based on the constraint rule table of the five-dimensional policy candidate space of [dp, tp, sp, moe_tp, moe_ep] (Table 1).

[0034] Table 1

[0035] To illustrate this more clearly, let's assume that after fitting and calculation, it's determined that N = 8 AI cards are needed for the MoE model inference. Then, we'll discuss the process of generating the policy space by combining parallel policy constraint rules: We need to find all possible integer combinations [dp, tp, sp, moe_tp, moe_ep] such that the product of these dimensions or a specific relation equals the total number of cards, 8.

[0036] Dimension 1: Sequence parallelism (sp), determined by the rule sp = 1 (fixed in the embodiment), value: 1.

[0037] Dimension 2: Tensor Parallelism (tp). According to the rules, tp must be a divisor of N and between 1 and N. Candidate values: 1, 2, 4, 8 (i.e., it can be a 1-card, 2-card, 4-card, or 8-card split).

[0038] Dimension 3: Expert Parallelism (moe_ep), further dividing experts based on tp. If tp=1, with 8 remaining resources, moe_ep can be 1, 2, 4, or 8. If tp=2, with 4 remaining resources, moe_ep can be 1, 2, or 4.

[0039] Dimension 4: Intra-expert parallelism (moe_tp), which further splits a single expert across multiple cards.

[0040] For example, if it is decided that there are 4 experts in parallel (moe_ep = 4), but each expert is too large and requires 2 cards to calculate one expert, then moe_tp = 2.

[0041] Dimension 5: Data parallelism (dp) means that the remaining resources are used to run multiple copies of the data.

[0042] The generated candidate policy space list (example): Strategy Option A (Pure Expert Parallelism): tp = 1 (no splitting); sp = 1; moe_tp = 1 (no splitting within experts); moe_ep = 8 (split experts onto 8 cards); dp = 1 (remaining resources are 0), result: [dp=1, tp=1, sp=1, moe_tp =1, moe_ep =8] Strategy Option B (Hybrid Expert Parallelism and Data Parallelism): tp = 1; sp = 1; moe_tp = 1; moe_ep = 4 (distribute experts across 4 cards); dp = 2 (run data parallel on the remaining 4 cards), result: dp=2, tp=1, sp=1, moe_tp =1, moe_ep =4] Strategy Option C (Hybrid Expert Intra-parallel): tp = 1; sp = 1; moe_tp = 2 (each expert occupies 2 cards); moe_ep = 4 (experts are distributed in 4 groups, 2 cards per group, 8 cards in total); dp = 1, result: [dp=1, tp=1, sp=1, moe_tp =2, moe_ep =4].

[0043] Strategy Option D (Introducing Tensor Parallelism): tp = 2 (model layer split into 2 cards); sp = 1; moe_tp = 1; moe_ep = 4 (experts split into 4 groups); dp = 1, result: [dp=1, tp=2, sp=1, moe_tp =1, moe_ep =4].

[0044] It can be understood that generating a five-dimensional candidate strategy space means that, given the requirement of 8 cards (N=8), the system automatically calculates and lists a variety of possible parallel combinations such as A, B, C, and D based on preset inference constraints.

[0045] In one embodiment, according to the formula: dp = / (tp sp), calculates the parallel values ​​of the data, where tp is the calibrated tensor parallel values, sp is the calibrated sequence parallel values, and N is the number of AI cards; according to the formula: moe_tp = / (moe_ep sp), calculates the value of expert parallelism, where moe_ep is the calibrated value of expert intra-parallelism.

[0046] Step S103: Based on the constructed latency prediction model, predict the total inference latency of each candidate parallel strategy.

[0047] Specifically, according to the formula: The prediction model estimates the total inference latency for each candidate parallel strategy, where... For AI card peak computing power, To calculate the efficiency factor, This represents the total computational cost of AI. For the data parallelism candidate strategy space, For the sequence parallel candidate policy space, As a load balancing factor, For link utilization, For parallel strategies, For the number of parameters in a hybrid expert model, This represents the number of bytes corresponding to the weight precision. For AI card communication bandwidth. This refers to the number of AI cards.

[0048] For ease of understanding and illustration, the following formula is used to construct a latency prediction model for computation-communication coordination: ,in To calculate the delay, This represents communication latency. P represents the number of model parameters, L represents the model input length, and Out_length represents the model output length.

[0049] (Parallel Strategy) = ,in For example, the corresponding weighting coefficients. This represents the weighting coefficient corresponding to tp.

[0050] Step S104: The total inference latency of each candidate parallel strategy is progressively screened by the set latency constraints and load balancing constraints in order to determine the optimal inference parallel strategy.

[0051] Specifically, the latency constraints include: the estimated total inference latency of each candidate parallel strategy is less than the latency upper limit; the load balancing constraints include: the ratio of the standard deviation to the average deviation of the load of all server devices is less than a threshold; the total inference latency of each candidate parallel strategy is filtered based on the latency constraints to confirm a first set of candidate parallel strategies that meets the latency constraints; the first set of candidate parallel strategies is filtered based on the load balancing constraints to determine the target inference parallel strategy.

[0052] Based on the model deployment requirements, the following latency constraints for the first layer are defined: < .

[0053] Based on the parallel strategy scheme, the load balancing constraint under the parallel strategy scheme is: the ratio of the standard deviation to the average deviation of the load of all server devices is less than a threshold.

[0054] According to the formula: Calculate the load on all server devices, among which, Calculate the intensity of the sequence. Strength calculations for experts This is the communication overhead coefficient. For sequence length, This represents the total number of experts.

[0055] The optimal parallel strategy is output by considering latency constraints and load balancing constraints.

[0056] It is understandable that a dual progressive constraint filtering condition is formulated to select the best parallel inference strategy.

[0057] In summary, the beneficial effects of this application include: 1) Saves users a lot of time and electricity costs: This patent adopts a hybrid expert model parallel strategy automatic screening method based on dynamic constraint modeling to select the best inference parallel strategy, without requiring users to traverse and find the best deployment scheme by iterating through the cluster through power-on testing. This patent solution helps users save a lot of time and electricity costs.

[0058] 2) Significantly improves hardware resource utilization: Through precise dual-constraint modeling of video memory and computing power (such as the video memory utilization factor η and the parallel loss factor β), optimal device resource configuration is ensured. Experiments show that this solution can achieve a device utilization rate of 95% in a 256-card cluster, which is 40% higher than traditional deployment solutions.

[0059] 3) Multi-scenario adaptive capability: It can automatically generate the optimal strategy [dp, tp, sp, moe_tp, moe_ep] for different hardware topologies (such as Ascend / NVIDIA / Cambricon) and model size (hundreds to hundreds of billions of parameters), reducing the adaptation cost of heterogeneous platforms for enterprises.

[0060] 4) Improves business efficiency: Based on this patent, a set of optimal parallel strategy deployment schemes for the moe large model can be quickly selected, which can accelerate the business deployment process and improve business efficiency.

[0061] Reference Figure 2 , Figure 2 The diagram shown is a schematic of a parallel strategy determination device for model inference provided by the present invention. Figure 2 As shown, the device includes: The computing module 201 is used to fit and calculate the minimum number of AI cards required for parallel inference of the hybrid expert model based on the acquired hybrid expert model parameters, inference deployment requirement parameters and inference device parameters, using the video memory constraint function and computing power constraint function. The generation module 202 is used to generate multiple candidate parallel strategies based on the number of AI cards and preset inference parallel strategy constraint rules; The prediction module 203 is used to predict the total inference latency of each candidate parallel strategy based on the constructed latency prediction model; The determination module 204 is used to progressively filter the total inference latency of each candidate parallel strategy by sequentially using set latency constraints and load balancing constraints, so as to determine the optimal inference parallel strategy.

[0062] Furthermore, in one possible implementation, the determining module is also used to determine the latency constraint condition including: the estimated total inference latency of each candidate parallel strategy is less than the latency upper limit; The load balancing constraint is determined by the following condition: the ratio of the standard deviation to the mean deviation of the load of all server devices is less than a threshold. Based on the aforementioned latency constraints, the total inference latency of each candidate parallel strategy is filtered to confirm the first set of candidate parallel strategies that meet the latency constraints. The first set of candidate parallel strategies is filtered based on the load balancing constraint to determine the target inference parallel strategy.

[0063] Furthermore, in one possible implementation, the calculation module is also used to obtain the hybrid expert model parameters including: the number of model parameters P, the total number of experts E, and the number of bytes corresponding to the weight precision B; The parameters required for deploying hybrid expert model inference include: input sequence length L, target throughput. and latency limit ; The acquired inference device parameters include the memory capacity of a single AI card. Peak computing power of a single AI card Communication bandwidth (BW) between AI cards; Calculate the minimum number of devices using memory constraint functions The memory constraint function is: ,in, This is the memory utilization factor; Calculate the minimum number of devices using computing power constraint functions The computing power constraint function is as follows: ,in, C is the multi-card parallel loss factor, and C is the computational cost per token coefficient. Based on the calculation results of the memory constraint function and the computing power constraint function, the minimum value is selected as the minimum number of AI cards required for final inference parallelism. ,in, .

[0064] Furthermore, in one possible implementation, the generation module is also used to include the preset inference parallel strategy constraint rules as follows: tensor parallel constraint rules, which include: determining that the tensor parallel communication overhead is within the hardware bandwidth tolerance range; The sequence parallel constraint rules include: ensuring that the sequence is correctly and evenly divided into different inference devices and that the inference device resources occupied by the sequence parallelism are less than or equal to the number of devices. The expert intra-parallelism constraint rules include: expert intra-parallelism is a subset of tensor parallelism; Data parallelism constraint rules include: the result of data parallel computation is an integer greater than or equal to 1; The expert parallelism constraint rules include: the result of expert parallel computation is an integer greater than or equal to 1, the number of expert parallel groups does not exceed the total number of experts in the hybrid expert model itself, and the ratio of the total number of experts to the number of parallel experts is equal to zero. Based on the preset inference parallel strategy constraints and the number of AI cards, the values ​​of tensor parallelism tp, sequence parallelism sp, expert intra-parallelism moe_ep, data parallelism dp, and expert parallelism moe_tp are determined to generate multiple candidate parallel strategies. Each candidate parallel strategy consists of a set of specific values ​​of dp, tp, sp, moe_tp, and moe_ep.

[0065] Furthermore, in one possible implementation, the calculation module is also used to calculate according to the formula: dp = / (tp sp), calculate the parallel values ​​of the data, where tp is the calibrated tensor parallel values, sp is the calibrated sequence parallel values, and N is the number of AI cards; According to the formula: moe_tp = / (moe_ep sp), calculates the value of expert parallelism, where moe_ep is the calibrated value of expert intra-parallelism.

[0066] Furthermore, in one possible implementation, the estimation module is also used to estimate according to the formula: The prediction model estimates the total inference latency for each candidate parallel strategy, where... For AI card peak computing power, To calculate the efficiency factor, This represents the total computational cost of AI. For the data parallelism candidate strategy space, For the sequence parallel candidate policy space, As a load balancing factor, For link utilization, For parallel strategies, For the number of parameters in a hybrid expert model, This represents the number of bytes corresponding to the weight precision. For AI card communication bandwidth. This refers to the number of AI cards.

[0067] Furthermore, in one possible implementation, the calculation module is also used to calculate according to the formula: Calculate the load on all server devices, among which, Calculate the intensity of the sequence. Strength calculations for experts This is the communication overhead coefficient. For sequence length, This represents the total number of experts.

[0068] Reference Figure 3 , Figure 3 The diagram shown illustrates the fitting calculation of the number of AI cards provided by this invention. Figure 3 As shown: Step S301: Input MoE model parameters: number of parameters P (175B), number of experts E (8), weight precision B (FP16, B=2).

[0069] Deployment requirements: Sequence length L (2048), target throughput (100 tokens / s), latency limit (200ms).

[0070] Device parameters: Single card video memory (80GB), single-card computing power (312 TFLOPS), inter-card bandwidth BW (600GB / s).

[0071] Step S301: Calculate the minimum number of devices using the video memory constraint function. , . The quantization key-value cache occupies 15% of the video memory.

[0072] Calculate the minimum number of devices using computing power constraint functions , β=0.7, multi-card parallel efficiency loss of 30%.

[0073] Step S303: Final number of cards: .

[0074] Reference Figure 4 , Figure 4 The diagram shown illustrates the optimal parallel strategy selection method provided by this invention. Figure 4 As shown: Step S401: Generate a five-dimensional candidate policy space based on the constraint rule table of the five-dimensional policy candidate space [dp, tp, sp, moe_tp, moe_ep].

[0075] Step S402: Construct a latency prediction model for computation-communication coordination: .

[0076] Step S403: Based on the model deployment requirements, formulate the first-layer latency constraints: .

[0077] Based on the parallel strategy scheme, define the load balancing constraints under the parallel strategy scheme: ,in For the load of all server devices.

[0078] Step S404: Output the optimal parallel strategy based on latency constraints and load balancing constraints.

[0079] Understandably, after the first layer of latency constraints is completed, the results are then subjected to a second round of filtering using load balancing constraints to obtain the final optimal parallel strategy.

[0080] The following reference Figure 5 To describe an electronic device 500 according to this embodiment of the present invention. Figure 5 The electronic device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0081] like Figure 5 As shown, the electronic device 500 is manifested in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, and a bus 530 connecting different system components (including storage unit 520 and processing unit 510).

[0082] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the "Embodiment Methods" section of this specification according to various exemplary embodiments of the present invention.

[0083] Storage unit 520 may include readable media in the form of volatile storage units, such as random access memory (RAM) 521 and / or cache memory 522, and may further include read-only memory (ROM) 523.

[0084] Storage unit 520 may also include a program / utility 524 having a set (at least one) of program modules 425, such program modules 525 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0085] Bus 530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0086] Electronic device 500 can also communicate with one or more external devices (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 500, and / or any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 560. As shown, network adapter 560 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0087] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0088] According to the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention can also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0089] refer to Figure 6As shown, a program product 600 for implementing the above-described method according to an embodiment of the present invention is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0090] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0091] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0092] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0093] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0094] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0095] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

[0096] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

Claims

1. A method for determining a parallel strategy for model inference, characterized in that, include: Based on the acquired parameters of the hybrid expert model, inference deployment requirements, and inference device, the minimum number of AI cards required for parallel inference of the hybrid expert model is calculated by using the memory constraint function and the computing power constraint function. Based on the number of AI cards and the preset inference parallel strategy constraints, multiple candidate parallel strategies are generated. The total inference latency of each candidate parallel strategy is estimated based on the constructed latency prediction model. The total inference latency of each candidate parallel strategy is progressively screened by setting latency constraints and load balancing constraints in order to determine the optimal inference parallel strategy.

2. The method according to claim 1, characterized in that, The process of progressively filtering the total inference latency of each candidate parallel strategy through set latency constraints and load balancing constraints to determine the optimal inference parallel strategy includes: The latency constraints include: the estimated total inference latency of each candidate parallel strategy is less than the latency limit; The load balancing constraint includes: the ratio of the standard deviation to the average deviation of the load of all server devices is less than a threshold. Based on the aforementioned latency constraints, the total inference latency of each candidate parallel strategy is filtered to confirm the first set of candidate parallel strategies that meet the latency constraints. The first set of candidate parallel strategies is filtered based on the load balancing constraint to determine the target inference parallel strategy.

3. The method according to claim 1, characterized in that, Based on the acquired hybrid expert model parameters, inference deployment requirement parameters, and inference device parameters, the minimum number of AI cards required for parallel inference of the hybrid expert model is calculated using memory constraint functions and computing power constraint functions, including: The obtained parameters of the hybrid expert model include: the number of model parameters P, the total number of experts E, and the number of bytes corresponding to the weight precision B; The parameters required for deploying hybrid expert model inference include: input sequence length L, target throughput. and latency limit ; The acquired inference device parameters include the memory capacity of a single AI card. Peak computing power of a single AI card Communication bandwidth (BW) between AI cards; Calculate the minimum number of devices using memory constraint functions The memory constraint function is: ,in, This is the memory utilization factor; Calculate the minimum number of devices using computing power constraint functions The computing power constraint function is as follows: ,in, C is the multi-card parallel loss factor, and C is the computational cost per token coefficient. Based on the calculation results of the memory constraint function and the computing power constraint function, the minimum value is selected as the minimum number of AI cards required for final inference parallelism. ,in, .

4. The method according to claim 1, characterized in that, Based on the number of AI cards and preset inference parallel strategy constraints, multiple candidate parallel strategies are generated, including: The preset inference parallel strategy constraint rules include: tensor parallel constraint rules, which include: determining that the tensor parallel communication overhead is within the hardware bandwidth tolerance range. The sequence parallel constraint rules include: ensuring that the sequence is correctly and evenly divided into different inference devices and that the inference device resources occupied by the sequence parallelism are less than or equal to the number of devices. The expert intra-parallelism constraint rules include: expert intra-parallelism is a subset of tensor parallelism; Data parallelism constraint rules include: the result of data parallel computation is an integer greater than or equal to 1; The expert parallelism constraint rules include: the result of expert parallel computation is an integer greater than or equal to 1, the number of expert parallel groups does not exceed the total number of experts in the hybrid expert model itself, and the ratio of the total number of experts to the number of parallel experts is equal to zero. Based on the preset inference parallel strategy constraints and the number of AI cards, the values ​​of tensor parallelism tp, sequence parallelism sp, expert intra-parallelism moe_ep, data parallelism dp, and expert parallelism moe_tp are determined to generate multiple candidate parallel strategies. Each candidate parallel strategy consists of a set of specific values ​​of dp, tp, sp, moe_tp, and moe_ep.

5. The method according to claim 4, characterized in that, include: According to the formula: dp = / (tp sp), calculate the parallel values ​​of the data, where tp is the calibrated tensor parallel values, sp is the calibrated sequence parallel values, and N is the number of AI cards; According to the formula: moe_tp = / (moe_ep sp), calculates the value of expert parallelism, where moe_ep is the calibrated value of expert intra-parallelism.

6. The method according to claim 1, characterized in that, The latency prediction model based on the constructed latency estimates the total inference latency of each candidate parallel strategy, including: According to the formula: The prediction model estimates the total inference latency for each candidate parallel strategy, where... For AI card peak computing power, To calculate the efficiency factor, This represents the total computational cost of AI. For the data parallelism candidate strategy space, For the sequence parallel candidate policy space, As a load balancing factor, For link utilization, For parallel strategies, For the number of parameters in a hybrid expert model, This represents the number of bytes corresponding to the weight precision. For AI card communication bandwidth. This refers to the number of AI cards.

7. The method according to claim 2, characterized in that, The calculation of the load on all server devices includes: According to the formula: Calculate the load on all server devices, among which, Calculate the intensity of the sequence. Strength calculations for experts This is the communication overhead coefficient. For sequence length, This represents the total number of experts.

8. A model inference parallel strategy determination device, characterized in that, include: The computing module is used to fit the minimum number of AI cards required for parallel inference of the hybrid expert model based on the acquired parameters of the hybrid expert model, parameters of inference deployment requirements, and parameters of inference devices, using memory constraint functions and computing power constraint functions. The generation module is used to generate multiple candidate parallel strategies based on the number of AI cards and preset inference parallel strategy constraints. The prediction module is used to predict the total inference latency of each candidate parallel strategy based on the constructed latency prediction model. The determination module is used to progressively filter the total inference latency of each candidate parallel strategy by sequentially using set latency constraints and load balancing constraints, so as to determine the optimal inference parallel strategy.

9. An electronic device, characterized in that, The electronic device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores computer program instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7.

Images