Construction method of reasoning simulation model, data processing method and related products

Abstract

The embodiment of the disclosure provides a construction method of an inference simulation model, a data processing method and related products, which relate to the technical field of model inference, and specifically comprises: obtaining operator running resources required by a plurality of operator units in a to-be-deployed model respectively in a model inference stage; obtaining a hardware performance baseline of a target graphics processing unit (GPU); determining a performance discount factor of the to-be-deployed model under a target parallel strategy; and constructing an inference simulation model for evaluating inference performance of the to-be-deployed model based on the operator running resources required by the plurality of operator units respectively in the model inference stage, the hardware performance baseline and the performance discount factor. In this way, the inference simulation model for evaluating the inference performance of the large-scale language model under the specific hardware configuration is accurately constructed.

Description

Technical Field

[0001] This disclosure relates to the field of model reasoning technology, and in particular to a method for constructing a reasoning simulation model, a data processing method, and related products. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale language models have demonstrated outstanding performance in natural language processing tasks. However, these large-scale language models typically contain a large number of model parameters, which places stringent demands on the computing and storage resources of the hardware devices running them. Therefore, in the practical deployment of large-scale language models, accurately evaluating and optimizing their inference performance under specific hardware configurations is crucial. Summary of the Invention

[0003] This disclosure provides a method for constructing a reasoning simulation model, a data processing method, and related products to at least solve the above-mentioned technical problems existing in the prior art.

[0004] In a first aspect, embodiments of this disclosure provide a method for constructing a reasoning simulation model, the method comprising:

[0005] Obtain the operator runtime resources required by multiple operator units in the model to be deployed during the model inference phase;

[0006] Obtain the hardware performance baseline of the target graphics processing unit (GPU);

[0007] The performance discount factor of the model to be deployed under the target parallel strategy is determined. The performance discount factor is determined based on the product of the hardware fixed discount and the operator resource discount corresponding to the target GPU. The hardware fixed discount represents the inherent performance loss of the target GPU, and the operator resource discount represents the performance discount degree of the operator unit.

[0008] Based on the operator runtime resources, hardware performance baseline, and performance discount factor required by multiple operator units in the model inference stage, an inference simulation model is constructed to evaluate the inference performance of the model to be deployed.

[0009] Secondly, embodiments of this disclosure provide a data processing method, the method comprising:

[0010] Acquire the data to be processed, which includes the operator resource parameters of the target deployment model, the target inference scenario data, and the hardware configuration information of the actual graphics processing unit (GPU) used to deploy the target deployment model;

[0011] The inference simulation model constructed using the methods and steps provided in the first aspect processes the data to be processed, and the output results of the inference simulation model are obtained. The output results include inference first-word latency, end-to-end inference throughput, and model parallelism strategy. The output results are used to characterize the inference performance of the target deployment model in the target scenario. The target scenario is determined based on the target inference scenario data and the actual GPU hardware configuration information.

[0012] Thirdly, embodiments of this disclosure provide an apparatus for constructing a reasoning simulation model, the apparatus comprising:

[0013] The acquisition module is used to acquire the operator runtime resources required by multiple operator units in the model to be deployed during the model inference phase.

[0014] The acquisition module is also used to acquire the hardware performance baseline of the target graphics processing unit (GPU);

[0015] The determination module is also used to determine the performance discount factor of the model to be deployed under the target parallel strategy. The performance discount factor is determined based on the product of the hardware fixed discount and the operator resource discount corresponding to the target GPU. The hardware fixed discount represents the inherent performance loss of the target GPU, and the operator resource discount represents the performance discount degree of the operator unit.

[0016] The building module is used to construct an inference simulation model for evaluating the inference performance of the model to be deployed, based on the operator running resources, hardware performance baseline and performance discount factor required by multiple operator units in the model inference stage.

[0017] Fourthly, embodiments of this disclosure provide a data processing apparatus, the apparatus comprising:

[0018] The acquisition module is used to acquire data to be processed, which includes operator resource parameters of the target deployment model, target inference scenario data, and hardware configuration information of the actual graphics processing unit (GPU) used to deploy the target deployment model.

[0019] The processing module is used to process the data to be processed using the inference simulation model constructed by the method steps provided in the first aspect, and to obtain the output results of the inference simulation model. The output results include inference first-word latency, end-to-end inference throughput and model parallelism strategy. The output results are used to characterize the inference performance of the target deployment model in the target scenario. The target scenario is determined based on the target inference scenario data and the actual GPU hardware configuration information.

[0020] Fifthly, embodiments of this disclosure provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the construction method of the inference simulation model provided in the first aspect or the data processing method of the second aspect.

[0021] In a sixth aspect, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute a method for constructing a reasoning simulation model according to the first aspect or a data processing method according to the second aspect.

[0022] Based on the inference simulation model construction method provided in this disclosure, an inference simulation model can be constructed with high precision by acquiring the operator running resources of the operator unit during the model inference stage, the hardware performance baseline of the target GPU, and the performance discount factor under the target parallel strategy. Since operator running resources reflect the resources required by the model to be deployed throughout the entire model inference process at the operator granularity, and the hardware performance baseline quantifies the differences in explicit parameters such as computing power and storage bandwidth among different GPUs, and the performance discount factor not only effectively reflects the degree of discount between the actual performance and the estimated performance before and after model deployment, but also further reflects the additional performance loss caused by different GPUs at the implicit parameter level through the fixed hardware discount, an inference simulation model can be accurately constructed based on the operator running resources of the operator unit during the model inference stage, the hardware performance baseline of the target GPU, and the performance discount factor under the target parallel strategy. This inference simulation model can then be used to accurately evaluate the inference performance of the model to be deployed.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a method for constructing a reasoning simulation model according to an embodiment of this disclosure;

[0025] Figure 2 This is one of the example diagrams of a method for constructing a reasoning simulation model provided in this disclosure embodiment;

[0026] Figure 3(a) is a second example of a method for constructing a reasoning simulation model provided in an embodiment of this disclosure;

[0027] Figure 3(b) is a third example of a method for constructing a reasoning simulation model provided in an embodiment of this disclosure;

[0028] Figure 4 This is the fourth example diagram of a method for constructing a reasoning simulation model provided in this embodiment of the disclosure;

[0029] Figure 5(a) is the fifth example of a method for constructing a reasoning simulation model provided in an embodiment of this disclosure;

[0030] Figure 5(b) is an example diagram of a method for constructing a reasoning simulation model provided in an embodiment of this disclosure;

[0031] Figure 5(c) is the seventh example of a method for constructing a reasoning simulation model provided in an embodiment of this disclosure;

[0032] Figure 5(d) is an example diagram of a method for constructing a reasoning simulation model provided in an embodiment of this disclosure;

[0033] Figure 6 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure;

[0034] Figure 7 This is a schematic diagram of the structure of a device for constructing a reasoning simulation model provided in an embodiment of this disclosure;

[0035] Figure 8 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure;

[0036] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0037] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0038] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this disclosure only and is not intended to limit the scope of this application.

[0041] With the rapid development of artificial intelligence technology, large-scale language models have demonstrated outstanding performance in language understanding and generation, and can be widely applied to many natural language processing tasks such as text generation, machine translation, question answering systems, text summarization, and sentiment analysis. The emergence of these models has greatly promoted the application and development of artificial intelligence technology in various fields.

[0042] However, large-scale language models typically contain a large number of model parameters, which places stringent demands on the computing and storage resources of the hardware devices running them. Therefore, accurately evaluating and optimizing the inference performance of large-scale language models under specific hardware configurations is a pressing technical problem that needs to be solved during the practical deployment of such models.

[0043] However, the existing technology has the following problems: First, the evaluation of model inference performance lacks systematicity: traditional theoretical estimation methods cannot accurately reflect the performance of the model in the actual hardware environment, resulting in a large difference between the actual performance and the estimated performance after the model is deployed on a specific graphics processing unit (GPU). The specific reasons are as follows: (1) The evaluation dimension is relatively simple. The existing technology only relies on the theoretical number of floating point operations (FLOPs) or a single latency index, without comprehensively considering multi-dimensional factors such as computing, memory, and communication; (2) The hardware modeling is rough. It does not distinguish the precision support of GPU models, such as half-precision floating-point / 16-bit floating-point (FP16), 8-bit floating-point (FP8), 4-bit floating-point (FP4), etc., and the impact of inter-GPU communication topology on performance, which leads to the simulation results deviating from the measured results.

[0044] Secondly, the model-hardware compatibility analysis is insufficient: different models of general-purpose computing on graphics processing units (GPGPUs) differ significantly not only in explicit parameters such as computing power, memory capacity, memory bandwidth, and number of computing cores, but also in implicit parameters such as communication overhead, instruction pipeline efficiency, and operator library optimization. However, existing model selection methods fail to adequately consider the matching relationship between explicit and implicit hardware characteristics and model requirements, resulting in significant deviations between simulation and experimental results.

[0045] Furthermore, it cannot guide the optimization of model inference performance: it cannot assess the impact of factors such as the parallel operation mode, input and output mode, and selection of alternative operators on the model inference performance during model operation. The simulation program cannot guide the optimization of model operation mode, cannot improve the predicament of low utilization of GPGPU hardware resources, and cannot fully realize the potential of hardware and model.

[0046] Furthermore, there is a lack of closed-loop optimization mechanisms: most existing prediction schemes are one-time evaluations, which cannot continuously optimize the simulation model based on actual test results, making it difficult to form a closed loop of "simulation-deployment-optimization".

[0047] This disclosure provides a method for constructing an inference simulation model, a data processing method, and related products, to at least solve the technical problem in the prior art that it is difficult to accurately evaluate and optimize the inference performance of large-scale language models under specific hardware configurations.

[0048] It should be noted that the execution entity of the inference simulation model construction method provided in this disclosure embodiment can be an inference simulation model construction device, or a part of the device used to execute the inference simulation model construction method. This disclosure embodiment takes the execution of the inference simulation model method by an inference simulation model construction device as an example to describe in detail the inference simulation model construction method provided in this disclosure embodiment.

[0049] Accordingly, the data processing method provided in this disclosure can be executed by a data processing device, or by a module of that device that performs the data processing method. This disclosure describes the data processing method provided in this disclosure in detail, using the execution of the data processing method by a data processing device as an example.

[0050] The method for constructing the inference simulation model provided in this disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Figure 1 This is a flowchart illustrating a method for constructing a reasoning simulation model according to an embodiment of this disclosure.

[0052] like Figure 1 As shown, the execution entity of the inference simulation model construction method can be an inference simulation model construction device. Based on this, the inference simulation model construction method provided in this embodiment can include the following steps:

[0053] S110: Obtain the operator runtime resources required by multiple operator units in the model to be deployed during the model inference phase.

[0054] The model to be deployed can be a large-scale language model to be deployed to a hardware device; no specific limitation is made here.

[0055] Additionally, the model to be deployed may include multiple operator units, such as multi-head attention, feedforward networks, normalization units, etc., without specific limitations here. Operator runtime resources refer to the resources required by the corresponding operator unit to perform related tasks (such as computation, memory access, communication, etc.) during the model inference phase, without specific limitations here.

[0056] S120, obtains the hardware performance baseline of the target graphics processing unit (GPU).

[0057] The target GPU mentioned above is the hardware execution carrier of the model to be deployed, and its specific model, quantity and specifications are not limited.

[0058] Furthermore, the aforementioned hardware performance baseline can be a performance benchmark value of the target GPU under ideal conditions. It should be noted that this hardware performance baseline can include performance baselines for computation, memory access, and communication, for example, it can include computing power, memory bandwidth, and communication bandwidth, without specific limitations here. It should also be noted that the aforementioned hardware performance baseline of the target GPU can be obtained based on the decomposition of the target graphics card's specifications. For example, if the target graphics card's specifications are H, then specifications H can be decomposed into... , For computing power, For memory bandwidth, This refers to communication bandwidth. Additionally, the target graphics card may integrate the target GPU; no specific limitations are specified here.

[0059] S130, determine the performance discount factor of the model to be deployed under the target parallel strategy.

[0060] The aforementioned target parallel strategy can be the default parallel strategy or a parallel strategy determined by relevant algorithms; no specific limitation is made here.

[0061] Furthermore, the performance discount factor can be determined by multiplying the fixed hardware discount and the operator resource discount corresponding to the target GPU. The fixed hardware discount characterizes the inherent performance loss of the target GPU, i.e., the performance degradation caused by the inherent characteristics of the target GPU such as hardware architecture and pipeline design. The operator resource discount characterizes the performance discount degree of the corresponding operator unit. Thus, this performance discount factor can be used to determine the degree of discount between the actual performance and the theoretical expected performance of the target GPU when running the model to be deployed.

[0062] S140, based on the operator running resources, hardware performance baseline and performance discount factor required by multiple operator units in the model inference stage, respectively, constructs an inference simulation model for evaluating the inference performance of the model to be deployed.

[0063] The inference simulation model can be any relevant model used to evaluate the inference performance of the model to be deployed; no specific limitations are imposed here.

[0064] Specifically, the inference simulation model building device can obtain the operator runtime resources required by multiple operator units in the model to be deployed during the model inference stage, obtain the hardware performance baseline of the target GPU, and determine the performance discount factor of the model to be deployed under the target parallel strategy. In this way, an inference simulation model for evaluating the inference performance of the model to be deployed can be constructed based on the operator runtime resources required by multiple operator units in the model inference stage, the hardware performance baseline, and the performance discount factor.

[0065] In this embodiment, an inference simulation model can be constructed with high precision by acquiring the operator runtime resources of the operator units during the model inference phase, the hardware performance baseline of the target GPU, and the performance discount factor under the target parallel strategy. Since operator runtime resources reflect the resources required by the model to be deployed throughout the entire model inference process at the operator granularity, and the hardware performance baseline quantifies the differences in explicit parameters such as computing power and storage bandwidth among different GPUs, and the performance discount factor not only effectively reflects the degree of discount between the actual performance and the estimated performance before and after model deployment, but also further reflects the additional performance loss caused by different GPUs at the implicit parameter level through the fixed hardware discount, an inference simulation model can be accurately constructed based on the operator runtime resources of the operator units during the model inference phase, the hardware performance baseline of the target GPU, and the performance discount factor under the target parallel strategy. This inference simulation model can then be used to accurately evaluate the inference performance of the model to be deployed.

[0066] In order to provide a comprehensive and detailed description of the method for constructing the inference simulation model provided in the embodiments of this disclosure, in one embodiment, the above-mentioned S110 may specifically include the following steps:

[0067] Determine the resource computation model for each operator unit during the model inference phase;

[0068] Based on the resource computation model of each operator unit in the model inference stage, the operator operation resources of the operator unit in the model inference stage are determined.

[0069] The resource calculation model described above can be used to characterize the correspondence between the corresponding operator unit and the operator running resources during the model inference stage, without any specific limitations here.

[0070] It should also be noted that, since the operator running resources in this embodiment are the resources required by the corresponding operator unit to perform related tasks (such as calculation, memory access, communication, etc.) in the model inference stage, the operator running resources in this embodiment may include at least one of the operator calculation requirement resources, operator memory access requirement resources, and operator communication requirement resources, which are not specifically limited here.

[0071] Specifically, since the model to be deployed may include multiple operator units, the device for constructing the inference simulation model can determine the resource calculation model of each operator unit among the multiple operator units. The resource calculation model represents the correspondence between the operator unit and the operator running resources. Therefore, based on the operator calculation model of the operator unit, the operator calculation model of the operator unit can be determined, thereby accurately obtaining the operator running resources required by the multiple operator units in the model to be deployed during the model inference stage.

[0072] In this embodiment, for each operator unit in the model to be deployed, based on the resource computing model of that operator unit, the operator computing resource requirement, operator memory access resource requirement, and operator communication resource requirement of each operator unit can be accurately and effectively quantified, thereby enabling accurate acquisition of the operator running resources required by multiple operator units in the model to be deployed during the model inference stage.

[0073] Since the aforementioned embodiments require determining the operator running resources of the operator unit in the model inference stage, and the model inference stage can generally be divided into a pre-filling stage and a decoding stage, the resource calculation model of each operator unit can include a first resource calculation model of the operator unit in the pre-filling stage and a second resource calculation model of the operator unit in the decoding stage. The first resource calculation model can include the correspondence between the operator unit and the operator running resources of the operator unit in the pre-filling stage, and the second resource calculation model can include the correspondence between the operator unit and the operator running resources of the operator unit in the decoding stage.

[0074] Based on this, the steps described above for determining the operator execution resources of each operator unit during the model inference phase, based on the resource computation model of each operator unit during the model inference phase, may specifically include the following steps:

[0075] Based on the first resource calculation model of each operator unit in the pre-filling stage, the first sub-running resource of the operator unit in the pre-filling stage is determined;

[0076] Based on the second resource calculation model of each operator unit in the decoding stage, the second sub-running resource of the operator unit in the decoding stage is determined.

[0077] Specifically, since the resource computation model corresponding to the operator unit includes a first resource computation model for the operator unit in the pre-filling stage and a second resource computation model for the operator unit in the decoding stage, the inference simulation model construction device can, for each of the multiple operator units, determine the first sub-running resources required by the operator unit to perform related tasks in the pre-filling stage based on the first resource computation model of the operator unit. Correspondingly, the inference simulation model construction device can, for each of the multiple operator units, determine the second sub-running resources of the operator unit in the decoding stage based on the second resource computation model of the operator unit.

[0078] Thus, the operator runtime resources corresponding to the operator unit in this embodiment may include a first sub-running resource of the operator unit in the pre-filling stage and a second sub-running resource of the operator unit in the decoding stage. The first and second sub-running resources are used only to distinguish the resource requirements of the same operator unit at different stages of model inference. Both the first and second sub-running resources may include runtime resources such as computational load, memory access load, and communication load, which are not specifically limited here.

[0079] To facilitate understanding of the inference simulation model construction method provided in this disclosure, in one example, a multi-head attention (MHA) mechanism is used as the operator unit for detailed explanation. If the model inference stage includes a pre-filling stage, the computational cost of the operator unit in the pre-filling stage is... , visit inventory , communication volume As shown in the formula below:

[0080] (1)

[0081] (2)

[0082] (3)

[0083] in, To hide the dimension, This represents the batch size for each GPU under the parallel strategy. For sequence length, This represents the total number of GPUs involved in the inference task.

[0084] If the model inference phase includes a decoding phase, the computational cost of this operator unit in the decoding phase is... , visit inventory , communication volume As shown in the formula below:

[0085] (4)

[0086] (5)

[0087] (6)

[0088] in, The number of bytes occupied by the weight precision.

[0089] The above weight precision occupies bytes Specifically, it can be shown in the formula:

[0090] (7)

[0091] Thus, the operator runtime resources required by multiple operator units during the model inference phase are respectively... It can be represented by the following formula:

[0092] (8)

[0093] in, Represents the inverted matrix of a matrix. Used to characterize the k-th operator unit, The resource requirements for operator computation of the k-th operator unit during the model inference phase can include the computational load of the pre-filling phase and the computational load of the decoding phase. This represents the memory access resource requirement of the k-th operator unit during the model inference phase, which may include the memory access amount during the pre-filling phase and the memory access amount during the decoding phase. The operator communication resource requirement of the k-th operator unit during the model inference phase can include the communication volume during the pre-filling phase and the communication volume during the decoding phase.

[0094] For ease of understanding, such as Figure 2 As shown, Figure 2 The diagram illustrates the variation of computational power requirements (i.e., operator operating resources) of operator units (such as the MHA operator) with the sequence input length s at different stages (i.e., the pre-filling stage and the decoding stage). The computational power requirements of the operator units show a quadratic increase in the pre-filling stage and a linear increase in the decoding stage. This effectively reflects the abrupt change in computational power requirements, which facilitates the automatic selection of subsequent parallel strategies.

[0095] In this embodiment, considering that the same operator unit has different requirements for operator operation resources at different stages, the operator operation resources of the operator unit at different stages can be accurately and effectively quantified by the operator calculation model of the operator unit at different stages, which facilitates the accurate construction of the inference simulation model in the future.

[0096] Since the performance discount factor of the model to be deployed under the target parallel strategy needs to be determined in the foregoing embodiments, in order to comprehensively and thoroughly describe the construction method of the inference simulation model provided by the embodiments of this disclosure, the target parallel strategy needs to be accurately determined before determining the performance discount factor. Therefore, in one embodiment, before S130, the construction method of the inference simulation model provided by the embodiments of this disclosure may further include the following steps:

[0097] Determine multiple candidate parallel strategies for the model to be deployed during the model inference phase. Each candidate parallel strategy includes tensor parallelism and data parallelism.

[0098] For each candidate parallel strategy, based on the number of tensor parallelisms, the number of data parallelisms, and the operator execution resources of each operator unit in the candidate parallel strategy, calculate the estimated operator execution resources of the operator unit under the candidate parallel strategy.

[0099] The target parallel strategy is determined from multiple candidate parallel strategies with the objective of minimizing the sum of time overhead generated by the predicted sub-operator resources under the candidate parallel strategies.

[0100] The aforementioned operator runtime resources can be determined based on tensor parallel variables and data parallel variables. Specifically, the operator runtime resources can be quantized into a function of tensor parallel variables and data parallel variables based on the resource calculation model of the aforementioned operator unit; this will not be elaborated further here. It should be noted that, in determining the target parallel strategy, the aforementioned operator runtime resources may include at least one of operator memory access requirements and operator communication requirements; no specific limitations are imposed here.

[0101] In addition, each of the above candidate parallel strategies may include candidate tensor parallelism and candidate data parallelism, which are not specifically limited here.

[0102] It should also be noted that, since the model inference stage may include a pre-filling stage and a decoding stage, and the same operator unit has different operating resources in different stages, the objective parallel strategy may include a first parallel strategy for the pre-filling stage and a second parallel strategy for the decoding stage.

[0103] Specifically, the device for constructing the inference simulation model can determine multiple candidate parallel strategies for the model to be deployed during the model inference stage. For each candidate parallel strategy, it can calculate the estimated operator running resources of the operator unit under the candidate parallel strategy based on the number of tensor parallelisms, the number of data parallelisms, and the operator running resources of each operator unit. Then, it can determine the target parallel strategy from multiple candidate parallel strategies with the goal of minimizing the sum of the time overhead generated by the estimated operator running resources of multiple operator units under the candidate parallel strategies.

[0104] In one example, the process of determining the target parallel strategy described above can be represented by the following formula:

[0105] (9)

[0106] Where tp is the tensor parallelism, dp is the data parallelism, and the number of GPUs n = tp dp, satisfying . Candidate parallel strategies The following operators require memory access resources. For graphics card memory bandwidth, The time overhead for the memory access requirements of the operator. Candidate parallel strategies The operator communication requirements of the following are resource-intensive. For graphics card communication bandwidth, The time overhead corresponding to the resources required for operator communication. Characterization from multiple candidate parallel strategies The (tp, dp) sequence with the minimum sum of time costs is selected, which is the target parallel strategy. It can include and ,in, This represents the first parallel strategy selected during the prefill phase. This represents the second parallel strategy selected during the decoding phase, and the first parallel strategy. Second parallel strategy Together they constitute a parallel strategy for achieving the goal.

[0107] For ease of understanding, as shown in FIGS. 3(a) and 3(b), FIGS. 3(a) and 3(b) illustrate the optimal parallel strategy selection mechanism under different GPU hardware configurations. Specifically, FIGS. 3(a) and 3(b) respectively show the end-to-end throughput curves and their intersection points of GPU A and GPU B under different parallel strategies, to illustrate the influence of hardware parameters (such as inter-card bandwidth) on the parallel strategy selection, thereby verifying the hardware self-adaptability of the parallel strategy formula. Among them, the FP16 computing power of GPU A is 480 Flops, the video memory bandwidth is 1800 GB / s, and the inter-card interconnect bandwidth is 480 GB / s; while the FP16 computing power of GPU B is 240 Flops, the video memory bandwidth is 1800 GB / s, and the inter-card interconnect bandwidth is 192 GB / s. Since the demarcation value of "Bytes(tp, dp) < COMM(tp, dp)" is reflected in the intersection point of the end-to-end throughput curve, taking FIG. 3(a) as an example, for GPU A, when the input sequence length s = 1024 and the output sequence length o = 1024, there is an intersection point between the end-to-end throughput curves of tp = 4 and tp = 8. The left region of this intersection point automatically falls into the tp = 8 branch. Therefore, in the left region of this intersection point, the system automatically selects tp = 8 as the optimal parallel strategy. The right region of this intersection point falls into the tp = 4 branch, and the system automatically selects tp = 4 as the optimal parallel strategy. As the number of parallel inferences continues to increase, the optimal parallel strategy will fall into the tp = 2 branch. Thus, for each GPU, only the intersection point of the performance curve corresponding to the model needs to be recorded. As a result, the number of parallel combinations that originally needed to be manually enumerated can be simplified to a single comparison operation, achieving minute-level strategy recommendation. FIG. 3(b) also gives the comparison curve of GPU B. It can be seen that due to the inter-card bandwidth of GPU B being only 1 / 2 of that of GPU A, the throughput intersection point shifts to the left, and the system will automatically tend to select a larger dp value. This figure explains why the same model can obtain completely different "optimal strategies" on different hardware, thus proving the hardware adaptive logic shown in the above formula.

[0108] In this embodiment, it is possible to calculate the pre-estimated operator running resources of each operator unit under each candidate parallel strategy, and then, with the goal of minimizing the sum of the time overheads generated by the pre-estimated operator running resources of multiple operator units under the candidate parallel strategies, determine the target parallel strategy from multiple candidate parallel strategies. In this way, the target parallel strategy of the model to be deployed in the model inference stage can be accurately determined.

[0109] In order to be able to comprehensively and detailedly describe the method for constructing the inference simulation model provided by the embodiments of the present disclosure, in one embodiment, the above S130 may specifically include the following steps:

[0110] Determine the operator resource discounts of multiple operator units respectively under the target parallel strategy;

[0111] Determine the fixed hardware discount corresponding to the target GPU;

[0112] The performance discount factor under the target parallel strategy is determined by multiplying at least one resource discount of each operator unit with the hardware fixed discount of the target GPU.

[0113] In one embodiment, the operator resource discount may include at least one resource discount, which may include at least one of computation resource discount, memory access resource discount and communication resource discount, without being specifically limited here.

[0114] Specifically, the device for constructing the inference simulation model can determine the operator resource discounts of multiple operator units under the target parallel strategy, and can also determine the hardware fixed discount corresponding to the target GPU. Thus, it can determine the performance discount factor under the target parallel strategy based on the product between at least one resource discount of each operator unit and the hardware fixed discount of the target GPU.

[0115] In one example, the above performance discount factor can be represented by the following formula:

[0116] (10)

[0117] in, , Characterizes the GPU model. Characterizing operator units. For the first The model of the GPU. For the first Operator unit, Corresponding operator computational resource requirements Represents the memory access requirements of operators. This represents the resources required for operator communication.

[0118] For the first The operator computation requirements of each operator unit are based on a fixed hardware configuration. Parallel strategy with target Discount factor below, for The operator memory access requirements of each operator unit are based on a fixed hardware configuration. Parallel strategy with target Discount factor below, for The operator communication requirements of each operator unit are based on a fixed hardware configuration. Parallel strategy with target The discount factor below.

[0119] Based on this, the steps described above for determining the performance discount factor under the target parallel strategy by multiplying at least one resource discount of each operator unit with the fixed hardware discount of the target GPU can satisfy the following formula:

[0120] (11)

[0121] in, This represents a hardware fixed discount to the nominal parameters of a GPU, such as This refers to the fixed computing power discount on GPUs due to pipeline issues. For example, resource discounts associated with operator execution. This refers to the GPU computational resource discount for the General Matrix-Vector Multiplication (GEMV) operator.

[0122] Based on this Characterization operator computational resource requirements In fixed hardware configuration Fixed discounts on hardware Characterizing the first Operator computation resource requirements for each operator unit In fixed hardware configuration Under the computing resource discount, Characterization operator computational resource requirements In the target parallel strategy The performance discount factor is applied accordingly. Characterizing the memory access resource requirements of operators In fixed hardware configuration Fixed discounts on hardware Characterizing the first Operator memory access requirements of each operator unit In fixed hardware configuration The following access to memory resources discount, Characterizing the memory access requirements of operators in the target parallel strategy The performance discount factor. Characterizing operator communication resource requirements In fixed hardware configuration Fixed discounts on hardware Characterizing the first Operator communication resource requirements of each operator unit In fixed hardware configuration The following communication resource discounts Characterizing operator communication requirements in target parallel strategies The performance discount factor is applied. It should be noted that memory access... It is less affected by the operator unit.

[0123] It should be noted that, considering the significant differences in the performance discount and shadow discount of the GEMV operator across different input feature dimensions, a fixed hardware discount needs to be determined. Based on this, the discount factor is finely calibrated for each input dimension by measuring or reusing the discount matrix of similar GPUs. Taking GPUA as an example, when measuring its discount factor, the single parallel inference rate in a high-parallel inference scenario is used as the full performance, and the single parallel inference rate in a low-parallel inference scenario is used as the discount factor, resulting in a discount vector as shown below:

[0124] "Gemv_discounts":{"1":0.05,"2":0.05,"4":0.15,"8":0.25,"16":0.25,"32":0.5,"64":0.6,"128":0.6,"256 ":0.7,"512":0.7,"1024":0.8,"2048":0.8,"4096":0.8,"8192":0.9,"16384":0.9,"32768":0.9,"65536":0.9}.

[0125] In this embodiment, by determining the operator resource discounts of multiple operator units under the target parallel strategy and the hardware fixed discount corresponding to the target GPU, the performance discount factor under the target parallel strategy can be accurately calculated based on the product of the resource discount and the hardware fixed discount for each operator unit. This effectively constructs a performance discount factor that accurately reflects the difference between the actual performance of the model to be deployed on the corresponding hardware device and its ideal performance.

[0126] In order to comprehensively and thoroughly describe the method for constructing the inference simulation model provided in the embodiments of this disclosure, in one embodiment, the above-mentioned S140 may specifically include the following steps:

[0127] The first word delay in the pre-filling stage is calculated based on the first sub-running resources of multiple operator units in the pre-filling stage, the hardware performance baseline, and the first discount factor in the pre-filling stage.

[0128] The average single-step latency of the decoding stage is calculated based on the second sub-running resources of multiple operator units in the decoding stage, the hardware performance baseline, and the second discount factor in the decoding stage.

[0129] The end-to-end throughput of the decoding stage is calculated based on the first word delay in the pre-padding stage, the average single-step delay in the decoding stage, and the length of the output sequence.

[0130] Based on the first-word delay in the pre-filling stage and the end-to-end throughput in the decoding stage, an inference simulation model is constructed to evaluate the inference performance of the model to be deployed.

[0131] It should be noted here that the above-mentioned operator running resources may include the first sub-running resources of the pre-filling stage and the second sub-running resources of the decoding stage. Correspondingly, the performance discount factor may include the first discount factor of the pre-filling stage and the second discount factor of the decoding stage.

[0132] Specifically, the inference simulation model building device can first calculate the first word latency of the pre-filling stage based on the first sub-running resources, hardware performance baseline, and first discount factor of the pre-filling stage of multiple operator units respectively. At the same time, it can calculate the single-step average latency of the decoding stage based on the second sub-running resources, hardware performance baseline, and second discount factor of the decoding stage of multiple operator units respectively. It can also calculate the end-to-end throughput of the decoding stage based on the first word latency of the pre-filling stage, the single-step average latency of the decoding stage, and the output sequence length. Finally, it can build an inference simulation model for evaluating the inference performance of the model to be deployed based on the first word latency of the pre-filling stage and the end-to-end throughput of the decoding stage.

[0133] In one example, the first-word delay TTFT during the pre-filling stage can be represented by the following formula:

[0134] (12)

[0135] (13)

[0136] in, This represents the set of first sub-running resources for each operator in the pre-filling phase, and H represents the GPU hardware performance baseline. This represents the performance discount factor for each operator under the first parallel strategy in the pre-filling stage, namely the computation discount factor, memory access discount factor, and communication discount factor caused by the limitations of the GPU architecture for each operator.

[0137] For the first The computational cost of each operator unit in the pre-filling stage (i.e., the aforementioned) ), For the computing power of the GPU, For the first The time overhead of computation for each operator unit. For the first The memory accesses of each operator unit during the pre-filling phase (i.e., the aforementioned) ), This refers to the GPU's memory bandwidth. For the first The time cost of each operator unit accessing the memory. For the first The communication volume of each operator unit in the pre-filling phase (i.e., the aforementioned) ), This refers to the communication bandwidth of the GPU. For the first The time overhead of each operator unit in communication. The H hardware performance baseline can be decomposed into: .

[0138] For the first The first discount factor of each operator unit in the pre-filling stage. Wherein, For the first The operator computation requirements of each operator unit are based on a fixed hardware configuration. Discount factor in the pre-filling stage, for The operator memory access requirements of each operator unit are based on a fixed hardware configuration. Discount factor in the pre-filling stage, for The operator communication requirements of each operator unit are based on a fixed hardware configuration. Discount factor in the pre-filling stage.

[0139] The average single-step delay of the above decoding stage It can be represented by the following formula:

[0140] (14)

[0141] in, It represents the set of second sub-running resources of multiple operator units in the decoding stage. This is the second discount factor under the second parallel strategy in the decoding phase.

[0142] End-to-end throughput of the above decoding stage It can be represented by the following formula:

[0143] (15)

[0144] in, Average generation time for each output token. TTFT and The unit is seconds, while the unit for THOT is tokens per second. This is the length of the output sequence.

[0145] Based on the above, it should be noted that, as Figure 4 As shown, Figure 4The inference performance of GPUA in a scenario with an input sequence length of 1024, an output sequence length of 1024, and a parallel inference of 32 is demonstrated. This shows that as the number of tensor parallels increases, the inference throughput and first-word latency also increase, consistent with the trends shown in Figures 3(a) and 3(b). Furthermore, Figure 4 This also demonstrates the rationality of using first-word delay and end-to-end throughput as the two indicators for the inference simulation model's output.

[0146] In this embodiment, based on the operator running resources required by the multiple operator units in the model inference stage, the hardware performance baseline, and the performance discount factor, the first word latency in the pre-filling stage and the end-to-end throughput in the decoding stage can be accurately constructed. This allows for the construction of an accurate inference simulation model, facilitating the accurate evaluation of the inference performance of the model to be evaluated.

[0147] In order to obtain a more accurate inference simulation model, in one embodiment, the method for constructing the inference simulation model provided by this disclosure may further include the following steps:

[0148] Obtain the measured performance data of the model to be deployed on the target GPU according to the target parallel strategy;

[0149] The inference simulation model is used to predict the inference scenarios corresponding to the measured performance data, and the predicted performance data is obtained.

[0150] If the error information determined based on measured performance data and predicted performance data does not meet the preset conditions, the performance discount factor is updated, the updated performance discount factor is determined as the performance discount factor, and the process returns to the step of constructing an inference simulation model for evaluating the inference performance of the model to be deployed based on the operator running resources, hardware performance baseline and performance discount factor required by multiple operator units in the model inference stage, until the error information meets the preset conditions.

[0151] The error information determined by the measured performance data and the predicted performance data can be used to characterize the degree of deviation between the two, such as the absolute difference or relative difference. Furthermore, the aforementioned preset conditions can be conditions pre-set based on actual experience or circumstances. For example, if the error information determined by the measured performance data and the predicted performance data is the difference between the two, then the preset condition could be that the error between the two is less than a certain threshold; no specific limitation is made here.

[0152] Specifically, it can obtain the measured performance data of the model to be deployed on the target GPU according to the target parallel strategy, and can use the inference simulation model to predict the inference scenario corresponding to the measured performance data to obtain the predicted performance data. Then, it can determine whether the error information between the measured performance data and the corresponding predicted performance data meets the preset conditions. If not, it can update the performance discount factor and determine the updated performance discount factor as the performance discount factor. Then, it returns to the step of executing the steps of constructing an inference simulation model for evaluating the inference performance of the model to be deployed based on the operator running resources, hardware performance baseline and performance discount factor required by multiple operator units in the model inference stage, until the error information meets the preset conditions, and the updated inference simulation model is obtained.

[0153] In this embodiment, after obtaining the inference simulation model, by acquiring measured performance data and predicted performance data, and by determining whether the error information between the measured performance data and the corresponding predicted performance data meets the preset conditions, if the error between the two is large, the performance discount factor can be updated to achieve continuous optimization of the inference simulation model and obtain a more accurate inference simulation model.

[0154] Accordingly, in one embodiment, since the performance data may include at least one of first-word latency and end-to-end throughput, the measured performance data may include at least one of measured first-word latency and measured end-to-end throughput, and the predicted performance data may include at least one of predicted first-word latency and predicted end-to-end throughput.

[0155] Based on this, when the error information determined based on measured performance data and predicted performance data does not meet the preset conditions, before updating the performance discount factor, the method for constructing the inference simulation model provided in this embodiment of the disclosure may specifically include the following steps:

[0156] Based on the measured first character delay and the predicted first character delay, the first error information is determined;

[0157] The second error information is determined based on the measured end-to-end throughput and the predicted end-to-end throughput.

[0158] If the first error information does not meet the first preset condition, and / or the second error information does not meet the second preset condition, then the error information is determined to not meet the preset condition.

[0159] It should be noted that the first error information between the measured first-letter delay and the predicted first-letter delay can be used to characterize the degree of deviation between the measured and predicted first-letter delays, such as the absolute or relative difference between the two. Furthermore, the aforementioned first preset condition can be a condition pre-set based on actual experience or circumstances. For example, if the error information determined by the measured and predicted first-letter delays is the difference between the two, then the preset condition could be that the error between the two is less than a certain threshold; no specific limitation is made here. The second error information between the measured end-to-end throughput and the predicted end-to-end throughput is similar to the first error information, and the second preset condition is similar to the first preset condition, so it will not be repeated here.

[0160] Specifically, it is possible to determine the first error information based on the measured first-word delay and the predicted first-word delay, and to determine the second error information based on the measured end-to-end throughput and the predicted end-to-end throughput. Thus, it is possible to determine that the error information does not meet the preset conditions if the first error information does not meet the first preset condition, and / or if the second error information does not meet the second preset condition.

[0161] In one example, the above error information It can be represented by the following formula:

[0162] (16)

[0163] Where N is the total number of samples, that is, the number of measured data points (including measured first-word delay and measured end-to-end throughput) or predicted data points (including predicted first-word delay and predicted end-to-end throughput) used to calculate error information. For the first A measured first-word delay or a measured end-to-end throughput. For the first Predicting the first word latency or predicting end-to-end throughput. .

[0164] In this embodiment, since the performance data may include at least one of the first-word delay and the end-to-end throughput, the first error information can be determined based on the measured first-word delay and the predicted first-word delay, and the second error information can be determined based on the measured end-to-end throughput and the predicted end-to-end throughput. Then, based on the first error information and the second error information, it can be accurately determined whether the inference simulation model needs to be optimized.

[0165] In one embodiment, the step of updating the performance discount factor described above may specifically include:

[0166] Update hardware fixed discounts according to preset update steps;

[0167] Based on the updated hardware fixed discount and operator resource discount, the performance discount factor of the model to be deployed under the target parallel strategy is re-determined.

[0168] The preset update step size can be set in advance based on actual experience or circumstances, and no specific limitation is made here.

[0169] Specifically, the device for building the inference simulation model can update the hardware fixed discount according to a preset update step size, and then redetermine the performance discount factor of the model to be deployed under the target parallel strategy based on the updated hardware fixed discount and operator resource discount.

[0170] In one example, the above follows a preset update step size (the preset update step size can be set to...). Hardware upgrade fixed discount Steps, fixed discount on updated hardware The following formula can be satisfied:

[0171] (17)

[0172] in, Learning rate It is 0.1, until... Alternatively, the maximum number of iterations can be reached, up to 20 steps, where c can be determined based on the actual situation; for example, c can be set to 5. This allows for a gradual reduction of the performance discount factor. In The components are calculated until the error converges to ≤c%, without specific limitations here.

[0173] It should be noted that in Figures 5(a) and 5(c), the horizontal axis represents the measured first-word delay, and the vertical axis represents the predicted first-word delay. In Figures 5(b) and 5(d), the horizontal axis represents the measured end-to-end throughput, i.e., the average time per output token in the measured inference, and the vertical axis represents the predicted end-to-end throughput, i.e., the average time per output token in the simulation. Based on this, Figure 5(a) shows a comparison between the measured and predicted first-word delays before introducing the performance discount factor, and Figure 5(b) shows a comparison between the measured and predicted end-to-end throughputs before introducing the performance discount factor. Based on the content shown in Figure 5(a), it can be seen that before introducing the performance discount factor, the error between the measured and predicted first-word delays (i.e., the first error information of the first-word delay) can reach 45.3%. Based on the content shown in Figure 5(b), it can be seen that before introducing the performance discount factor, the error between the measured and predicted end-to-end throughputs (i.e., the second error information of the end-to-end throughput) can reach 243.5%.

[0174] Figure 5(c) shows a comparison between the measured first-word delay and the predicted first-word delay after introducing the performance discount factor, and Figure 5(d) shows a comparison between the measured end-to-end throughput and the predicted end-to-end throughput after introducing the performance discount factor. As shown in Figures 5(c) and 5(d), after introducing the performance discount factor, the error between the measured inference performance and the simulated inference performance of the inference simulation model is reduced. Taking Figure 5(d) as an example, the error between the measured end-to-end throughput and the predicted end-to-end throughput can be reduced to 3.9%, and the fitting curve has a slope close to 45 degrees, ensuring the consistency between the measured end-to-end throughput and the predicted end-to-end throughput. This verifies that the "explicit-implicit joint" model has better physical consistency than the single peak parameter model. It also shows that the discount factor search process is not only mathematically convergent but also consistent with the trend of the measured data, ensuring the engineering replicability of the "simulation-measurement-calibration" closed loop. The results obtained based on the method provided in this embodiment have a high degree of agreement with the measured results, which is beneficial for accurate performance prediction of untested inference scenarios based on the inference simulation model. In this embodiment of the disclosure, the learning rate It can be set to 0.1, and the iteration termination condition can be... Or it can reach the maximum number of iterations, 20 steps.

[0175] In addition, this disclosure provides a data processing method, which will be described below in conjunction with... Figure 6 The data processing method provided in the embodiments of this disclosure will be described in detail.

[0176] Figure 6 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure.

[0177] like Figure 6 As shown, the entity executing this data processing method can be a data processing device. Based on this, the data processing method provided in this embodiment of the disclosure may specifically include the following steps:

[0178] S610, acquire the data to be processed.

[0179] S620, utilizing Figure 1 The inference simulation model constructed using the methods and steps shown is used to process the data to be processed, and the output results of the inference simulation model are obtained.

[0180] In some embodiments, the data to be processed includes operator resource parameters of the target deployment model, target inference scenario data, and hardware configuration information of the actual graphics processing unit (GPU) used to deploy the target deployment model.

[0181] In addition, the above output results include inference first-word latency, end-to-end inference throughput, and model parallelism strategy. These output results are used to characterize the inference performance of the target deployment model in the target scenario, where the target scenario is determined based on the target inference scenario data and the actual GPU hardware configuration information.

[0182] In this embodiment, the data processing device can acquire data to be processed, which may include operator resource parameters of the target deployment model, target inference scene data, and hardware configuration information of the actual graphics processing unit (GPU) used to deploy the target deployment model. The data is then processed using input values ​​of the operator resource parameters of the target deployment model, target inference scene data, and hardware configuration information of the actual GPU used to deploy the target deployment model. Figure 1 The method and steps shown construct the inference simulation model for use. Figure 1 The method and steps shown demonstrate how to construct an inference simulation model to process the data to be processed, thereby obtaining the output of the inference simulation model. This allows for an accurate and effective evaluation of the target model's inference performance.

[0183] The method provided in this disclosure constructs a complete framework encompassing "operator-level requirement modeling—hardware-level discount factor—closed-loop calibration optimization." During the simulation phase, bottlenecks in computation, memory access, and communication are decoupled and quantified. A performance discount factor is used to simultaneously integrate factors such as GPU explicit computing power, bandwidth, implicit pipeline, and instruction issuance efficiency. After online deployment, MAPE is automatically calculated based on measured TTFT / THOT, and a reverse performance discount factor is applied, forming an iterative closed loop of "simulation—measurement—calibration," stabilizing the error. Compared to traditional methods, this reduces the selection cycle from weeks to minutes, significantly lowering trial-and-error costs. It also supports pre-evaluation of strategies such as FP16 / FP8 / FP4 multi-precision, TP / DP hybrid parallelism, and key-value cache compression (KV-Cache), effectively improving prediction accuracy and resource utilization in large-model inference scenarios.

[0184] It's important to note that the core of this solution lies in constructing a "performance digital twin"—breaking down complex workloads into operator-level requirements, reflecting explicit hardware parameters and implicit micro-losses through a performance discount factor, and performing reverse calibration using measured metrics. Essentially, it can be transferred to other heterogeneous acceleration scenarios. For example, in the supercomputing field, simply replacing pre-filling / decoding with computationally / communication-intensive areas, replacing multi-head attention operators with high-performance computing (HPC) operators such as matrix multiplication, Fast Fourier Transform (FFT), and Stencil Computation (Stencil), while maintaining the performance discount factor (computing power-memory-interconnect) architecture, and adjusting the closed-loop calibration metrics to time consumption per iteration and parallel efficiency, allows for the prediction of performance scaling curves for clusters with tens of millions of cores within minutes, guiding selection and task partitioning.

[0185] Based on the same inventive concept, the present disclosure provides a device for constructing a reasoning simulation model, which can be specifically described in conjunction with the appendix. Figure 7 This disclosure provides a detailed description of an apparatus for constructing a reasoning simulation model according to an embodiment.

[0186] Figure 7 This is a schematic diagram of the structure of a device for constructing a reasoning simulation model provided in an embodiment of this disclosure.

[0187] like Figure 7 As shown, the inference simulation model construction device 700 may include:

[0188] The acquisition module 710 is used to acquire the operator runtime resources required by multiple operator units in the model to be deployed during the model inference stage.

[0189] The acquisition module 710 is also used to acquire the hardware performance baseline of the target graphics processing unit (GPU);

[0190] The determination module 720 is also used to determine the performance discount factor of the model to be deployed under the target parallel strategy. The performance discount factor is determined based on the product of the hardware fixed discount and the operator resource discount corresponding to the target GPU. The hardware fixed discount represents the inherent performance loss of the target GPU, and the operator resource discount represents the performance discount degree of the operator unit.

[0191] Module 730 is used to build an inference simulation model for evaluating the inference performance of the model to be deployed, based on the operator running resources, hardware performance baseline and performance discount factor required by multiple operator units in the model inference stage.

[0192] In one embodiment, the apparatus for constructing a reasoning simulation model provided in this disclosure includes:

[0193] The determination module is also used to determine the resource computation model for each operator unit during the model inference phase;

[0194] The determination module is also used to calculate the model based on the resources of each operator unit in the model inference phase, and to determine the operator running resources of the operator unit in the model inference phase.

[0195] The operator runtime resources include at least one of the following: operator computational resources, operator memory access resources, and operator communication resources.

[0196] In one embodiment, the model inference stage includes a pre-filling stage and a decoding stage, and the resource computing model corresponding to the operator unit includes a first resource computing model of the operator unit in the pre-filling stage and a second resource computing model of the operator unit in the decoding stage; the inference simulation model construction apparatus provided in this embodiment includes:

[0197] The determination module is specifically used to determine the first sub-running resource of the operator unit in the pre-filling stage based on the first resource calculation model of each operator unit in the pre-filling stage;

[0198] The determination module is specifically used to determine the second sub-running resources of the operator unit in the decoding stage based on the second resource calculation model of each operator unit in the decoding stage;

[0199] The operator running resources corresponding to the operator unit include the first sub-running resources of the operator unit in the pre-filling stage and the second sub-running resources of the operator unit in the decoding stage.

[0200] In one embodiment, the operator runtime resources are determined based on tensor parallel variables and data parallel variables, and the operator runtime resources include at least one of operator memory access requirements and operator communication requirements; the inference simulation model construction apparatus provided in this disclosure includes:

[0201] The determination module is also used to determine multiple candidate parallel strategies for the model to be deployed during the model inference phase, each candidate parallel strategy including tensor parallelism and data parallelism.

[0202] The computation module is used to calculate the estimated operator running resources of each operator unit under the candidate parallel strategy, based on the number of tensor parallelisms, the number of data parallelisms, and the operator running resources of each operator unit in the candidate parallel strategy.

[0203] The determination module is also used to determine the target parallel strategy from multiple candidate parallel strategies with the goal of minimizing the sum of the time overhead generated by the pre-estimated sub-operator resources under the candidate parallel strategies.

[0204] The model inference stage includes a pre-filling stage and a decoding stage, and the target parallel strategy includes a first parallel strategy for the pre-filling stage and a second parallel strategy for the decoding stage.

[0205] In one embodiment, the apparatus for constructing a reasoning simulation model provided in this disclosure includes:

[0206] The determination module is specifically used to determine the operator resource discounts of multiple operator units under the target parallel strategy. The operator resource discounts include at least one resource discount, and the at least one resource discount includes at least one of computational resource discount, memory access resource discount, and communication resource discount.

[0207] The determination module is specifically used to determine the fixed hardware discount corresponding to the target GPU;

[0208] The determination module is specifically used to determine the performance discount factor under the target parallel strategy based on the product between at least one resource discount of each operator unit and the hardware fixed discount of the target GPU.

[0209] In one embodiment, the apparatus for constructing a reasoning simulation model provided in this disclosure includes:

[0210] The calculation module is used to calculate the first word delay in the pre-filling stage based on the first sub-running resources of multiple operator units in the pre-filling stage, the hardware performance baseline, and the first discount factor in the pre-filling stage.

[0211] The calculation module is also used to calculate the single-step average latency of the decoding stage based on the second sub-running resources of multiple operator units in the decoding stage, the hardware performance baseline, and the second discount factor in the decoding stage.

[0212] The calculation module is also used to calculate the end-to-end throughput of the decoding stage based on the first word delay in the pre-padding stage, the average single-step delay in the decoding stage, and the length of the output sequence.

[0213] The building module is specifically used to build an inference simulation model for evaluating the inference performance of the model to be deployed, based on the first-word delay in the pre-filling stage and the end-to-end throughput in the decoding stage.

[0214] The operator runtime resources include the first sub-running resources of the pre-filling stage and the second sub-running resources of the decoding stage, and the performance discount factor includes the first discount factor of the pre-filling stage and the second discount factor of the decoding stage.

[0215] In one embodiment, the apparatus for constructing a reasoning simulation model provided in this disclosure further includes:

[0216] The acquisition module is also used to acquire the measured performance data of the model to be deployed on the target GPU according to the target parallel strategy;

[0217] The prediction module is used to predict the inference scenarios corresponding to the measured performance data using the inference simulation model, and obtain the predicted performance data.

[0218] The update module is used to update the performance discount factor when the error information determined based on measured performance data and predicted performance data does not meet the preset conditions. The updated performance discount factor is determined as the performance discount factor, and the steps of executing the steps of constructing an inference simulation model for evaluating the inference performance of the model to be deployed based on the operator running resources, hardware performance baseline and performance discount factor required by multiple operator units in the model inference stage are returned until the error information meets the preset conditions.

[0219] In one embodiment, the performance data includes at least one of first-word latency and end-to-end throughput; the apparatus for constructing the inference simulation model provided in this disclosure includes:

[0220] The determination module is also used to determine the first error information based on the measured first character delay and the predicted first character delay;

[0221] The determination module is also used to determine the second error information based on the measured end-to-end throughput and the predicted end-to-end throughput;

[0222] The determining module is further configured to determine that the error information does not meet the preset conditions when the first error information does not meet the first preset condition and / or the second error information does not meet the second preset condition.

[0223] In one embodiment, the apparatus for constructing a reasoning simulation model provided in this disclosure includes:

[0224] The update module is specifically used to update the fixed hardware discount according to a preset update step size;

[0225] The determination module is also used to determine the performance discount factor of the model to be deployed under the target parallel strategy based on the updated hardware fixed discount and operator resource discount.

[0226] It is understood that the inference simulation model construction apparatus provided in the above embodiments, when implementing the corresponding inference simulation model construction method, can allocate the above processing to different program modules as needed to complete all or part of the processing described above. Furthermore, the apparatus and corresponding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0227] Based on the same inventive concept, this disclosure provides a data processing apparatus, which can be specifically described in conjunction with the appendix. Figure 8 A data processing apparatus provided in the embodiments of this disclosure will be described in detail.

[0228] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this disclosure.

[0229] like Figure 8 As shown, the data processing device 800 may include:

[0230] The acquisition module 810 is used to acquire data to be processed, which includes operator resource parameters of the target deployment model, target inference scene data, and hardware configuration information of the actual graphics processing unit (GPU) used to deploy the target deployment model.

[0231] Processing module 820, used to utilize Figure 1 The inference simulation model constructed by the method steps shown processes the data to be processed and obtains the output results of the inference simulation model. The output results include inference first-word latency, end-to-end inference throughput, and model parallelism strategy. The output results are used to characterize the inference performance of the target deployment model in the target scenario. The target scenario is determined based on the target inference scenario data and the actual GPU hardware configuration information.

[0232] It is understood that, when implementing the corresponding data processing method, the data processing apparatus provided in the above embodiments can allocate the processing to different program modules as needed to complete all or part of the processing described above. Furthermore, the apparatus and the corresponding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0233] This disclosure provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the inference simulation model construction method or data processing method provided in this disclosure.

[0234] This disclosure provides a computer-readable storage medium storing executable instructions, wherein the executable instructions, when executed by a processor, will cause the processor to execute the inference simulation model construction method or data processing method provided in this disclosure.

[0235] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0236] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0237] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0238] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0239] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure; as shown below. Figure 9 As shown, the electronic device 90 includes: a processor 901, and a memory 902 communicatively connected to the processor 901; the memory 902 stores instructions executable by the processor 901. The instructions are executed by the processor 901 to enable the processor 901 to perform:

[0240] Obtain the operator runtime resources required by multiple operator units in the model to be deployed during the model inference phase;

[0241] Obtain the hardware performance baseline of the target graphics processing unit (GPU);

[0242] The performance discount factor of the model to be deployed under the target parallel strategy is determined. The performance discount factor is determined based on the product of the hardware fixed discount and the operator resource discount corresponding to the target GPU. The hardware fixed discount represents the inherent performance loss of the target GPU, and the operator resource discount represents the performance discount degree of the operator unit.

[0243] Based on the operator running resources required by the multiple operator units in the model inference stage, the hardware performance baseline, and the performance discount factor, an inference simulation model for the inference performance of the model to be deployed is constructed.

[0244] Alternatively, the instruction is executed by processor 901 to enable processor 901 to perform the following:

[0245] Acquire the data to be processed, which includes the operator resource parameters of the target deployment model, the target inference scenario data, and the hardware configuration information of the actual graphics processing unit (GPU) used to deploy the target deployment model;

[0246] use Figure 1 The inference simulation model constructed by the method steps shown processes the data to be processed to obtain the output result of the inference simulation model. The output result includes inference first-word latency, end-to-end inference throughput, and model parallel strategy. The output result is used to characterize the inference performance of the target deployment model in the target scenario. The target scenario is determined based on the target inference scenario data and the actual GPU hardware configuration information.

[0247] The electronic devices and the corresponding inference simulation model construction methods or data processing methods provided in the above embodiments belong to the same concept. For details of their implementation process, please refer to the method embodiments, which will not be repeated here.

[0248] In practical applications, the electronic device 90 may further include at least one network interface 903. The various components of the electronic device 90 are coupled together via a bus system 904. It is understood that the bus system 904 is used to implement communication between these components. In addition to a data bus, the bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 9 All buses are labeled as bus system 904. The number of processors 901 and the number of memories 902 can be at least one. The network interface 903 is used for wired or wireless communication between the electronic device 90 and other devices.

[0249] The memory 902 in this embodiment is used to store various types of data to support the operation of the electronic device 90.

[0250] The methods disclosed in the above embodiments of this disclosure can be applied to or implemented by processor 901. Processor 901 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 901 or by instructions in software form. The processor 901 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 901 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 902. Processor 901 reads the information in memory 902 and, in conjunction with its hardware, completes the steps of the aforementioned inference simulation model construction method or data processing method.

[0251] In some embodiments, the electronic device 90 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned methods.

[0252] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0253] In the above description, the term "some embodiments" refers to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0254] Unless otherwise defined, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used in this disclosure is for the purpose of describing embodiments of this disclosure only and is not intended to be limiting of this disclosure.

[0255] It should be understood that in the various embodiments of this disclosure, the sequence number of each implementation process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0256] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0257] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for constructing a reasoning simulation model, characterized in that, The method includes: Obtain the operator runtime resources required by multiple operator units in the model to be deployed during the model inference phase; Obtain the hardware performance baseline of the target graphics processing unit (GPU); The performance discount factor of the model to be deployed under the target parallel strategy is determined. The performance discount factor is determined based on the product of the hardware fixed discount and the operator resource discount corresponding to the target GPU. The hardware fixed discount represents the inherent performance loss of the target GPU, and the operator resource discount represents the performance discount degree of the operator unit. Based on the operator running resources required by the multiple operator units in the model inference stage, the hardware performance baseline, and the performance discount factor, an inference simulation model for evaluating the inference performance of the model to be deployed is constructed. Determining the performance discount factor of the model to be deployed under the target parallel strategy includes: Determine the operator resource discounts for the plurality of operator units under the target parallel strategy, wherein the operator resource discounts include at least one resource discount, and the at least one resource discount includes at least one of computation resource discount, memory access resource discount, and communication resource discount; Determine the fixed hardware discount corresponding to the target GPU; The performance discount factor under the target parallel strategy is determined based on the product of at least one resource discount of each operator unit and the hardware fixed discount of the target GPU. The inference simulation model for evaluating the inference performance of the model to be deployed is constructed based on the operator runtime resources required by the multiple operator units in the model inference stage, the hardware performance baseline, and the performance discount factor, including: The first word delay of the pre-filling stage is calculated based on the first sub-running resources of the multiple operator units in the pre-filling stage, the hardware performance baseline, and the first discount factor of the pre-filling stage. Based on the second sub-running resources of the multiple operator units in the decoding stage, the hardware performance baseline, and the second discount factor in the decoding stage, the single-step average latency of the decoding stage is calculated. Based on the first-word delay of the pre-filling stage, the average single-step delay of the decoding stage, and the length of the output sequence, the end-to-end throughput of the decoding stage is calculated. Based on the first-word latency of the pre-filling stage and the end-to-end throughput of the decoding stage, an inference simulation model is constructed to evaluate the inference performance of the model to be deployed. The operator running resources include the first sub-running resources of the pre-filling stage and the second sub-running resources of the decoding stage, and the performance discount factor includes the first discount factor of the pre-filling stage and the second discount factor of the decoding stage.

2. The method according to claim 1, characterized in that, The acquisition of operator runtime resources required by multiple operator units in the model to be deployed during the model inference phase includes: Determine the resource computation model for each operator unit during the model inference phase; Based on the resource computation model of each operator unit in the model inference phase, determine the operator operation resources of the operator unit in the model inference phase; The operator operating resources include at least one of the following: operator computational resources, operator memory access resources, and operator communication resources.

3. The method according to claim 2, characterized in that, The model inference stage includes a pre-filling stage and a decoding stage. The resource calculation model corresponding to the operator unit includes a first resource calculation model of the operator unit in the pre-filling stage and a second resource calculation model of the operator unit in the decoding stage. The method of determining the operator execution resources of each operator unit during the model inference phase based on the resource computation model of each operator unit during the model inference phase includes: Based on the first resource calculation model of each operator unit in the pre-filling stage, the first sub-running resource of the operator unit in the pre-filling stage is determined; Based on the second resource calculation model of each operator unit in the decoding stage, the second sub-running resources of the operator unit in the decoding stage are determined; The operator running resources corresponding to the operator unit include the first sub-running resources of the operator unit in the pre-filling stage and the second sub-running resources of the operator unit in the decoding stage.

4. The method according to claim 1, characterized in that, The operator runtime resources are determined based on tensor parallel variables and data parallel variables, and the operator runtime resources include at least one of operator memory access requirements and operator communication requirements. Before determining the performance discount factor of the model to be deployed under the target parallel strategy, the method further includes: Multiple candidate parallel strategies are determined for the model to be deployed during the model inference phase, and each candidate parallel strategy includes tensor parallelism and data parallelism. For each candidate parallel strategy, based on the number of tensor parallelisms, the number of data parallelisms, and the operator running resources of each operator unit in the candidate parallel strategy, the estimated operator running resources of the operator unit under the candidate parallel strategy are calculated. The target parallel strategy is determined from the multiple candidate parallel strategies with the objective of minimizing the sum of the time overhead generated by the predicted sub-operator resources under the candidate parallel strategies. The model inference stage includes a pre-filling stage and a decoding stage, and the target parallel strategy includes a first parallel strategy for the pre-filling stage and a second parallel strategy for the decoding stage.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the measured performance data of the model to be deployed on the target GPU according to the target parallel strategy; The inference simulation model is used to predict the inference scenario corresponding to the measured performance data to obtain the predicted performance data. If the error information determined based on the measured performance data and the predicted performance data does not meet the preset conditions, the performance discount factor is updated, the updated performance discount factor is determined as the performance discount factor, and the step of constructing an inference simulation model for evaluating the inference performance of the model to be deployed is returned to the execution of the step of constructing an inference simulation model based on the operator running resources required by the multiple operator units in the model inference stage, the hardware performance baseline and the performance discount factor, until the error information meets the preset conditions.

6. The method according to claim 5, characterized in that, Performance data includes at least one of first-word latency and end-to-end throughput; before updating the performance discount factor when the error information determined based on the measured performance data and the predicted performance data does not meet preset conditions, the method further includes: Based on the measured first character delay and the predicted first character delay, the first error information is determined; The second error information is determined based on the measured end-to-end throughput and the predicted end-to-end throughput. If the first error information does not meet the first preset condition, and / or the second error information does not meet the second preset condition, then it is determined that the error information does not meet the preset condition.

7. The method according to claim 5, characterized in that, The update of the performance discount factor includes: Update the hardware fixed discount according to the preset update step size; Based on the updated hardware fixed discount and the operator resource discount, the performance discount factor of the model to be deployed under the target parallel strategy is re-determined.

8. A data processing method, characterized in that, The method includes: Acquire the data to be processed, which includes the operator resource parameters of the target deployment model, the target inference scenario data, and the hardware configuration information of the actual graphics processing unit (GPU) used to deploy the target deployment model; The inference simulation model constructed using any one of the method steps shown in claims 1 to 7 is used to process the data to be processed, and the output result of the inference simulation model is obtained. The output result includes inference first-word latency, end-to-end inference throughput and model parallel strategy. The output result is used to characterize the inference performance of the target deployment model in the target scenario. The target scenario is determined based on the target inference scenario data and the actual GPU hardware configuration information.

9. A device for constructing a reasoning simulation model, characterized in that, The device includes: The acquisition module is used to acquire the operator runtime resources required by multiple operator units in the model to be deployed during the model inference phase. The acquisition module is also used to acquire the hardware performance baseline of the target graphics processing unit (GPU); The determination module is further configured to determine the performance discount factor of the model to be deployed under the target parallel strategy. The performance discount factor is determined based on the product of the hardware fixed discount and the operator resource discount corresponding to the target GPU. The hardware fixed discount represents the inherent performance loss of the target GPU, and the operator resource discount represents the performance discount degree of the operator unit. A construction module is used to construct an inference simulation model for evaluating the inference performance of the model to be deployed, based on the operator running resources required by the multiple operator units in the model inference stage, the hardware performance baseline, and the performance discount factor. The determining module is specifically used to determine the operator resource discounts of the plurality of operator units under the target parallel strategy, wherein the operator resource discounts include at least one resource discount, and the at least one resource discount includes at least one of computational resource discount, memory access resource discount, and communication resource discount; The determining module is specifically used to determine the fixed hardware discount corresponding to the target GPU; The determining module is specifically used to determine the performance discount factor under the target parallel strategy based on the product between at least one resource discount of each operator unit and the hardware fixed discount of the target GPU. The calculation module is used to calculate the first word delay of the pre-filling stage based on the first sub-running resources of the plurality of operator units in the pre-filling stage, the hardware performance baseline, and the first discount factor of the pre-filling stage. The calculation module is also used to calculate the single-step average latency of the decoding stage based on the second sub-running resources of the plurality of operator units in the decoding stage, the hardware performance baseline, and the second discount factor of the decoding stage. The calculation module is also used to calculate the end-to-end throughput of the decoding stage based on the first word delay of the pre-filling stage, the single-step average delay of the decoding stage, and the length of the output sequence. The construction module is specifically used to construct an inference simulation model for evaluating the inference performance of the model to be deployed, based on the first-word delay of the pre-filling stage and the end-to-end throughput of the decoding stage. The operator running resources include the first sub-running resources of the pre-filling stage and the second sub-running resources of the decoding stage, and the performance discount factor includes the first discount factor of the pre-filling stage and the second discount factor of the decoding stage.

10. The apparatus according to claim 9, characterized in that, The device includes: The determining module is also used to determine the resource computing model of each operator unit in the model inference stage; The determining module is further configured to determine the operator running resources of the operator unit in the model inference stage based on the resource calculation model of each operator unit in the model inference stage; The operator operating resources include at least one of the following: operator computational resources, operator memory access resources, and operator communication resources.

11. The apparatus according to claim 10, characterized in that, The model inference stage includes a pre-filling stage and a decoding stage. The resource computation model corresponding to the operator unit includes a first resource computation model of the operator unit in the pre-filling stage and a second resource computation model of the operator unit in the decoding stage. The device includes: The determining module is specifically used to determine the first sub-running resource of the operator unit in the pre-filling stage based on the first resource calculation model of each operator unit in the pre-filling stage; The determining module is specifically used to determine the second sub-running resources of the operator unit in the decoding stage based on the second resource calculation model of each operator unit in the decoding stage; The operator running resources corresponding to the operator unit include the first sub-running resources of the operator unit in the pre-filling stage and the second sub-running resources of the operator unit in the decoding stage.

12. The apparatus according to claim 9, characterized in that, The operator runtime resources are determined based on tensor parallel variables and data parallel variables, and the operator runtime resources include at least one of operator memory access requirements and operator communication requirements; the apparatus includes: The determining module is further configured to determine multiple candidate parallel strategies for the model to be deployed during the model inference phase, each candidate parallel strategy including tensor parallelism and data parallelism. The calculation module is used to calculate the estimated operator running resources of the operator unit under the candidate parallel strategy for each candidate parallel strategy, based on the tensor parallelism, data parallelism and operator running resources of the candidate parallel strategy. The determining module is further configured to determine the target parallel strategy from the multiple candidate parallel strategies, with the goal of minimizing the sum of the time overhead generated by the predicted sub-operator resources of the multiple operator units under the candidate parallel strategies respectively; The model inference stage includes a pre-filling stage and a decoding stage, and the target parallel strategy includes a first parallel strategy for the pre-filling stage and a second parallel strategy for the decoding stage.

13. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire data to be processed, which includes operator resource parameters of the target deployment model, target inference scenario data, and hardware configuration information of the actual graphics processing unit (GPU) used to deploy the target deployment model. The processing module is used to process the data to be processed using the inference simulation model constructed by the method steps shown in any one of claims 1 to 7, and to obtain the output result of the inference simulation model. The output result includes inference first-word latency, end-to-end inference throughput and model parallel strategy. The output result is used to characterize the inference performance of the target deployment model in the target scenario. The target scenario is determined based on the target inference scenario data and the actual GPU hardware configuration information.

Images