A hierarchical modeling method based on a large language model

Abstract

The application discloses a hierarchical modeling method based on a large language model, comprising the following steps: performing task division on the large language model according to obtained configuration parameters of model running to obtain card-in analysis tasks of each acceleration card; wherein the configuration parameters of model running comprise a parallel strategy, an optimization strategy, model parameters and input data of the large language model; performing card-in task division on each acceleration card according to obtained hardware parameters and the card-in analysis tasks of each acceleration card to obtain core-in analysis tasks corresponding to each computing core; performing performance modeling on the core-in analysis tasks of each computing core according to the hardware parameters to obtain performance data of each computing core; and integrating the performance data of each computing core according to the parallel strategy to obtain performance data of the large language model; and the application realizes multi-level performance modeling and unified evaluation of the large language model under multiple types of heterogeneous hardware architectures, and improves the precision, expansibility and computing resource utilization efficiency of performance analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a hierarchical modeling method based on a large language model. BACKGROUND

[0002] With the rapid development of artificial intelligence (AI) technology, large language models are undergoing evolution from single modality to multi-modality, from traditional transformer architecture to mixture of experts architecture (MoE), and their parameter scale is gradually expanding from million to trillion. In this evolution process, how to quickly obtain the performance of large models on different hardware architectures, especially GPU clusters or self-developed accelerator chip platforms, has become a technical problem that needs to be solved. For AI infrastructure providers, the new generation of model architecture is usually accompanied by new parallel computing strategies, operator fusion methods, and computing, memory, and communication masking technologies, so an efficient and reliable performance evaluation method is needed to judge the applicability and actual benefits of these innovative solutions under specific hardware conditions.

[0003] The existing performance evaluation methods mainly include three categories. First, the actual test-based method, that is, by directly running the model in the target hardware environment to obtain performance data. Although this method can obtain relatively realistic results, when the software environment, hardware configuration, network topology or model size changes, repeated testing is often required, which is time-consuming, labor-intensive and resource-consuming. Second, the experience-based method, that is, by referring to the performance of existing models on similar hardware to infer the performance of new models. However, with the continuous improvement of model and hardware architecture complexity, the reliability of experience-based inference gradually decreases. Third, the modeling analysis-based method, that is, by using system parameters and model parameters to build a theoretical model to estimate the execution performance of the model. This method alleviates the shortcomings of actual measurement and experience methods to some extent, but due to the lack of hierarchical abstract modeling ideas, its scope of application is limited. When new model types or new hardware structures appear, the analysis process often needs to be redesigned, resulting in low efficiency and difficulty in meeting the needs of rapid iteration of large models and continuous evolution of hardware architecture.

[0004] In summary, the existing performance evaluation techniques have obvious limitations when facing the simultaneous development of large models and hardware architectures. The actual test cost is too high, the experience-based inference lacks reliability, and the traditional modeling method lacks scalability, all of which make it difficult to meet the technical needs of efficient performance analysis of large models on diversified hardware platforms.

[0005] This section is intended to provide background or context to the embodiments of the application recited in the claims. The description herein does not constitute admission of prior art. SUMMARY

[0006] The embodiments of the present application provide a hierarchical modeling method based on a large language model, which is used to realize multi-level performance modeling and unified evaluation of the large language model under a heterogeneous hardware architecture. Through hierarchical modeling and standardized interface design, the accuracy, scalability and computing resource utilization efficiency of performance analysis are improved.

[0007] The hierarchical modeling method based on the large language model comprises:

[0008] According to the obtained model running configuration parameters, task division is performed on the large language model to obtain card-in analysis tasks of each acceleration card; wherein the model running configuration parameters include: parallel strategy, optimization strategy, model parameters and input data of the large language model;

[0009] According to the obtained hardware parameters and the card-in analysis tasks of each acceleration card, card-in task division is performed on each acceleration card to obtain core-in analysis tasks corresponding to each computing core;

[0010] According to the hardware parameters and the model running configuration parameters, performance modeling is performed on the core-in analysis tasks of each computing core to obtain performance data of each computing core;

[0011] The performance data of each computing core is integrated according to the parallel strategy to obtain performance data of the large language model.

[0012] In some embodiments, the parallel strategy includes a card-level parallel strategy and a core-level parallel strategy, and the card-level parallel strategy includes a model parallel strategy; the task division on the large language model according to the obtained model running configuration parameters to obtain card-in analysis tasks of each acceleration card comprises:

[0013] According to the model parallel strategy, the model parameters are divided to obtain a plurality of model parameter subsets;

[0014] The model parameter subsets are respectively mapped to the corresponding acceleration cards to obtain the card-in analysis tasks of each acceleration card.

[0015] In some embodiments, the parallel strategy includes a card-level parallel strategy and a core-level parallel strategy, and the card-level parallel strategy further includes a data parallel strategy; the task division on the large language model according to the obtained model running configuration parameters to obtain card-in analysis tasks of each acceleration card comprises:

[0016] According to the batch data amount of the input data and the data parallel strategy, the input data is divided to obtain a plurality of data sub-samples;

[0017] The data sub-samples are respectively mapped to corresponding acceleration cards to obtain card-in analysis tasks corresponding to each acceleration card.

[0018] In some embodiments, the performance data of each computing core includes vector calculation execution time of a vector calculation unit, the hardware parameters include: vector computing power parameters of the vector calculation unit in each computing core, first startup delay of the vector calculation unit in each computing core, and first hardware utilization rate of the vector calculation unit in each computing core; the model running configuration parameters further include: data block size in each computing core, and the performance modeling of the in-core analysis task of each computing core according to the hardware parameters and the model running configuration parameters obtains the performance data of each computing core, including:

[0019] According to the data block size, the vector computing power parameters, the startup delay, the hardware utilization rate, and the vector modeling model, the vector calculation execution time of the vector calculation unit in each computing core is determined.

[0020] In some embodiments, the performance data of each computing core includes matrix calculation execution time of a matrix calculation unit in each computing core, the hardware parameters include: tensor computing power parameters of the matrix calculation unit in each computing core, second startup delay of the matrix calculation unit in each computing core, and second hardware utilization rate of the matrix calculation unit in each computing core; the model running configuration parameters further include: left matrix block of the matrix calculation unit in each computing core and right matrix block of the matrix calculation unit in each computing core; the performance modeling of the in-core analysis task of each computing core according to the hardware parameters and the model running configuration parameters obtains the performance data of each computing core, including:

[0021] According to the right matrix block, the left matrix block, the tensor computing power parameters, the second startup delay, the second hardware utilization rate, and the matrix modeling model, the matrix calculation execution time of the matrix calculation unit in each computing core is determined.

[0022] In some embodiments, the performance data of each computing core includes memory access time and corresponding memory access data amount of each computing core, the hardware parameters include: memory bandwidth of each computing core, memory access delay of each computing core, and memory bandwidth utilization rate of each computing core, and the model running configuration parameters further include: data block size in each computing core and numerical precision of the data block; the performance modeling of the in-core analysis task of each computing core according to the hardware parameters and the model running configuration parameters obtains the performance data of each computing core, including:

[0023] determine a memory access time of each computing core and a corresponding memory access data volume according to the data block size, the numerical precision, the memory bandwidth, the memory access delay, the memory bandwidth utilization, and a memory access modeling model.

[0024] In some embodiments, the performance data of each computing core includes a communication time of each computing core, the hardware parameters include a communication bandwidth of each computing core, a communication delay of each computing core, and a communication bandwidth utilization of each computing core, and the model running configuration parameters further include a data block size in each computing core and a numerical precision of the data block; and the performance modeling of the in-core analysis task of each computing core according to the hardware parameters and the model running configuration parameters to obtain the performance data of each computing core includes:

[0025] determine a communication time of each computing core according to the data block size, the numerical precision, the communication bandwidth, the communication delay, the communication bandwidth utilization, and a communication modeling model.

[0026] In some embodiments, the performance data of each computing core includes a total execution time of each computing core, an activation data volume, a weight data volume, and a key-value pair cache data volume; and the performance modeling of the in-core analysis task of each computing core according to the hardware parameters to obtain the performance data of each computing core includes:

[0027] determine a total execution time of each computing core, an activation data volume, a weight data volume, and a key-value pair cache data volume according to the vector computation execution time, the matrix computation execution time, the memory access time, the memory access data volume, the communication time, a preset overlap strategy parameter, and an overlap modeling model.

[0028] In some embodiments, the parallel strategy includes a card-level parallel strategy and a core-level parallel strategy, and the integration of the performance data of each computing core according to the parallel strategy to obtain the performance data of the large language model includes:

[0029] integrate the total execution time of each computing core, the activation data volume, the weight data volume, the key-value pair cache data volume, the vector computation execution time, the matrix computation execution time, the memory access time, the memory access data volume, and the communication time according to the core-level parallel strategy to obtain the performance data of each acceleration card;

[0030] integrate the performance data of each acceleration card according to the card-level parallel strategy to obtain the performance data of the large language model.

[0031] In some embodiments, the method further comprises: standardizing the performance data of the large language model to obtain performance analysis results of the large language model after standardization.

[0032] In some embodiments, the standardizing the performance data of the large language model to obtain performance analysis results of the large language model after standardization comprises:

[0033] The performance data of the large language model is standardized according to the hierarchical structure of the large language model to obtain performance data corresponding to each level of the large language model after standardization.

[0034] In some embodiments, the standardizing the performance data of the large language model to obtain performance analysis results of the large language model after standardization comprises:

[0035] The performance data of the large language model is standardized according to the task stage of the large language model to obtain performance data corresponding to each task stage of the large language model after standardization; wherein the task stage comprises: a forward propagation stage of a training task, a backward propagation stage of the training task, a pre-padding stage of an inference task, and a generation stage of the inference task.

[0036] In some embodiments, the standardizing the performance data of the large language model to obtain performance analysis results of the large language model after standardization comprises:

[0037] The performance data of the large language model is standardized according to the hardware index of the large language model to obtain hardware indexes corresponding to each task stage of the large language model after standardization; wherein the hardware index comprises: network-on-chip utilization, memory bandwidth utilization, vector computation unit utilization, matrix computation unit utilization, and storage space utilization.

[0038] In some embodiments, the standardizing the performance data of the large language model to obtain performance analysis results of the large language model after standardization comprises:

[0039] The performance data of the large language model is standardized according to the performance index of the large language model to obtain performance indexes corresponding to each task stage of the large language model after standardization; wherein the performance index comprises: first Token generation delay, Token-to-Token generation delay, total generation delay, average Token generation time, and Token throughput.

[0040] Embodiments of the present application also provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the large language model-based hierarchical modeling method described above.

[0041] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the hierarchical modeling method based on a large language model.

[0042] The embodiment of the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the hierarchical modeling method based on a large language model.

[0043] The embodiment of the present application provides a hierarchical modeling method based on a large language model. The present application realizes full-link performance analysis of a large language model from a model structure to a hardware architecture by introducing a hierarchical performance modeling framework. The framework comprises three levels of an accelerator card level modeling, a computing core level modeling and an in-core modeling, and through gradually refining the modeling process, the model task division and the hardware resource allocation are corresponded to each other, thereby effectively reducing the complexity and coupling degree of performance modeling. In the accelerator card level modeling module, the efficiency and rationality of model task division are improved by executing a model parallel strategy and a data parallel strategy. In the computing core level modeling module, the task balance and synchronization between different computing cores are realized by establishing a cooperative mechanism of multiple computing cores. In the in-core modeling module, vector modeling, matrix modeling, memory modeling and communication modeling are respectively performed, so as to accurately depict the timing characteristics of underlying computing power, storage and communication. Through parallel integration of multi-level performance data, unified performance evaluation from a single computing core to the entire large language model is realized. The present application can realize efficient and scalable performance analysis of a large language model on a multi-architecture computing platform, and significantly improves the accuracy and universality of performance modeling. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0045] Figure 1 It is a flowchart of the hierarchical modeling method based on a large language model in an embodiment of the present application;

[0046] Figure 2 It is a flowchart of the hierarchical modeling method based on a large language model in another embodiment of the present application;

[0047] Figure 3 It is a principle diagram of the optimization strategy configuration interface in the embodiment of the present application;

[0048] Figure 4 A principle schematic diagram of a model parameter configuration interface in an embodiment of the present application;

[0049] Figure 5 A principle schematic diagram of a parallel strategy configuration interface in an embodiment of the present application;

[0050] Figure 6 A principle schematic diagram of a hardware parameter configuration interface in an embodiment of the present application;

[0051] Figure 7 A flow schematic diagram of a hierarchical modeling method based on a large language model in another embodiment of the present application;

[0052] Figure 8 A principle schematic diagram of a model parallel strategy in an embodiment of the present application;

[0053] Figure 9 A flow schematic diagram of a hierarchical modeling method based on a large language model in another embodiment of the present application;

[0054] Figure 10 A principle schematic diagram of a data parallel strategy in an embodiment of the present application;

[0055] Figure 11 A flow schematic diagram of a hierarchical modeling method based on a large language model in another embodiment of the present application;

[0056] Figure 12 A flow schematic diagram of a computing core level performance data integration in an embodiment of the present application;

[0057] Figure 13 A flow schematic diagram of an accelerator card level performance data integration in an embodiment of the present application;

[0058] Figure 14 An electronic device entity structure schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0059] To make the objects, technical solutions and advantages of embodiments of the present application clearer, further detailed description will be made to the embodiments of the present application with reference to the drawings. Herein, the illustrative embodiments of the present application and the description thereof are used to explain the present application, but not as a limitation to the present application. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other at will. The acquisition, storage, use, processing, etc. of data in the technical solutions in the present application all conform to the relevant provisions of laws and regulations. The user information in the embodiments of the present application is obtained through a legal and compliant way, and the acquisition, storage, use, processing, etc. of the user information is authorized and agreed by the client.

[0060] In order to facilitate understanding of the technical solutions provided in the present application, the related content of the technical solutions of the present application will be described first.

[0061] In order to solve the problems of lack of uniformity in performance analysis of large language models on multi-architecture computing platforms, complex modeling process and insufficient expansion and adaptation capability in the prior art, the present application provides a hierarchical modeling method based on a large language model. The method divides the entire analysis process into three levels of accelerator-level modeling, computing core-level modeling and in-core modeling by introducing a hierarchical performance modeling structure, and realizes mapping analysis from model structure to hardware execution layer by layer by combining model parameters and hardware parameters of the large language model.

[0062] In the accelerator-level modeling module, the computing tasks of the large language model are divided by parallel strategy and optimization strategy, so that the model tasks can be reasonably distributed and efficiently executed among the accelerators. In the computing core-level modeling module, the task scheduling and collaboration relationship between the computing cores are established to ensure load balancing and synchronous execution in the multi-computing core environment. In the in-core modeling module, the modeling processes such as vector modeling, matrix modeling, memory modeling and communication modeling are further refined, and an overlapping modeling mechanism of computing, storage and communication processes is adopted to generate multi-level performance data. The obtained performance data is standardized according to different dimensions such as model level, computing stage and hardware index, so as to realize visual analysis of the performance of the large language model in each computing stage. The present application can effectively improve the performance evaluation efficiency and analysis accuracy of the large language model under the heterogeneous computing architecture, and significantly enhance the modularity and scalability of the performance modeling framework.

[0063] As shown in Figure 1 and Figure 2 , the present application provides a hierarchical modeling method based on a large language model, comprising steps 101 to 104.

[0064] Step 101: According to the obtained model running configuration parameters (Model Configuration Parameters), the tasks of the large language model are divided to obtain the in-card analysis tasks of each accelerator. The model running configuration parameters include: parallel strategy of the large language model, optimization strategy of the large language model, model parameters of the large language model and input data of the large language model.

[0065] Step 102: According to the obtained hardware parameters and the in-card analysis tasks of each accelerator, the in-card tasks of each accelerator are divided to obtain the in-core analysis tasks corresponding to each computing core.

[0066] Step 103: Performance modeling of the in-core analysis task of each computing core according to the hardware parameters and model running configuration parameters, to obtain performance data of each computing core (Compute Core).

[0067] Step 104: Integrating the performance data of each computing core according to the parallel strategy to obtain the performance data of the large language model.

[0068] According to the above embodiment, the present application realizes the full-link performance analysis of the large language model from the model structure to the hardware architecture by introducing a hierarchical performance modeling framework. The framework includes three levels of accelerator card-level modeling, computing core-level modeling and in-core modeling. By gradually refining the modeling process, the model task division and hardware resource allocation are corresponded to each other, thereby effectively reducing the complexity and coupling degree of performance modeling. In the accelerator card-level modeling module, the efficiency and rationality of model task division are improved by executing model parallel strategy and data parallel strategy. In the computing core-level modeling module, the task balance and synchronization between different computing cores are realized by establishing a cooperative mechanism of multiple computing cores. In the in-core modeling module, vector modeling, matrix modeling, memory modeling and communication modeling are performed respectively, so as to accurately depict the timing characteristics of the underlying computing power, storage and communication. By parallel integration of multi-level performance data, unified performance evaluation from a single computing core to the entire large language model is realized. The present application can realize efficient and scalable performance analysis of the large language model on multi-architecture computing platforms, significantly improving the accuracy and universality of performance modeling.

[0069] In the embodiment of the present application, the parallel strategy of the large language model (LLM) includes model parallelism (Model Parallelism), data parallelism (Data Parallelism, DP), tensor parallelism (Tensor Parallelism, TP), pipeline parallelism (Pipeline Parallelism, PP) and expert parallelism (Expert Parallelism, EP) and the like.

[0070] Among them, model parallelism refers to dividing the model parameters of a large language model to multiple acceleration cards for parallel processing, and each acceleration card is only responsible for the calculation task of part of the model parameters of the large language model. Data parallelism refers to dividing the input data of a large language model to multiple acceleration cards, each acceleration card saves a copy of the model parameters, and respectively calculates the gradient of the corresponding input data subset. Tensor parallelism refers to dividing the large-scale matrix or tensor in the large language model in a certain dimension to multiple acceleration cards for parallel calculation to share the computational load of large-scale matrix operation. Pipeline parallelism refers to distributing different stages of a large language model to different acceleration cards for parallel processing of multiple data batches in a pipeline manner. Expert parallelism is suitable for a mixture of experts (MoE), which activates only part of the experts to participate in the calculation by distributing different expert networks (Experts) to different acceleration cards to improve the overall calculation efficiency.

[0071] The optimization strategies of a large language model include: quantization and per-component precision, operator fusion, recomputation and memory reuse, computation, memory and communication overlap, etc.

[0072] The model parameters of a large language model include: structural parameters, computational parameters and runtime parameters. Among them, the structural parameters are used to describe the hierarchical structure and network size of the large language model, the computational parameters are used to represent the calculation precision, calculation mode and calculation stage of the large language model, and the runtime parameters are used to define the input and output configurations and task running environment of the large language model in the execution process.

[0073] The above hardware parameters are used to describe the performance characteristics of the computing system supporting the running of a large language model. The hardware parameters include: computational parameters, on-chip network parameters, memory parameters and interconnect topology parameters, etc.

[0074] The input data of the large language model includes, but is not limited to, different types of data forms such as a text token sequence, an image patch and a speech feature sequence, and the application is not limited thereto.

[0075] Before performing task division of the large language model, the optimization strategy of the large language model, the model parameter of the large language model, the parallel strategy of the large language model and the hardware parameter of the large language model are uniformly set as an input interface to ensure that the formats of various input information in the model modeling process are consistent and the interaction is standardized. Through the unified input interface, the standardization management of the optimization strategy, the model parameter, the parallel strategy and the hardware parameter can be realized in the performance modeling stage.

[0076] As shown in Figure 3 , the application structures and abstracts the optimization strategy of the large language model according to the model component dimension and the calculation stage dimension to form a standardized optimization strategy configuration interface. The model component dimension includes an Attention component, an FFN component and an MoE component, and the calculation stage dimension includes a training stage, an inference stage, a forward stage, a reverse stage, a pre-padding stage and a decoding stage.

[0077] Figure 3 The left side shows three types of large language models, namely a Transformer model, an MOE model and other models. The above models each have an independent optimization strategy. By structuring and abstracting the optimization strategies of different models according to the model component dimension and the calculation stage dimension, the optimization strategies can be uniformly mapped to the standardized optimization strategy configuration interface shown on the right side. Figure 3

[0078] As shown in Figure 3 , in the internal part of the optimization strategy configuration interface, corresponding optimization strategy configuration blocks are set for different model component dimensions (including an Attention component, an FFN component and an MoE component). Each optimization strategy configuration block includes the following four main optimization strategies:

[0079] Quantization and mixed precision are used to control the numerical precision configuration of different model components in the large language model. For example, in the Attention component, low-precision quantization may cause information loss due to global interaction, so FP16 precision is retained for calculation. In the FFN (MLP) component, the calculation is mainly local matrix operation, and FP8 quantization can be used, which not only ensures the model precision, but also improves the calculation performance and the utilization rate of computing power.

[0080] ​Operator fusion is used to fuse multiple computing operators into a single operator execution at the level of the computation graph to reduce the storage and data transfer of intermediate results. For example, combining the Query Matrix, Key Matrix and Value Matrix projection operations in the Attention component with the Softmax operation can effectively reduce memory access and improve overall computing throughput.

[0081] Recalculation and memory reuse are used to reduce GPU memory usage by recalculating part of the intermediate activation values during the training phase. During the inference phase, the memory bandwidth utilization and cache hit rate are improved through caching sharing and memory block reuse (such as Key-Value Cache) and weight reuse mechanisms.

[0082] Computing, memory access and communication masking are used to overlap the execution of computing, data access and communication operations in parallel through micro-batch scheduling or pipelining, reducing the waiting time between the computing process, data access process and communication operation process, and improving the overall computing power utilization and parallel execution efficiency of the accelerator card.

[0083] Through the above structured abstraction method, a unified optimization strategy description and parameterized configuration can be implemented for different types of large language models, thereby improving the flexibility and scalability of performance modeling.

[0084] During the performance analysis of large language models, different types of large language models (such as Transformer models, MoE models and other types of models) have differences in model structure, computing method and running characteristics. In order to achieve unified performance modeling of different models on multiple hardware platforms and multiple optimization strategies, as shown in Figure 4 The present application structures the model parameters of the large language model to form a standardized model parameter configuration interface.

[0085] As shown in Figure 4 The present application integrates and standardizes the model parameters of Transformer models, MoE models and other types of models to construct a unified model parameter configuration interface. The model parameter configuration interface includes model structure parameters, computing parameters and model running parameters.

[0086] The model structure parameters include a number of layers, a hidden dimension, a number of attention heads, a vocabulary size, an intermediate dimension, and MoE expert related parameters.

[0087] The computing parameters include precision, mode, and stage. The precision includes INT8, FP8, FP16, and FP32. The mode includes training or inference. The stage includes a forward propagation stage, a backward propagation stage, a prefill stage, or a decode stage.

[0088] The model running parameters include a batch size, a micro-batch size, an input sequence length, and an output sequence length.

[0089] By establishing a unified model parameter configuration interface, the parameter standardization input of different types of large language models on multiple hardware platforms and multiple optimization strategies is realized, and the generality and scalability of performance analysis are significantly improved.

[0090] As shown in Figure 5 The application forms a standardized parallel strategy configuration interface according to the structural characteristics and parallel characteristics of different computing components in the large language model.

[0091] Figure 5 The left side of the middle is a typical transformer structure, which is composed of multiple root mean square layer normalization (RMSNorm), attention mechanisms, and feedforward networks (FFN). Different computing components have significant differences in computing characteristics and communication modes.

[0092] Among them, the Attention component relies on global sequence interaction and belongs to communication-intensive computing. The FFN component is mainly composed of large-scale matrix multiplication and belongs to computing-intensive operation. In the Mixture of Experts (MoE) model, a sparse activation mechanism is adopted, only part of the expert network participates in the calculation, and an Expert Parallel strategy is adopted.

[0093] As shown in Figure 5 As shown on the right, the application structures and abstracts the parallel strategies of different computing components to form a unified parallel strategy configuration interface, so that the optimal parallel strategy can be independently configured for the Attention component, the FFN component and other components in different computing stages such as the training stage, the prefill stage and the decoding stage.

[0094] Through the above structured parallel strategy abstraction mechanism, the component-level parallel scheme can be flexibly combined under different model architectures and various hardware platforms, achieving higher computing power utilization and performance modeling accuracy.

[0095] As shown in Figure 6 As shown in

[0096] Figure 6 As shown on the left, three typical hardware architectures are shown, namely Graphics Processing Unit (GPU), Neural Processing Unit (NPU) and other special computing architectures (such as Tensor Processing Unit (TPU) or Application-Specific Integrated Circuit (ASIC)). Different hardware architectures differ in internal structure, computing unit type, storage method and interconnection method. Among them, the graphics processor architecture takes Streaming Multiprocessor (SM) as the computing core, the neural network processor architecture takes AI Core as the computing core, and the TPU or ASIC architecture realizes computing based on customized instruction set and topology.

[0097] As shown in Figure 6 As shown on the right, the hardware parameter configuration interface includes computing power parameters, on-chip network parameters, storage parameters and interconnection topology parameters.

[0098] Among them, the computing power parameter is used to describe the basic computing power characteristics of the computing hardware, including: the number of computing cores, vector computing throughput, and tensor computing throughput.

[0099] The number of computing cores represents the total number of computing cores integrated on a single chip, for example: NVIDIA H100 contains 144 SM units, and Ascend 910 contains 32 AI cores. The vector computing power parameter is used to represent the vector operation parallelism, vector operation throughput, and vector operation latency under different numerical precisions. The tensor computing power parameter is used to describe the matrix operation parallelism, matrix operation throughput, and matrix operation latency under different numerical precisions.

[0100] The on-chip network parameter is used to represent the communication characteristics between computing cores within the chip, including on-chip bandwidth, communication latency, and average routing hop count.

[0101] The storage parameter is used to represent the hierarchical storage structure from the chip internal cache to the chip external memory, which can be numbered as L1, L2, L3, …, L n , etc. according to the memory access distance. Each storage level includes parameters such as storage capacity, memory bandwidth, and memory access latency.

[0102] The interconnection topology parameter is used to represent the interconnection structure characteristics between multiple chips or multiple computing nodes, including topology type, node link bandwidth, node link latency, and network congestion degree.

[0103] Through the common extraction, difference screening, and hierarchical organization of the above hardware parameters, the decoupling of large language model performance modeling and hardware architecture is realized.

[0104] In some embodiments, the parallel strategy includes a card-level parallel strategy (Card-level ParallelStrategy) and a core-level parallel strategy (Core-level Parallel Strategy), and the card-level parallel strategy includes a model parallel strategy. As shown in Figure 7 Step 101 includes steps 701-702.

[0105] Step 701: According to the model parallel strategy, the model parameters are divided to obtain a plurality of subsets of model parameters.

[0106] Step 702: The subsets of model parameters are respectively mapped to the corresponding accelerator cards to obtain intra-card analysis tasks of each accelerator card.

[0107] In the embodiments of the present application, as shown in Figure 8 The hardware device can include a plurality of accelerator cards (Accelerator Card), such as GPU cards or NPU cards, for parallel execution of inference tasks or training tasks of large language models.

[0108] In the model parallel scenario, the entire model parameters of the large language model are divided according to a preset division number (for example, 16) to obtain a plurality of subsets of model parameters. At the same time, the computing cluster is divided into a corresponding number of accelerator cards (for example, 16 accelerator cards). Each accelerator card stores the same input data (Input Data) of the large language model, which can include but is not limited to text token sequences, image blocks, and speech feature sequences.

[0109] The subsets of model parameters obtained by division are respectively mapped to the corresponding accelerator cards, so that each accelerator card performs calculation on the corresponding subset of model parameters, thereby forming a parallel execution structure at the card level. When each accelerator card performs independent calculation tasks, it realizes necessary data interaction through on-chip interconnection (On-Chip Interconnect) and high-speed communication link (High-speed Communication Link) to ensure the correctness and synchronization of the calculation process of the large language model in the forward propagation (Forward Propagation) stage and the backward propagation (Backward Propagation) stage.

[0110] According to the above-mentioned embodiments, the application realizes efficient parallel computing of large language models in a computing cluster environment. By dividing and mapping model parameters to multiple acceleration cards, the model computing load is evenly distributed among hardware resources, effectively improving the computing power utilization rate and computing throughput. Each acceleration card independently performs in-card analysis tasks of the large language model while maintaining consistent input data, and efficiently synchronizes intermediate results through on-chip interconnection and high-speed communication links, ensuring the correctness and gradient consistency of the large language model in the forward propagation and backward propagation stages. The above method significantly reduces the memory occupation and computing bottleneck of a single acceleration card, and improves the parallel efficiency and system expansion capability of large-scale models in the training and inference stages.

[0111] In some embodiments, the parallel strategy includes a card-level parallel strategy and a core-level parallel strategy, and the card-level parallel strategy further includes a data parallel strategy. As shown in Figure 9 Step 101 includes steps 901-902.

[0112] Step 901: According to the batch data amount of the input data and the data parallel strategy, the input data is divided to obtain a plurality of data sub-samples.

[0113] Step 902: Map the data sub-samples to the corresponding acceleration cards respectively to obtain the in-card analysis tasks corresponding to each acceleration card.

[0114] In the embodiments of the application, as shown in Figure 10 The computing cluster is taken as a hardware device, and a card-level parallel strategy is taken as a data parallel strategy. The hardware device can include multiple acceleration cards, such as GPU cards or NPU cards, for parallel execution of inference tasks or training tasks of large language models.

[0115] In the data parallel scenario, the entire input data of the large language model is divided according to a preset division number (for example, 16) to obtain a plurality of data sub-samples (Sub-samples of Input Data). At the same time, the computing cluster is divided into a corresponding number of acceleration cards (for example, 16 acceleration cards). Each acceleration card stores the same model parameters of the large language model, which can include model structure parameters, calculation parameters, and model running parameters.

[0116] The divided data sub-samples are respectively mapped to the corresponding acceleration cards, so that each acceleration card independently performs the same computing task on the corresponding data sub-sample, thereby forming a parallel execution structure at the card level. After completing independent calculation, each acceleration card synchronizes gradient information and updates parameters through on-chip interconnection and high-speed communication link, to ensure the consistency and convergence accuracy of forward propagation and back propagation in the model training process.

[0117] According to the above embodiment, the present application realizes efficient distributed computing of large language models among multiple acceleration cards by introducing a data parallel strategy in the computing cluster. By dividing the input data into multiple data sub-samples and distributing them to different acceleration cards, each acceleration card independently performs the computing task while keeping the model parameters consistent, thereby improving the parallelism and hardware resource utilization in the model training and inference process. Gradient synchronization and parameter updating are achieved between each acceleration card through on-chip interconnection and high-speed communication link, ensuring the consistency and convergence stability of large language models in the forward propagation and back propagation process. The above method can effectively reduce the computing load and storage pressure of a single acceleration card, significantly improving the computing throughput and expansion performance of the overall system.

[0118] In some embodiments, after the above modeling is completed by the acceleration card-level modeling module, the intra-card analysis tasks of each acceleration card can be delivered to the compute core-level modeling module through a unified parallel strategy configuration interface, an optimization strategy configuration interface, a model parameter configuration interface, and a hardware parameter configuration interface.

[0119] In some embodiments, in the compute core-level modeling module, step 102 specifically includes: dividing the intra-card analysis tasks according to the hardware parameters of each acceleration card (including the number of computing cores, vector computing power parameters, tensor computing power parameters, on-chip network parameters, storage parameters, and interconnection topology parameters, etc.), and distributing them to each computing core (Compute Core), forming multiple intra-core analysis tasks (Intra-core Analytical Task). Each computing core independently executes its corresponding intra-core analysis task, and its computing process includes vector computation, matrix computation, memory operation, and communication analysis, etc. After the modeling is completed by the compute core-level modeling module, the intra-core analysis tasks of each computing core are delivered to the compute core-level modeling module through a unified parallel strategy configuration interface, an optimization strategy configuration interface, a model parameter configuration interface, and a hardware parameter configuration interface.

[0120] In some embodiments, the performance data of each computing core comprises a vector computation execution time of the vector compute unit. The hardware parameters comprise a vector compute throughput of the vector core in each computing core, a first launch latency of the vector compute unit in each computing core, and a first hardware utilization efficiency of the vector compute unit in each computing core. The model running configuration parameter further comprises a data block size in each computing core. Step 103 specifically comprises:

[0121] According to the data block size, the vector compute throughput, the launch latency, the hardware utilization efficiency, and the vector modeling model, the vector computation execution time of the vector compute unit in each computing core is determined. The vector modeling model is a vector modeling function.

[0122] In the embodiments of the present application, the data block size , the vector compute throughput of the vector compute unit , the launch latency of the vector compute unit , and the hardware utilization efficiency of the vector compute unit are input into the vector modeling function , and the vector computation execution time required by the vector compute unit when processing the input data block is calculated by the vector modeling function .

[0123] Exemplarily, the mathematical expression of the vector modeling function can be represented as:

[0124]

[0125] wherein the data block size is used to represent the scale of the input data, is the vector compute throughput of the vector compute unit, used to represent the parallel computing capability of the vector compute unit at a specific numerical precision; is the hardware utilization efficiency of the vector compute unit, used to represent the actual use efficiency of the computing resource; is the launch latency of the vector compute unit, used to represent the time delay between receiving the in-core analysis task and starting execution of the vector compute unit.

[0126] According to the above embodiment, the application quantitatively models the data block size, vector computing power parameter, start delay and hardware utilization rate and other hardware parameters by constructing a vector modeling function, and forms a unified mathematical description model, thereby realizing quantitative characterization of the performance of the vector computing unit. The above method can accurately evaluate the computing delay and computing power utilization under different hardware configurations and numerical precision conditions in the modeling stage. By introducing the parameterized modeling mechanism, the application improves the accuracy and universality of the performance analysis program, and realizes unified modeling and analysis of the performance of the vector computing unit under various hardware architectures.

[0127] In some embodiments, the performance data of each computing core includes: a matrix computation execution time of a matrix computation unit in each computing core. The hardware parameters include: a tensor computing throughput of the matrix computation unit in each computing core, a second start delay of the matrix computation unit in each computing core, and a second hardware utilization efficiency of the matrix computation unit in each computing core. The model running configuration parameters further include: a right matrix block of the matrix computation unit (Matrix Core) in each computing core and a left matrix block of the matrix computation unit in each computing core. Step 103 specifically includes:

[0128] According to the right matrix block, the left matrix block, the tensor computing throughput, the second start delay, the second hardware utilization efficiency and the matrix modeling model, the matrix computation execution time of the matrix computation unit in each computing core is determined. The matrix modeling model is a matrix modeling function.

[0129] In the embodiment of the application, the left matrix block split into the matrix computation unit in each computing core and the right matrix block split into the matrix computation unit in each computing core , the tensor computing throughput of the matrix computation unit , the start delay of the matrix computation unit and the hardware utilization rate of the matrix computation unit are input into the matrix modeling function , and the matrix computation execution time required by the matrix computation unit when processing the input matrix block is calculated by the matrix modeling function, i.e. The matrix modeling function is used to establish a quantitative mapping relationship between the matrix computation performance and the hardware parameters, so as to realize accurate modeling and analysis of the matrix computation performance under different matrix sizes, hardware parameters and parallel strategies.

[0130] Exemplarily, the mathematical expression of the matrix modeling function can be represented as: Exemplarily, the mathematical expression of the matrix modeling function can be represented as:

[0131]

[0132] wherein, is a left matrix block of the matrix calculation unit, is a right matrix block of the matrix calculation unit, is a tensor computing power parameter of the matrix calculation unit, used to represent the matrix operation capability of the matrix calculation unit under a specific numerical precision; is a hardware utilization rate of the matrix calculation unit, used to represent the actual use efficiency of the computing resource; is a start-up delay of the matrix calculation unit, used to represent the time delay between the matrix calculation unit receiving an in-core analysis task and starting execution.

[0133] In some embodiments, the performance data of each computing core includes: a memory access time of each computing core and a corresponding memory access data volume of each computing core. The hardware parameters include: a memory bandwidth of each computing core, a memory access latency of each computing core, and a memory bandwidth utilization rate of each computing core. The model running configuration parameters further include: a data block size of each computing core, and a numerical precision of the data block. The step 103 specifically includes:

[0134] According to the data block size, the numerical precision, the memory bandwidth, the memory access latency, the memory bandwidth utilization rate, and the memory access modeling model, the memory access time of each computing core and the corresponding memory access data volume of each computing core are determined.

[0135] In the embodiments of the present application, the data block size , the numerical precision of the data block , the memory bandwidth , the memory access latency , and the memory bandwidth utilization rate of each computing core are input into a memory access modeling function . The memory access modeling function is used to calculate the time required to move the data block from the storage unit to the computing core, i.e., the memory access time , and the corresponding data transfer volume .

[0136] Exemplarily, the mathematical expression of the memory access modeling function can be represented as:

[0137]

[0138]

[0139] wherein the data block size is used to represent the scale of input data, is used to represent the bit width of each data unit in a single memory access operation; is used to represent the data transfer capability of the memory system, is used to represent the actual use efficiency of the memory bandwidth; is used to represent the time delay between the initiation of a memory request and the availability of data.

[0140] According to the above embodiment, the application quantifies the key hardware parameters such as data block size, numerical precision, memory bandwidth, memory access delay and bandwidth utilization into mathematical models by constructing a memory modeling function, and realizes the computable description of the performance of the memory system. The above memory modeling method can accurately evaluate the data transfer time and bandwidth occupation under different hardware architectures without relying on actual tests, thereby effectively depicting the data transfer characteristics between the computing core and the memory unit. Through the above method, potential memory bottlenecks can be quickly identified in the model design stage.

[0141] In some embodiments, the performance data of each computing core includes the communication time of each computing core. The hardware parameters include the data block size of each computing core, the numerical precision of each computing core, the communication bandwidth of each computing core, the communication delay of each computing core and the communication bandwidth utilization of each computing core. Step 103 specifically includes:

[0142] According to the data block size, the numerical precision, the communication bandwidth, the communication delay, the communication bandwidth utilization and the communication modeling model, the communication time of each computing core is determined.

[0143] In the embodiment of the application, the data block size to be processed, the numerical precision, the communication bandwidth, the communication delay, and the communication bandwidth utilization are input into a communication modeling function , and the time consumed by the data block for communication operation, i.e. the communication time and the corresponding data transfer amount are calculated by the communication modeling function.

[0144] For example, the mathematical expression of the communication modeling function can be represented as:

[0145]

[0146]

[0147] Among them, data block size Used to characterize the data size during communication operations. For numerical precision, it is used to characterize the bit width of each data unit in a communication operation; Communication bandwidth is used to characterize the data transmission capacity of a communication link. Communication bandwidth utilization rate is used to characterize the actual efficiency of communication link bandwidth usage. Communication access delay is used to characterize the time delay between the initiation of a communication request and the start of data transmission.

[0148] According to the above embodiments, this invention constructs a communication modeling function to quantitatively describe key communication-related hardware parameters in the form of a mathematical model, achieving accurate calculation and predictability analysis of communication performance. This method can quantitatively evaluate data transmission time under different communication topologies and link bandwidth configurations without actual testing, thereby effectively identifying communication bottlenecks and guiding the optimization design of parallel strategies. Through this communication modeling mechanism, the communication efficiency and synchronization accuracy of large language models in multi-accelerator card parallel computing scenarios can be improved.

[0149] In some embodiments, the performance data for each computing core includes: the total execution time, activation data volume, weight data volume, and key-value cache volume for each computing core. Step 103 specifically includes:

[0150] Based on the vector computation execution time, matrix computation execution time, memory access time, memory access data volume, communication time, preset overlap policy parameters, and overlap modeling model, determine the total execution time, activated data volume, weighted data volume, and key-value pair cached data volume for each computation core.

[0151] In this embodiment of the invention, the vector computation execution time is... Matrix calculation execution time Memory access time Data transmission volume Communication time and overlap strategy parameters Input to overlap modeling function By using an overlapping modeling function, the execution times of different types of operations, such as computation, memory access, and communication, are fused and overlaid for analysis, and the total execution time is calculated separately. Activation data volume Weighted data volume and the amount of key-value pair cached data .

[0152] For example, the mathematical expression for the overlap modeling function can be represented as:

[0153]

[0154] in, Calculate the execution time for the vector. Calculate the execution time for the matrix. For memory access time, This refers to the communication time.

[0155] Alternatively, the mathematical expression for the overlap modeling function can also be expressed as:

[0156]

[0157] in, Calculate the execution time for the vector. Calculate the execution time for the matrix. For memory access time, For communication time, The overlap weight coefficient is determined by the overlap strategy parameters. control.

[0158] According to the above embodiments, this invention constructs an overlap modeling function to integrate and analyze the execution times of different types of operations, such as computation, memory access, and communication, thereby achieving unified timing modeling of the execution process of multiple parallel tasks. By introducing overlap strategy parameters, the parallel overlap relationship between different operations is quantitatively described, enabling accurate characterization of the overall execution characteristics of the system. The above method significantly improves the accuracy and scalability of performance analysis.

[0159] In some embodiments, the parallel strategies include: card-level parallel strategy and core-level parallel strategy. For example... Figure 11 As shown, step 104 includes steps 1101 to 1102.

[0160] Step 1101: Integrate the total execution time, active data volume, weight data volume, key-value pair cache data volume, vector calculation execution time, matrix calculation execution time, memory access time, memory access data volume, and communication time of each computing core according to the core-level parallel strategy to obtain the performance data of each accelerator card.

[0161] Step 1102: Integrate the performance data of each accelerator card according to the card-level parallelism strategy to obtain the performance data of the large language model. The card-level parallelism strategy includes: DataParallel (DP), Tensor Parallel (TP), Pipeline Parallel (PP), and Expert Parallel (EP). The core-level parallelism strategy is used to determine the task allocation and collaborative execution method among multiple computing cores within a single accelerator card.

[0162] In this embodiment of the invention, the performance data of each computing core is transmitted to the computing core-level modeling module through a unified parallel strategy configuration interface, optimization strategy configuration interface, model parameter configuration interface, and hardware parameter configuration interface.

[0163] like Figure 12 As shown, the performance data output by computing core 0, computing core 1 to computing core x, and the core-level parallelism strategy are input into the core-level rule function. Aggregation is performed to calculate the performance data of a single accelerator card. This includes core-level rule functions. Based on a core-level parallelism strategy, the output results of different computing cores are weighted, averaged, or accumulated to calculate the performance data of a single accelerator card. Computing core 0 and computing cores 1 through x output the following: total execution time. Activation data volume Weighted data volume Key-value pair cache data volume Vector computation execution time Matrix calculation execution time Memory access time Memory access data volume and communication time Parameters such as these.

[0164] Performance data from each accelerator card is transmitted to the accelerator card-level modeling module through a unified parallel strategy configuration interface, optimization strategy configuration interface, model parameter configuration interface, and hardware parameter configuration interface.

[0165] likeFigure 13 As shown, the performance data output by accelerator cards 0, 1 to x, and the card-level parallel strategy are input into the card-level rule function. Aggregation is performed to calculate the performance data of the large language model. Among these, the card-level rule function... Based on a card-level parallelism strategy, the outputs of different accelerator cards are weighted, averaged, or accumulated to calculate the performance data of the large language model. Accelerator card 0, accelerator card 1 through accelerator card x outputs including: total execution time. Activation data volume Weighted data volume Key-value pair cache data volume Vector computation execution time Matrix calculation execution time Memory access time Memory access data volume and communication time Parameters such as these.

[0166] According to the above embodiments, this invention achieves hierarchical aggregation and unified computation of performance data by introducing rule functions into both the core-level modeling module and the accelerator card-level modeling module. In the core-level modeling module, the performance data output by each computing core is weighted, averaged, or accumulated using core-level rule functions, comprehensively reflecting the overall computing power and load distribution of a single accelerator card under multi-core collaborative execution. In the accelerator card-level modeling module, the performance data of multiple accelerator cards is further aggregated using card-level rule functions, enabling a unified evaluation of the overall large language model execution performance of the computing cluster. This hierarchical aggregation modeling approach effectively improves the accuracy and scalability of performance analysis results.

[0167] In some embodiments, the hierarchical modeling method for large language models further includes: standardizing the performance data of the large language model to obtain the performance analysis results of the standardized large language model.

[0168] In this embodiment of the invention, the performance data of the large language model is output in a formatted manner. Based on the input output control information, the performance data obtained from modeling at each level is structured and standardized to generate performance analysis results of the large language model in a unified format.

[0169] The standardization process includes: hierarchical structure standardization, task stage standardization, hardware metric standardization, and performance metric standardization. Those skilled in the art can choose to combine one or more of these standardization methods to standardize the performance data of large language models based on specific performance analysis needs, achieving unified performance representation and comparability analysis across different dimensions. Output control information includes: output format parameters, output dimension parameters, output content parameters, and output granularity parameters. Output format parameters define the output format of the performance analysis results, including document format (Word), table format (Excel), graphical format (such as bar charts or pie charts), and data format (such as JSON files). Output dimension parameters determine the organization of performance data, which can be output according to model layer level, task stage, or hardware metric. Output content parameters limit the output range of performance data, allowing users to select the desired metric categories to display. The output granularity parameter is used to control the level of detail in the performance analysis results display. It can output single-level indicators, single accelerator card indicators, full model indicators, or cluster-level summary results as needed.

[0170] In some embodiments, the performance data of the large language model is standardized to obtain the performance analysis results of the standardized large language model, specifically including:

[0171] The performance data of the large language model is standardized according to the hierarchical structure of the large language model to obtain the performance data corresponding to each level of the standardized large language model.

[0172] In this embodiment of the invention, the LLaMA large language model is used as an example for illustration.

[0173] When the output dimension parameter is set to model level, the performance data can be standardized and output according to the hierarchical structure of the large language model. The hierarchical structure of the LLaMA large language model includes: RMSNorm layer, Q_proj layer, RoPE(Q) layer, K_proj layer, RoPE(K) layer, V_proj layer, QK^T layer, SoftMax layer, ScoreV layer, Linear layer, Res ADD layer, RMSNorm layer, FFN1 layer, Gate_FFN layer, SwiGLU layer, FFN2 layer, and Res ADD layer.

[0174] During performance analysis, performance data can be output for each of the above layers. This performance data includes: total execution time. Activation data volume Weighted data volume Key-value pair cache data volume Vector computation execution time Matrix calculation execution time Memory access time Memory access data volume and communication time wait.

[0175] According to the above embodiments, by using performance index output based on the model hierarchy, a refined analysis of the computation, memory access, and communication performance of each level of the large language model can be achieved, thereby providing a quantitative basis for the optimization of the large language model and the scheduling of hardware resources.

[0176] In some embodiments, the performance data of the large language model is standardized to obtain the performance analysis results of the standardized large language model, specifically including:

[0177] The performance data of the large language model is standardized according to the task stages of the large language model to obtain the standardized performance data corresponding to each task stage. The task stages include: the forward propagation stage of the training task, the back propagation stage of the training task, the pre-filling stage of the inference task, and the generation stage of the inference task.

[0178] In this embodiment of the invention, the overall performance data of the large language model at different task stages can be standardized and output. These task stages include: the forward propagation stage of the training task, the backward propagation stage of the training task, the prefill phase of the inference task, and the decoding phase of the inference task. Corresponding overall performance data can be output for each task stage to quantify the operational efficiency and computational load of the large language model at each stage.

[0179] Overall performance data includes, but is not limited to, the following metrics: total execution time of the attention mechanism, total execution time of the feedforward network (MLP), total model time, time to first token (TTFT), and average tokens per second (TPS). Additionally, other relevant metrics used to characterize the computational and communication performance of the large language model may also be included.

[0180] According to the above embodiments, by using a performance data output method based on task stages, it is possible to quantitatively analyze the performance data of a large language model throughout the entire training and inference process.

[0181] In some embodiments, the performance data of the large language model is standardized to obtain the performance analysis results of the standardized large language model, including:

[0182] The performance data of the large language model is standardized according to the hardware metrics of the large language model to obtain the standardized hardware metrics of the large language model at each task stage. The hardware metrics include: on-chip network utilization, memory bandwidth utilization, vector computation unit utilization, matrix computation unit utilization, and storage space utilization.

[0183] In this embodiment of the invention, the output can be standardized according to the hardware utilization of the large language model under different task stages. The task stages include: the forward propagation stage of the training task, the backward propagation stage of the training task, the prefill phase of the inference task, and the decode phase of the inference task.

[0184] In different task phases, the utilization of various hardware resources during the operation of the large language model is quantitatively evaluated to comprehensively reflect the performance characteristics of the large language model in terms of computation, storage, and communication. Hardware utilization metrics include, but are not limited to: on-chip network utilization, memory bandwidth utilization, vector core utilization, tensor core utilization, overall computational throughput utilization, and memory space utilization.

[0185] According to the above embodiments, by analyzing hardware utilization based on task phases, it is possible to accurately characterize the resource scheduling efficiency and computing power usage of large language models at different operating phases, thereby providing a quantitative basis for large language model optimization and hardware resource scheduling.

[0186] In some embodiments, the standardization process of the performance data of the large language model to obtain the performance analysis results of the standardized large language model includes:

[0187] The performance data of the large language model is standardized according to the performance indicators of the large language model to obtain the standardized performance indicators of the large language model at each task stage. The performance indicators include: first token generation latency (Time To First Token), inter-token generation latency (Inter-Token Latency), total generation latency (Total Latency), average token generation time (Average Token Time), and token throughput (Tokens Per Second).

[0188] In this embodiment of the invention, the output can be standardized according to the performance metrics of the large language model under different task stages. The task stages include: the forward propagation stage of the training task, the backward propagation stage of the training task, the prefill phase of the inference task, and the decode phase of the inference task.

[0189] In different task phases, various performance metrics are quantitatively evaluated during the operation of the large language model to comprehensively reflect its performance characteristics. These performance metrics include, but are not limited to: first token generation latency (TimeToFirstToken), inter-token generation latency (Inter-Token Latency), total generation latency (TotalLatency), average token generation time (AverageTokenTime), and token throughput (TokensPerSecond).

[0190] According to the above embodiments, performance analysis based on task stages can accurately characterize the resource scheduling efficiency and computing power utilization of large language models at different running stages, thereby providing a quantitative basis for performance optimization of large language models.

[0191] Figure 14 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 14 As shown, the electronic device includes: a processor 1401, a memory 1402, and a bus 1403.

[0192] The processor 1401 and the memory 1402 communicate with each other via the bus 1403.

[0193] The processor 1401 is used to call program instructions in the memory 1402 to execute the methods provided in the above-described method embodiments.

[0194] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described hierarchical modeling method based on a large language model.

[0195] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described hierarchical modeling method based on a large language model.

[0196] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0198] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0200] In the description of this specification, the references to terms such as "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0201] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hierarchical modeling method based on a large language model, characterized in that, include: The large language model is divided into tasks based on the obtained model running configuration parameters to obtain the in-card analysis tasks of each accelerator card; among which, the model running configuration parameters include: the parallel strategy, optimization strategy, model parameters and input data of the large language model; Based on the obtained hardware parameters and the in-card analysis tasks of each accelerator card, the in-card tasks of each accelerator card are divided to obtain the in-core analysis tasks corresponding to each computing core. Based on the hardware parameters and the model running configuration parameters, the performance of the in-core analysis task of each computing core is modeled to obtain the performance data of each computing core; The performance data of each computing core are integrated according to the parallel strategy to obtain the performance data of the large language model; The performance data of each computing core includes the vector computation execution time of the vector computation unit. The hardware parameters include: the vector computing power parameters of the vector computation unit in each computing core, the first startup latency of the vector computation unit in each computing core, and the first hardware utilization rate of the vector computation unit in each computing core. The model running configuration parameters also include: the data block size in each computing core. The performance modeling of the in-core analysis task of each computing core based on the hardware parameters and the model running configuration parameters to obtain the performance data of each computing core includes: determining the vector computation execution time of the vector computation unit in each computing core based on the data block size, the vector computing power parameters, the startup latency, the hardware utilization rate, and the vector modeling model.

2. The method according to claim 1, characterized in that, The parallel strategy includes: card-level parallel strategy and core-level parallel strategy, wherein the card-level parallel strategy includes a model parallel strategy; the step of dividing the large language model into tasks based on the obtained model running configuration parameters to obtain the in-card analysis tasks of each accelerator card includes: The model parameters are divided according to the model parallelization strategy to obtain multiple subsets of model parameters; The model parameter subsets are mapped to the corresponding accelerator cards to obtain the in-card analysis tasks for each accelerator card.

3. The method according to claim 1 or 2, characterized in that, The parallel strategy includes: card-level parallel strategy and core-level parallel strategy, wherein the card-level parallel strategy further includes a data parallel strategy; the step of dividing the large language model into tasks based on the obtained model running configuration parameters to obtain the in-card analysis tasks of each accelerator card includes: Based on the batch processing data volume of the input data and the data parallel strategy, the input data is divided into multiple data sub-samples; The data subsamples are mapped to the corresponding accelerator cards to obtain the in-card analysis tasks for each accelerator card.

4. The method according to claim 1, characterized in that, The performance data of each computing core includes the matrix computation execution time of the matrix computation unit in each computing core. The hardware parameters include: tensor computing power parameters of the matrix computation unit in each computing core, the second startup latency of the matrix computation unit in each computing core, and the second hardware utilization rate of the matrix computation unit in each computing core. The model running configuration parameters also include: the left matrix block and the right matrix block of the matrix computation unit in each computing core. The performance modeling of the in-core analysis task of each computing core based on the hardware parameters and the model running configuration parameters yields the performance data of each computing core, including: Based on the right matrix block, the left matrix block, the tensor computing power parameters, the second startup delay, the second hardware utilization rate, and the matrix modeling model, the matrix computation execution time of the matrix computation unit in each computing core is determined.

5. The method according to claim 4, characterized in that, The performance data of each computing core includes the memory access time of each computing core and the corresponding amount of memory accessed data. The hardware parameters include: the memory bandwidth of each computing core, the memory access latency of each computing core, and the memory bandwidth utilization of each computing core. The model running configuration parameters also include: the data block size and the numerical precision of the data blocks in each computing core. The performance modeling of the in-core analysis task of each computing core based on the hardware parameters and the model running configuration parameters yields the performance data of each computing core, including: Based on the data block size, numerical precision, memory bandwidth, memory access latency, memory bandwidth utilization, and memory access modeling model, the memory access time of each computing core and the corresponding amount of memory access data are determined.

6. The method according to claim 5, characterized in that, The performance data of each computing core includes the communication time of each computing core. The hardware parameters include: the communication bandwidth of each computing core, the communication latency of each computing core, and the communication bandwidth utilization of each computing core. The model running configuration parameters also include: the data block size and numerical precision of the data blocks in each computing core. The performance modeling of the in-core analysis task of each computing core based on the hardware parameters and the model running configuration parameters yields the performance data of each computing core, including: The communication time of each computing core is determined based on the data block size, the numerical precision, the communication bandwidth, the communication delay, the communication bandwidth utilization, and the communication modeling model.

7. The method according to claim 6, characterized in that, The performance data of each computing core includes: the total execution time, active data volume, weighted data volume, and key-value pair cache data volume of each computing core; the performance modeling of the in-core analysis task of each computing core based on the hardware parameters to obtain the performance data of each computing core includes: Based on the vector computation execution time, the matrix computation execution time, the memory access time, the amount of data accessed in memory, the communication time, the preset overlap strategy parameters, and the overlap modeling model, the total execution time, the amount of activated data, the amount of weighted data, and the amount of key-value pair cached data for each computing core are determined.

8. The method according to claim 7, characterized in that, The parallel strategy includes: card-level parallel strategy and core-level parallel strategy. The performance data of each computing core is integrated according to the parallel strategy to obtain the performance data of the large language model, including: The total execution time, active data volume, weight data volume, key-value pair cache data volume, vector computation execution time, matrix computation execution time, memory access time, memory access data volume, and communication time of each computing core are integrated according to the core-level parallelism strategy to obtain the performance data of each accelerator card. The performance data of each accelerator card are integrated according to the card-level parallel strategy to obtain the performance data of the large language model.

9. The method according to claim 1, characterized in that, Also includes: The performance data of the large language model is standardized to obtain the performance analysis results of the standardized large language model.

10. The method according to claim 9, characterized in that, The standardization process for the performance data of the large language model, to obtain the performance analysis results of the standardized large language model, includes: The performance data of the large language model is standardized according to the hierarchical structure of the large language model to obtain the performance data corresponding to each level of the standardized large language model.

11. The method according to claim 9, characterized in that, The standardization process for the performance data of the large language model, to obtain the performance analysis results of the standardized large language model, includes: The performance data of the large language model is standardized according to the task stages of the large language model to obtain the standardized performance data corresponding to each task stage of the large language model; wherein, the task stages include: the forward propagation stage of the training task, the backward propagation stage of the training task, the pre-filling stage of the inference task, and the generation stage of the inference task.

12. The method according to claim 9, characterized in that, The standardization process for the performance data of the large language model, to obtain the performance analysis results of the standardized large language model, includes: The performance data of the large language model is standardized according to the hardware indicators of the large language model to obtain the hardware indicators of the standardized large language model at each task stage; wherein, the hardware indicators include: on-chip network utilization, memory bandwidth utilization, vector computing unit utilization, matrix computing unit utilization, and storage space utilization.

13. The method according to claim 9, characterized in that, The standardization process for the performance data of the large language model, to obtain the performance analysis results of the standardized large language model, includes: The performance data of the large language model is standardized according to the performance indicators of the large language model to obtain the standardized performance indicators of the large language model at each task stage; among which, the performance indicators include: first token generation delay, inter-token generation delay, total generation delay, average token generation time and token throughput.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 13.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 13.

Images