Model performance estimation method and device, and computer device

Abstract

The application relates to the technical field of artificial intelligence chips, and discloses a model performance estimation method and device and a computer device, wherein model configuration data and a candidate distributed strategy of a target model are determined; an original calculation graph corresponding to a single-card deployment state is converted based on strategy configuration data of the model configuration data and the candidate distributed strategy, so that a distributed overhead calculation graph corresponding to a multi-card deployment state is obtained; performance of the candidate distributed strategy is estimated according to basic overhead and additional overhead in the distributed overhead calculation graph, so that strategy performance data of the target model under the candidate distributed strategy is obtained, the single-card model is converted into the multi-card model, and simulation calculation of a multi-card interconnection mode is realized on the premise of limited hardware resources, so that the influence of a communication operator and a topological structure corresponding to the multi-card interconnection mode on the overall operation of the model is reflected, and the upper limit of the model performance can be accurately evaluated in the pre-silicon simulator verification stage.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence chip technology, and in particular to a model performance prediction method, apparatus and computer equipment. Background Technology

[0002] Large models are the main application scenario for current artificial intelligence chips. Therefore, how to obtain the estimated upper limit of large model performance through theoretical calculations during the design stage of artificial intelligence chips is an important aspect of examining the design verification of artificial intelligence chips.

[0003] In the design and verification phase of artificial intelligence chips, performance evaluation is one of the core steps. Related technologies typically rely on hardware simulators to directly run large models to evaluate the actual performance of the chip.

[0004] Since it is difficult to directly simulate multi-GPU interconnect modes in the pre-silicon stage, an effective model performance prediction method is needed to predict the performance ceiling of large models under multi-GPU distributed deployment. Summary of the Invention

[0005] This application provides a model performance prediction method, apparatus, and computer device, which solves the technical problem in related technologies that silicon pre-verification relies on hardware simulators and is difficult to reproduce large-scale distributed scenarios, and provides a reasonable and effective performance prediction scheme for artificial intelligence chip design.

[0006] To achieve the above objectives, the main technical solutions adopted in this application include:

[0007] In a first aspect, embodiments of this application provide a model performance prediction method, the method comprising:

[0008] The model configuration data and candidate distributed strategies of the target model are determined; wherein, the target model has a single-card deployment state and a multi-card deployment state in the simulator verification stage before silicon; the single-card deployment state corresponds to the original computation graph;

[0009] Data is segmented based on the model configuration data and the policy configuration data of the candidate distributed strategy to construct distributed nodes for multi-card parallel execution based on the single-card nodes in the original computation graph, thereby obtaining the distributed overhead computation graph corresponding to the multi-card deployment state; wherein, the nodes of the distributed overhead computation graph are used to represent the basic overhead of the operation operator, and the edges of the distributed overhead computation graph are used to represent the additional overhead between connected operation operators.

[0010] The performance of the candidate distributed strategies is estimated based on the distributed overhead calculation graph to obtain the strategy performance data of the target model under the candidate distributed strategies.

[0011] Optionally, the step of constructing distributed nodes for multi-card parallel execution based on single-card nodes in the original computation graph to obtain the distributed overhead computation graph corresponding to the multi-card deployment state includes:

[0012] The single-card nodes in the original computation graph are transformed into distributed nodes that execute in parallel across multiple cards, and communication operator nodes are inserted into the original computation graph based on the strategy configuration data to obtain the distributed overhead computation graph.

[0013] Optionally, the basic overhead includes computational overhead, communication overhead, and video memory overhead, and the additional overhead includes resegmentation overhead;

[0014] The step of performing performance prediction on the candidate distributed strategies based on the distributed overhead calculation graph to obtain the performance data of the target model under the candidate distributed strategies includes:

[0015] The performance data of the target model under the candidate distributed strategy is obtained by summarizing the computational overhead, the communication overhead, the memory overhead, and the resharding overhead.

[0016] Optionally, when the candidate distributed strategy is a data parallel strategy, the step of constructing distributed nodes for multi-card parallel execution based on single-card nodes in the original computation graph to obtain the distributed overhead computation graph corresponding to the multi-card deployment state includes:

[0017] The single-device node corresponding to each operation operator in the original computation graph is converted into a data parallel node; wherein, the input tensor is split along the batch processing dimension to simulate the allocation of each data fragment to the corresponding data parallel node; the model weights are split in a full-fragment data parallel manner to simulate the storage of the corresponding first fragment weights by the data parallel node.

[0018] Based on the data parallel configuration parameters, a fully aggregated operator node is inserted into the original computation graph to obtain the distributed overhead computation graph.

[0019] Optionally, when the candidate distributed strategy is a tensor parallel strategy, the step of constructing distributed nodes for multi-card parallel execution based on single-card nodes in the original computation graph to obtain the distributed overhead computation graph corresponding to the multi-card deployment state includes:

[0020] The single-device node corresponding to each operation operator in the original computation graph is converted into a tensor parallel node; wherein, the size of the input tensor remains unchanged, and the model weights are split along the tensor parallel dimension to simulate the storage of the corresponding second slice weights by the tensor parallel node.

[0021] Based on the tensor parallel configuration parameters, fully aggregated operator nodes and fully reduced operator nodes are inserted into the original computation graph to obtain the distributed overhead computation graph.

[0022] Optionally, the candidate distributed strategy includes at least one of the following: data parallel strategy, tensor parallel strategy, pipeline parallel strategy, and hybrid parallel strategy.

[0023] Optionally, the candidate distributed strategy is determined by any of the following methods:

[0024] In response to the policy configuration operation, the candidate distributed policy is generated;

[0025] In response to the strategy selection operation, the candidate distributed strategy is generated;

[0026] In response to the modification operation of the initial distributed strategy, the candidate distributed strategy is obtained;

[0027] Hardware constraint data is input into the policy generator to generate distributed policies, thus obtaining the candidate distributed policies.

[0028] Optionally, the method further includes:

[0029] Identify repeating modules in the target model; wherein, the repeating modules have labeling information;

[0030] Based on the labeling information of the repeating modules, the repeating modules are split from the target model;

[0031] Construct a partial computational subgraph for the repeating module;

[0032] The overall computational performance data of the target model is determined by estimating the performance based on the number of repeating modules and the sub-computational performance data corresponding to the partial computational subgraph.

[0033] Optionally, the repeating module is identified based on the model configuration data of the target model.

[0034] Optionally, the step of determining the overall computational performance data of the target model by performing performance estimation based on the number of repeating modules and the sub-computational performance data corresponding to the partial computational subgraph includes:

[0035] The number of performance calculation repetitions is configured based on the number of repeating modules; wherein the number of performance calculation repetitions is equal to the number of repeating modules minus 1;

[0036] The sub-computation performance data is repeated according to the number of times the performance calculation is repeated, so as to restore the overall computation performance data of the target model.

[0037] Optionally, the sub-computation performance data corresponding to the partial computation subgraph can be determined in the following way:

[0038] Calculate the operator performance data for each computation operator in the aforementioned computational subgraph;

[0039] The sub-computation performance data is obtained by summarizing the operator performance data of each computation operator.

[0040] Secondly, embodiments of this application provide a model performance prediction device, the device comprising:

[0041] A distributed strategy determination module is used to determine the model configuration data and candidate distributed strategies for the target model; wherein, the target model corresponds to a single-card deployment state and a multi-card deployment state in the simulator verification stage before silicon; the single-card deployment state corresponds to the original computation graph;

[0042] The overhead computation graph generation module is used to perform data segmentation based on the model configuration data and the policy configuration data of the candidate distributed policy, so as to construct a distributed node for multi-card parallel execution based on the single-card node in the original computation graph, and obtain the distributed overhead computation graph corresponding to the multi-card deployment state; wherein, the nodes of the distributed overhead computation graph are used to represent the basic overhead of the operation operator, and the edges of the distributed overhead computation graph are used to represent the additional overhead between the connected operation operators.

[0043] The distributed strategy prediction module is used to predict the performance of the candidate distributed strategies based on the distributed overhead calculation graph, and obtain the strategy performance data of the target model under the candidate distributed strategies.

[0044] Thirdly, embodiments of this application provide a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip executes the computer program to implement the method described in any of the above-mentioned embodiments.

[0045] In this embodiment, the model configuration data and candidate distributed strategies of the target model are determined; based on the model configuration data and the strategy configuration data of the candidate distributed strategies, the original computation graph corresponding to the single-card deployment state is transformed to obtain the distributed overhead computation graph corresponding to the multi-card deployment state; the performance of the candidate distributed strategies is predicted according to the basic overhead and additional overhead in the distributed overhead computation graph to obtain the strategy performance data of the target model under the candidate distributed strategies, thereby realizing the transformation of the single-card model into a multi-card model, and realizing the simulation calculation of the multi-card interconnection mode under the premise of limited hardware resources based on the multi-card model, so as to reflect the impact of the communication operators and topology corresponding to the multi-card interconnection mode on the overall operation of the model, so as to accurately evaluate the upper limit of model performance in the simulator verification stage before silicon. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 A schematic structural diagram of a general-purpose graphics processor provided in the embodiments of this application;

[0048] Figure 2a A flowchart illustrating the model performance prediction method provided in this application embodiment;

[0049] Figure 2b A schematic diagram of the original computational graph provided for embodiments of this application;

[0050] Figure 3 A flowchart illustrating the model performance prediction method provided in this application embodiment;

[0051] Figure 4a A flowchart illustrating the model performance prediction method provided in this application embodiment;

[0052] Figure 4b A schematic diagram of the distributed overhead computation graph corresponding to the data parallelism strategy provided in the embodiments of this application;

[0053] Figure 5a A flowchart illustrating the model performance prediction method provided in this application embodiment;

[0054] Figure 5b A schematic diagram of the distributed overhead computation graph corresponding to the tensor parallel strategy provided in the embodiments of this application;

[0055] Figure 6 A flowchart illustrating the model performance prediction method provided in this application embodiment;

[0056] Figure 7 A schematic diagram illustrating the marking information of repeating modules in an embodiment of this application;

[0057] Figure 8 A schematic diagram of the framework of the model performance prediction device provided in the embodiments of this application;

[0058] Figure 9 A schematic structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] The model performance method in this application is based on Artificial Intelligence (AI). Artificial intelligence utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a manner similar to human intelligence.

[0061] The artificial intelligence processor involved in this application can be any one of GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), or GPGPU (General-Purpose Graphics Processing Unit). The following explanation uses a general-purpose graphics processing unit (GPGPU) as an example.

[0062] Figure 1 The diagram shown is a schematic representation of a general-purpose graphics processing unit (GPGPU). Please refer to [link / reference]. Figure 1 A general-purpose graphics processing unit (GPU) is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), which includes... Figure 1The diagram shows streaming processor clusters 1 through M-1, where M is a positive integer greater than 1. In a general-purpose graphics processing unit (GPU), one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple execution units (EUs). Each execution unit includes an arithmetic logic unit (ALU), a floating-point unit, etc., and is used to execute specific computational tasks. Each execution unit includes a thread local register (TLR) to store source and destination data related to the computational task. A global memory buffer (GMB) connects the EUs and high-bandwidth memory (HBM) for data transfer, improving the transmission efficiency between the EUs and HBM. HBM provides large-capacity, low-latency storage to support the high-performance computing needs of GPUs. Instructions enter the SPC from the outside and are distributed to each EU for execution. Input data is read from the HBM, transmitted through the GMB to the SPC, and then distributed to the EUs for processing. The results (output data) processed by EU are returned to GMB via SPC and finally written to HBM.

[0063] Since a large-scale model can have hundreds of billions of parameters, a single machine or single GPU is no longer sufficient to handle the training task. Therefore, distributed training is widely used for large-scale model training in artificial intelligence. Distributed training refers to using multiple machines to work collaboratively to accelerate the training process of large deep learning models. In distributed training, the original model, dataset, and training process are broken down and distributed across multiple machines for simultaneous processing, thereby effectively utilizing more computing resources and shortening training time.

[0064] For large models, communication and computation time often become the main factors affecting model execution time. However, it is difficult for simulators to deploy a complete 4-card interconnected machine or 8-card interconnected machine. They can usually only deploy a maximum of 2 cards, and the number of execution units is limited.

[0065] Furthermore, during the training of large models, different distributed algorithms have a significant impact on model performance. It is necessary to use a method to measure the performance of different algorithms, analyze the performance bottlenecks of different large models under different distributed algorithms, and provide guidance and suggestions for optimizing the performance of large models. Mainstream distributed algorithms include data-parallel algorithms (DP), tensor-parallel algorithms (TP), and pipelined parallel algorithms (PP).

[0066] Data-parallel algorithms refer to partitioning the model's training dataset into multiple subsets, which are then distributed across multiple devices. Each device holds a complete copy of the model, allowing it to independently train its assigned subset. For example, each device can independently perform forward and backward propagation computations on its subset. For instance, a training dataset might be partitioned into three parts and assigned to device 0, device 1, and device 2, each with a complete model. After each device completes backpropagation, the gradients calculated by each device are aggregated, and the global model parameters are updated.

[0067] Tensor parallelism refers to splitting a layer (or operator) of a model and distributing some of the operator's weights across different devices, thereby reducing the memory usage on each device (e.g., a GPU). A neural network model consists of multiple layers, each of which can be understood as a function that performs specific mathematical operations on the input tensor. For example, fully connected layers can perform linear transformations on tensors, convolutional layers can perform convolution operations, pooling layers can perform pooling operations, etc., to extract or transform features. These operations can change the content and shape of the tensor, achieving a gradual transformation from raw data to a higher-level abstract representation. In tensor parallelism, the tensor of a layer is divided into multiple parts, and these parts are distributed across multiple devices, such as device 0, device 1, and device 2. Each device is responsible for only a part of the model's computation and exchanges necessary intermediate results through communication.

[0068] Pipeline parallelism refers to dividing different layers of a model into multiple stages, each assigned to a different device to form a pipeline. Activation values ​​and gradients are passed sequentially between these devices. For example, during forward propagation, each device passes activation values ​​to the device containing the next pipeline stage; during backward propagation, each device passes gradients back to the device containing the previous pipeline stage. This allows multiple devices to be used for computation simultaneously, improving computational efficiency. For instance, if a model has four layers, these four layers can be divided into three stages, with each stage assigned to a device. For example, the first layer might be assigned to device 0, the second to device 1, and the third and fourth to device 2.

[0069] As model size increases, using a single parallel mode often fails to simultaneously meet the requirements of device memory limitations and high computational efficiency. Therefore, for large-scale models, it is usually necessary to combine multiple parallel techniques, such as data parallelism, tensor parallelism, and pipelined parallelism, for distributed training.

[0070] Furthermore, Table 1 illustrates the characteristics, advantages, and disadvantages of different distributed algorithms:

[0071] Table 1: Comparison of various distributed algorithms

[0072]

[0073] In related technologies, the training and inference of large models typically rely on multiple GPUs or distributed computing frameworks. During the chip design phase (pre-silicon stage), due to the limited resources of hardware simulators (such as FPGAs, Zebu, etc.), it is difficult to deploy a complete multi-GPU interconnect environment (such as a 4-GPU cluster or an 8-GPU cluster). Furthermore, the multi-GPU cluster interconnect performance of simulators may differ significantly from that of actual machines, making it impossible to accurately assess the impact of communication overhead on performance. As shown in Table 1, distributed algorithms (such as data-parallel DP, tensor-parallel TP, pipelined parallel PP, etc.) each have their advantages and disadvantages. Related technologies lack a systematic method to simulate the communication overhead and performance bottlenecks under different distributed strategies in the pre-silicon stage.

[0074] Therefore, this application proposes a model performance prediction method. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown here. Please refer to... Figure 2a , Figure 2a The diagram shown is a flowchart of the model performance prediction method in an embodiment of this application. The model performance prediction method includes the following steps:

[0075] S110. Determine the model configuration data of the target model and candidate distributed strategies.

[0076] The target model undergoes simulator verification in the pre-silicon phase, which includes both single-GPU and multi-GPU deployment states. The single-GPU deployment state corresponds to the original computation graph. Candidate distributed strategies include at least one of the following: data parallelism, tensor parallelism, pipelined parallelism, and hybrid parallelism.

[0077] In this embodiment, the target model refers to the artificial intelligence model to be evaluated for performance, which includes executable computational logic and associated parameters. Specifically, the target model can be a large language model based on the Transformer architecture (such as Llama, GPT series), or a convolutional neural network model (such as ResNet). The model configuration data of the target model refers to descriptive data describing the model structure and parameters, including but not limited to the number of model layers, the number of parameters per layer, the type of activation function, and the amount of GPU memory used.

[0078] In this embodiment, the target model has two deployment modes. Single-GPU deployment refers to the model running on a single computing device (such as a GPU), while multi-GPU deployment refers to the model being distributed across multiple computing devices using a distributed strategy. For example, in single-GPU deployment, the target model is entirely loaded onto a single GPU; in multi-GPU deployment, the target model is split across four GPUs using a distributed strategy.

[0079] In this embodiment, the candidate distributed strategy is a set of parallel computing schemes designed to achieve multi-GPU deployment. The number of candidate distributed strategies can be one or more. Specifically, the candidate distributed strategies include basic strategies such as data parallelism (DP), tensor parallelism (TP), pipelined parallelism (PP), and expert parallelism (EP), or hybrid parallelism strategies. The hybrid parallelism strategy can be a 3D parallelism strategy, that is, a combination of DP, TP, and PP.

[0080] In this embodiment, by determining the model configuration data of the target model and the candidate distributed strategies, input parameters are provided for the subsequent distributed overhead computation graph transformation. For example, the model configuration data is used to quantify the basic overhead of operators, and the candidate distributed strategies are used to define the structure of the distributed overhead computation graph.

[0081] In this embodiment, candidate distributed strategies are determined using any of the following methods:

[0082] In response to the policy configuration operation, candidate distributed policies are generated.

[0083] In response to the policy selection operation, candidate distributed policies are generated.

[0084] In response to modifications to the initial distributed strategy, candidate distributed strategies are obtained.

[0085] Hardware constraint data is input into the policy generator to generate distributed policies, resulting in candidate distributed policies.

[0086] The above method for determining candidate distributed strategies can support modifications to the model computation graph for various machine topologies and distributed strategies, allowing for automatic selection or manual configuration of distributed strategies.

[0087] S120. Based on the model configuration data and the policy configuration data of the candidate distributed policies, the original computation graph corresponding to the single-card deployment state is transformed to obtain the distributed overhead computation graph corresponding to the multi-card deployment state.

[0088] In this distributed overhead computation graph, nodes represent the basic overhead of an operation operator, and edges represent the additional overhead between connected operations operators.

[0089] In this embodiment, the strategy configuration data is a set of technical parameters for implementing a specific candidate distributed strategy. The original computation graph refers to an abstract representation of the computation process of the model in a single-card deployment state. For example, please refer to... Figure 2b , Figure 2b The diagram shown is a schematic of the original computation graph. The original computation graph can contain operators (such as matrix multiplication and convolution) and their input-output relationships.

[0090] In this embodiment, the distributed overhead computation graph refers to the computation flow representation obtained after transforming the original computation graph using a distributed strategy. Its nodes and edges represent the basic and additional overhead in distributed deployment, respectively. For example, nodes in the distributed overhead computation graph can correspond to the computation time of operators (such as the number of MAC operations in matrix multiplication), and edges can correspond to the cross-device communication time (such as the latency of All-Reduce operations). It should be noted that the distributed overhead computation graph adopts a graph structure data form, consisting of two basic elements: nodes and edges. These two elements work together to form the mathematical model foundation for pre-silicon performance prediction. The distributed overhead computation graph can reflect the computation and data flow logic of the target model under multi-GPU deployment in the pre-silicon simulator verification stage. The distributed overhead computation graph embeds complete overhead information, facilitating the quantification and simulation of distributed execution performance in the pre-silicon environment. Each node in the distributed overhead computation graph corresponds to a basic computation or communication operator, and the nodes are labeled with computation overhead, communication overhead, and memory overhead, etc. In the distributed cost computation graph, the edges represent the data dependencies between nodes, and each edge is marked with an additional cost, such as the re-sharding cost caused by policy mismatch.

[0091] Understandably, by expressing candidate distributed strategies as a concrete and quantifiable graph structure through a distributed overhead computation graph, the technical challenge of not being able to practically run multi-GPU models in the pre-silicon stage is solved. The process of determining the distributed overhead computation graph can be viewed as constructing a performance simulation model corresponding to the candidate distributed strategies. Through the edges and nodes of this model, the performance influencing factors in a multi-GPU distributed environment can be fully modeled.

[0092] In this embodiment, the basic overhead of the operator refers to the computation time, communication time, and memory usage of the operator across multiple GPUs. Additional overhead refers to the extra time consumed due to cross-device communication or synchronization in a distributed deployment.

[0093] In some embodiments, the original computation graph corresponding to the single-card deployment state is transformed based on model configuration data and policy configuration data of candidate distributed strategies to obtain a distributed overhead computation graph corresponding to the multi-card deployment state, including step S122. Step S122 involves data segmentation based on model configuration data and policy configuration data of candidate distributed strategies to construct distributed nodes for parallel execution of multiple cards based on the single-card nodes in the original computation graph, thereby obtaining the distributed overhead computation graph corresponding to the multi-card deployment state.

[0094] Specifically, based on model configuration data, the operation operators corresponding to the deployment status of a single card are determined, and these operation operators are represented as nodes to construct the original computation graph. To achieve distributed allocation of computational load, the tensor data and / or weight data of the single-card nodes are segmented and distributed across multiple computing devices. This allows for the construction of distributed nodes for multi-card parallel execution based on the single-card nodes in the original computation graph. For example, these distributed nodes can be obtained by transforming or converting the single-card nodes in the original computation graph, or they can be newly added distributed nodes based on the single-card nodes in the original computation graph. It can be understood that whether the distributed nodes are transformed from single-card nodes or newly added distributed nodes, they are all nodes corresponding to operation operators.

[0095] Furthermore, basic or fundamental overhead is set for the nodes corresponding to the operation operators, and additional or supplementary overhead due to cross-device deployment is set for the edges connecting the operation operators, resulting in a distributed overhead computation graph corresponding to the multi-GPU deployment state. By converting the original computation graph into a distributed overhead computation graph, the conversion from a single-GPU model to a multi-GPU model is realized. The performance ceiling of the target model under a multi-GPU distributed strategy can be predicted without actual hardware deployment, which is beneficial for evaluating the impact of different interconnect architectures on model performance in the pre-silicon stage, thereby optimizing hardware design. For example, the strategy configuration data of candidate distributed strategies can be parsed to determine the communication operations corresponding to the candidate distributed strategies. Based on the communication operations corresponding to the candidate distributed strategies, corresponding operators are added to the original computation graph, such as adding nodes representing communication operators to the original computation graph.

[0096] S130. Based on the distributed overhead calculation graph, the performance of the candidate distributed strategies is estimated to obtain the strategy performance data of the target model under the candidate distributed strategies.

[0097] In this embodiment, performance estimation refers to the process of quantitatively evaluating the operational efficiency of candidate distributed strategies based on a distributed overhead computation graph. Strategy performance data refers to the results of performance estimation, including but not limited to total time consumption, throughput, speedup ratio, or memory usage. Specifically, after converting the original computation graph into a distributed overhead computation graph, the total overhead of the target model under the candidate distributed strategies is calculated, including the sum of the basic overhead of each node and the sum of the additional overhead of each edge.

[0098] Furthermore, basic overhead can include computational overhead, communication overhead, and memory overhead, while additional overhead can include resharding overhead. Specifically, computational overhead can refer to the equivalent single-card computational overhead after deploying the computation corresponding to a single device to multiple devices. For example, under a distributed strategy, each device (such as a GPU) handles a portion of the computational tasks, and computational overhead can quantify the computational burden of each device. For example, computational overhead can be the pure computational time or computation duration consumed by executing the corresponding operation operator on the computing device marked on the node in the distributed overhead computation graph. This computational overhead can be distinguished according to different operator types. It is determined by identifying and modeling the computational bottleneck types of different operation operators. Different computational bottleneck types include: 1) operators with vcore instruction issuance bottlenecks, such as the dwc operator; 2) tcore computational bottlenecks, such as the general matrix multiplication (gemm) operator; 3) memory access bottlenecks, such as pointwise operation operators; 4) for some more complex operators, such as self-attention operators, if there are data dependencies, the operator needs to be segmented according to the algorithm. By taking the slowest position among the various types of computation and memory access overhead for different operators, the computation overhead can be calculated.

[0099] Communication overhead involves communication between devices, such as the All-Reduce operation. Communication overhead refers to the time spent on cross-device communication (such as All-Reduce and All-Gather operations) of the target model in a multi-GPU deployment, after being distributed to each device according to a distributed strategy, and the communication time incurred by devices in synchronizing or aggregating data. For example, communication overhead can be the time consumed by communication operations (such as All-Reduce and All-Gather) corresponding to some nodes in the distributed overhead calculation graph under a specific topology. Communication can utilize various protocols, including low-latency or Simple protocols, and algorithms such as ring or tree structures based on different topologies. For a given communication algorithm, the communication overhead determines whether performance is latency-driven or bandwidth-driven based on the size of the data blocks to be transmitted. If bandwidth-driven, the time is calculated using the topology's bandwidth; if latency-driven, the latency is directly calculated.

[0100] GPU memory overhead refers to the amount of GPU memory used by the target model in a multi-GPU deployment (such as parameter storage and intermediate activation value storage), allocated to each device according to a distributed strategy, and the amount of GPU memory used by each device. For example, the size of the GPU memory overhead primarily depends on the weight parameters and the size of the computational data after the distributed strategy partitioning during computation, including the space occupied by the optimizer during training. Additionally, some buffer space needs to be reserved for distributed communication.

[0101] Resharding overhead refers to the additional time consumed in a distributed strategy due to adjustments in data distribution methods (such as sharding and replication) or computation order (such as pipeline stage division). For example, in a tensor parallel strategy, input data needs to be split and rearranged across different devices. This process introduces additional memory copies and synchronization operations, the overhead of which is resharding overhead. This resharding overhead is used to quantify the additional requirements of distributed strategies for data preprocessing and post-processing. Resharding overhead can be illustrated by thinking of it as an implicit communication operator. If multiple operators use different data splitting methods in a distributed system, distributed send / receive operations are required when interactions occur.

[0102] Accordingly, the performance of candidate distributed strategies is estimated based on the distributed overhead computation graph to obtain the performance data of the target model under the candidate distributed strategies, including: summarizing the computational overhead, communication overhead, memory overhead and resharding overhead to obtain the performance data of the target model under the candidate distributed strategies.

[0103] Specifically, computational overhead, communication overhead, memory overhead, and resharding overhead are integrated into a quantifiable comprehensive performance index, serving as performance data for the target model under candidate distributed strategies. This performance data reflects the degree of impact of different overheads on model performance. This enables a comprehensive evaluation of the performance of candidate distributed strategies, providing multi-dimensional optimization basis for the chip design stage, such as selecting the candidate distributed strategy with the lowest overhead. By subdividing and quantifying various overheads in distributed strategies, the limitations of traditional techniques that only focus on the single dimension of computational overhead are overcome. By summarizing various basic and additional overheads, a data foundation is provided for dynamically adjusting the performance evaluation model, thereby achieving more accurate performance prediction.

[0104] In the above embodiments, the model configuration data of the target model and candidate distributed strategies are determined; based on the model configuration data and the strategy configuration data of the candidate distributed strategies, the original computation graph corresponding to the single-card deployment state is transformed to obtain the distributed overhead computation graph corresponding to the multi-card deployment state; the performance of the candidate distributed strategies is predicted according to the basic overhead and additional overhead in the distributed overhead computation graph to obtain the strategy performance data of the target model under the candidate distributed strategies, thereby realizing the transformation of the single-card model into a multi-card model, and realizing the simulation calculation of the multi-card interconnection mode under the premise of limited hardware resources based on the multi-card model, so as to reflect the impact of the communication operators and topology corresponding to the multi-card interconnection mode on the overall operation of the model, so as to accurately evaluate the upper limit of model performance in the simulator verification stage before silicon.

[0105] In some embodiments, a distributed node for multi-card parallel execution is constructed based on the single-card node in the original computation graph to obtain a distributed overhead computation graph corresponding to the multi-card deployment state. This includes: transforming the single-card node in the original computation graph into a distributed node for multi-card parallel execution, and inserting communication operator nodes into the original computation graph based on policy configuration data to obtain the distributed overhead computation graph. Further, please refer to... Figure 3 Based on model configuration data and policy configuration data of candidate distributed policies, the original computation graph corresponding to the single-card deployment state is transformed to obtain the distributed overhead computation graph corresponding to the multi-card deployment state, which may include:

[0106] S310. Data is split based on model configuration data and strategy configuration data to transform single-card nodes in the original computation graph into distributed nodes that can be executed in parallel by multiple cards.

[0107] S320. Based on the policy configuration data, a communication operator node is inserted into the original computation graph to obtain a distributed overhead computation graph.

[0108] In this context, a single-card node refers to an operation operator node designed to run on a single computing device in the original computation graph. A distributed node executing in parallel across multiple cards can be an operation operator node designed to run on multiple computing devices in a distributed overhead computation graph. A communication operator node can be a communication operation unit inserted to coordinate the execution of distributed nodes. Specifically, the type of communication operator node is defined by policy configuration data; for example, a communication operator node can be a set of communication nodes.

[0109] Specifically, to achieve distributed allocation of computing load, the tensor data and / or weight data of a single-card node are segmented and distributed across multiple computing devices, transforming the single-card node in the original computation graph into a distributed node executing in parallel across multiple cards, thus generating a computational flow representation in a distributed deployment scenario. Further, communication operations are added at specific locations in the computation graph, such as the TP strategy: inserting an All-Gather node after the matrix multiplication node. This achieves graph structure reconstruction, improving the data flow connections. For example, the original computation graph "Computation Node A → Computation Node B" is reconstructed as "Computation Node A → All-Reduce Node → Computation Node B". Further, corresponding overheads are added or labeled for computation operator nodes and communication operator nodes, resulting in a distributed overhead computation graph. In this embodiment, the completeness modeling of distributed computing is achieved by inserting communication operator nodes, explicitly expressing the data coordination process between devices. This allows for the quantification of communication overhead and the identification of performance bottlenecks.

[0110] It should be noted that for the PP strategy, dividing the operators in the original computation graph onto different devices does not involve changing the computation graph itself.

[0111] In this embodiment, when the candidate distributed strategy is a data parallel strategy, please refer to [link to relevant documentation]. Figure 4a Based on the single-card nodes in the original computation graph, a distributed node for multi-card parallel execution is constructed, resulting in a distributed overhead computation graph corresponding to the multi-card deployment state, including:

[0112] S410. Convert the single-device node corresponding to each operation operator in the original computation graph into a data-parallel node.

[0113] S420. Insert fully aggregated operator nodes into the original computation graph according to the data parallel configuration parameters to obtain a distributed overhead computation graph.

[0114] Specifically, the input tensor is partitioned along the batch processing dimension to simulate the allocation of each data partition to the corresponding data parallel node. The model weights are partitioned using a fully partitioned data parallel approach, with the first partition weight stored on the simulated data parallel node.

[0115] Specifically, please refer to Figure 4b , Figure 4bThe diagram illustrates the distributed overhead computation graph. First, by converting the single-device nodes of the original computation graph into data-parallel nodes and partitioning the input tensor along the batch processing dimension, the computational distribution under the data-parallel strategy is achieved. Second, model weights are partitioned using a fully sharded data-parallel approach, ensuring that each device only stores and computes the parameters of its corresponding shard. Finally, by inserting all-gather operator nodes, the parameter synchronization process in distributed training is simulated. The generated distributed overhead computation graph accurately characterizes the computation time, communication latency, and memory usage under the data-parallel strategy, enabling a more comprehensive assessment of the overall hardware resource requirements of the data-parallel strategy, thus providing a theoretical basis for performance optimization during the chip design phase.

[0116] In this embodiment, when the candidate distributed strategy is a tensor parallel strategy, please refer to [link to relevant documentation]. Figure 5a Based on the single-card nodes in the original computation graph, a distributed node for multi-card parallel execution is constructed, resulting in a distributed overhead computation graph corresponding to the multi-card deployment state, including:

[0117] S510. Convert the single-device node corresponding to each operation operator in the original computation graph into a tensor parallel node.

[0118] S520. Based on the tensor parallel configuration parameters, insert full aggregation operator nodes and full reduction operator nodes into the original computation graph to obtain the distributed overhead computation graph.

[0119] In this model, the input tensor size remains constant, and the model weights are partitioned along the tensor parallel dimension, with the corresponding second-part weights stored in the simulation tensor parallel node. The All-Gather operator node is a collection communication node used to gather scattered data; its function is to concatenate the fragmented data from each device into a complete tensor. For example, in tensor parallelism, it concatenates partial output tensors from multiple devices into a complete result. The All-Reduce operator node is a collection communication node used to aggregate data across devices; its function is to perform a summation operation on the input tensors of each device and broadcast the results, such as summing the gradient tensors of each device in a gradient synchronization scenario.

[0120] Specifically, please refer to Figure 5b , Figure 5bThe diagram illustrates a distributed overhead computation graph. First, the single-device nodes of the original computation graph are transformed into tensor-parallel nodes, while maintaining the size of the input tensor. The model weights are then partitioned along the tensor-parallel dimension to simulate the storage of the corresponding second-segment weights in the tensor-parallel nodes. During the forward propagation phase, partial results computed by each device are aggregated using a full-gather operator (e.g., All-Gather) to ensure consistent output across all devices. During the backpropagation phase, gradients computed by each device are synchronized using a full-reduce operator (e.g., All-Reduce) to guarantee global consistency of model parameters. Finally, the tensor-parallel nodes and communication operator nodes are integrated to form a complete distributed computation flow representation. For example, nodes in the distributed overhead computation graph represent the basic overhead of the operation operator, while edges represent the additional overhead of cross-device communication (e.g., the latency of All-Gather). The technical advantage of this step is that, through theoretical modeling and algorithm simulation, the performance ceiling of the target model under a multi-GPU tensor-parallel strategy can be predicted without actual hardware deployment.

[0121] In some embodiments, please refer to Figure 6 The method also includes the following steps:

[0122] S610. Identify the repeating modules in the target model.

[0123] In this context, a repeating module refers to a functional unit in the target model that is used multiple times. Repeating modules are identified based on the model configuration data of the target model and are used to distinguish the location and attributes of different repeating modules. Please refer to [link / reference]. Figure 7 The duplicate modules have a scope (702) label; the scope refers to data used to uniquely identify the duplicate modules. The scope is used to uniquely identify the location and attributes of the duplicate modules, ensuring the accuracy of the splitting operation. Specifically, it identifies reusable functional units in the model structure, thus providing a basis for subsequent performance prediction. For example, by parsing the target model's configuration file, it can be determined that its 61 Transformer layers are all duplicate modules, each containing the same combination of operators (such as linear layers and attention mechanisms).

[0124] S620. Based on the labeling information of repeating modules, split repeating modules from the target model.

[0125] Specifically, based on the tagging information, repetitive modules in the target model are extracted from the overall structure to form independent functional units. The splitting operation is implemented by parsing the target model's configuration file, ensuring the integrity and independence of each repetitive module. For example, in this embodiment, if the target model contains 61 Transformer layers, then __module.layers.1 to __module.layers.61 can be split layer by layer using the tagging information to generate 61 independent repetitive modules.

[0126] S630, Construct a partial computational subgraph for repeating modules.

[0127] In this embodiment, a partial computational subgraph refers to a small-scale computational graph composed of a single repeating module and its internal operators, used to quantify the module's performance metrics (such as computational load, memory usage, and communication overhead). The construction process of the partial computational subgraph includes extracting the operator structure of the repeating module, defining the input and output dimensions, and generating the corresponding computational flow.

[0128] S640. Based on the number of repeating modules and the sub-computation performance data corresponding to some computational subgraphs, the overall computational performance data of the target model is determined.

[0129] The sub-computational performance data refers to the summarized performance metrics of each operator in a partial computational subgraph, including computational cost (FLOPs), memory usage (GPU memory requirements), and communication overhead. The overall computational performance data refers to the theoretical upper limit of the target model's performance, including total computational cost (TFLOPs), peak GPU memory usage (GB), and communication overhead in distributed training (GB / s). The overall computational performance data is used to evaluate the adaptability of AI chips or clusters. Specifically, the sub-computational performance data is obtained by traversing the operators in the partial computational subgraph. Through linear superposition or nonlinear correction algorithms, performance is summarized and extrapolated based on the number of repeating modules and the sub-computational performance data corresponding to the partial computational subgraph to obtain the overall computational performance data of the target model.

[0130] The above embodiments can significantly reduce the resource consumption for performance prediction of the target model (e.g., reducing memory usage from TB levels to the GPU memory required for a few layers of computation graph), while supporting rapid testing of the performance impact of different architectural changes (e.g., adjusting the number of layers or parameter scale). The performance estimation operation based on the sub-computation performance data of repeating modules in this embodiment is simple to implement and applicable to various large models, making it very convenient to test the performance impact of different architectural changes on large models. Furthermore, even without a complete computation graph, changing the input batch or sequence is easy, allowing for the real-time construction of new small computation graphs before calculating the performance of the complete computation graph. This enables deployment of large models that are difficult to fully deploy to a server, and improves efficiency for large models that have already been deployed.

[0131] In some embodiments, performance estimation is performed based on the number of repeating modules and the sub-computation performance data corresponding to the partial computation subgraphs to determine the overall computation performance data of the target model, including: configuring the number of performance calculation repetitions based on the number of repeating modules; and repeating the sub-computation performance data according to the number of performance calculation repetitions to restore the overall computation performance data of the target model.

[0132] In this context, the number of performance calculation repetitions equals the number of repeating modules minus 1. The number of repeating modules refers to the total number of repeating modules with the same structure and function in the target model. The number of performance calculation repetitions is a dynamically configured parameter based on the number of repeating modules in the target model, used to calculate the overall computational performance data. This parameter is obtained by subtracting 1 from the number of repeating modules to avoid extra counting of the first repeating module, thus more accurately reflecting the impact of module repeatability on overall performance. For example, in this embodiment, if the target model contains 61 repeating modules, the number of performance calculation repetitions is 61 minus 1, which equals 60, indicating that the performance of the subsequent 60 repeating modules needs to be superimposed on the performance of the first module through repeated calculations.

[0133] Specifically, the overall computational performance data of the target model is reconstructed through linear superposition. This step simulates the performance superposition effect of multiple repeated modules by multiplying the sub-computational performance data by the number of performance calculation repetitions. By combining the number of performance calculation repetitions and the sub-computational performance data, the theoretical performance ceiling of the target model is generated, including total computational cost, peak memory usage, and communication overhead in distributed training.

[0134] In some embodiments, the sub-computation performance data corresponding to a partial computation subgraph is determined by: calculating the operator performance data of each computation operator in the partial computation subgraph; and summarizing the operator performance data of each computation operator to obtain the sub-computation performance data.

[0135] It is understandable that, in the above embodiments, on the one hand, resource consumption is reduced and memory requirements are decreased by several orders of magnitude because a complete computation graph is not required. On the other hand, input parameters (such as Batch Size and Sequence Length) can be adjusted in real time to quickly generate a new computation graph, thereby improving computational efficiency.

[0136] It should be noted that this method of splitting computation and then restoring it has no additional impact on different distributed strategies:

[0137] 1) Impact on single-machine, single-card performance prediction: No impact on single-machine, single-card performance prediction.

[0138] 2) Impact on DP performance prediction: Under the premise of data parallelism, the performance of each single-card machine repeating the layer is the same, so there is no impact.

[0139] 3) Impact on TP performance prediction: TP splits each layer horizontally and distributes it to multiple GPUs. For hidden layers, the TP configuration is generally the same, so we can assume that the performance of each hidden layer is also the same, and therefore there is no impact.

[0140] 4) Impact on PP performance prediction: In PP, the layers of the complete large model are distributed to different machines, that is, each layer is computed on a complete machine. It can be assumed that the performance of the same layer is the same, so there is no impact.

[0141] As can be seen, the above method of splitting computation and then restoring is compatible with different distributed strategies (DP, TP, PP) and requires no additional adaptation.

[0142] This application provides a model performance prediction device. Figure 8 This is a schematic diagram of the framework of the model performance prediction device provided in this application embodiment. The model performance prediction device 800 may include a distributed policy determination module 810, a cost calculation graph generation module 820, and a distributed policy prediction module 830.

[0143] The distributed strategy determination module 810 is used to determine the model configuration data and candidate distributed strategies of the target model; wherein, the target model has a single-card deployment state and a multi-card deployment state in the simulator verification stage before silicon; the single-card deployment state corresponds to the original computation graph;

[0144] The overhead computation graph generation module 820 is used to perform data segmentation based on model configuration data and policy configuration data of candidate distributed policies, so as to construct distributed nodes for multi-card parallel execution based on the single-card nodes in the original computation graph, and obtain the distributed overhead computation graph corresponding to the multi-card deployment state; wherein, the nodes of the distributed overhead computation graph are used to represent the basic overhead of the operation operator, and the edges of the distributed overhead computation graph are used to represent the additional overhead between the connected operation operators.

[0145] The distributed strategy prediction module 830 is used to predict the performance of candidate distributed strategies based on the distributed cost calculation graph, and obtain the strategy performance data of the target model under the candidate distributed strategies.

[0146] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0147] This application provides a non-transitory computer-readable storage medium. The storage medium can be a non-transitory computer-readable storage medium on which one or more computer-readable instructions can be stored non-transitoryly. For example, when the computer-readable instructions are executed by a processor, one or more steps in the above-described model performance prediction method can be performed. This storage medium can be applied in electronic devices; for example, it can include a storage device in an electronic device.

[0148] The storage device may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and the processor may execute these instructions to implement various processor functions. Various application programs and various data may also be stored in the storage medium.

[0149] Storage media may include memory cards for smartphones, cache components for tablet computers, hard drives for personal computers, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0150] This application provides a computer device, such as... Figure 9 As shown, it includes at least one artificial intelligence chip 1510 and a memory 1520 connected to at least one artificial intelligence chip 1510. In this embodiment, the specific connection medium between the artificial intelligence chip 1510 and the memory 1520 is not limited. Figure 9 Taking the connection between the AI ​​chip 1510 and the memory 1520 via a bus as an example, the bus can be divided into an address bus, a data bus, a control bus, etc. In this embodiment, the memory 1520 stores instructions that can be executed by at least one AI chip 1510. By executing the instructions stored in the memory 1520, at least one AI chip 1510 can perform the steps of the above-mentioned model performance prediction method.

[0151] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0154] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0155] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0156] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method of model performance estimation, the method comprising: The method comprises: determining model configuration data and a candidate distributed strategy of a target model; wherein the target model corresponds to a single-card deployment state and a multi-card deployment state in a pre-silicon simulator verification stage; the single-card deployment state corresponds to an original computation graph; performing data splitting based on the model configuration data and strategy configuration data of the candidate distributed strategy to construct distributed nodes for multi-card parallel execution based on single-card nodes in the original computation graph, to obtain a distributed overhead computation graph corresponding to the multi-card deployment state; wherein nodes of the distributed overhead computation graph are used to represent basic overhead of operation operators, and edges of the distributed overhead computation graph are used to represent additional overhead between connected operation operators; performing performance estimation on the candidate distributed strategy according to the distributed overhead computation graph, to obtain strategy performance data of the target model under the candidate distributed strategy.

2. The method of claim 1, wherein, The method comprises: transforming the single-card nodes in the original computation graph into distributed nodes for multi-card parallel execution, and inserting a communication operator node in the original computation graph based on the strategy configuration data, to obtain the distributed overhead computation graph.

3. The method of claim 1, wherein, The basic overhead comprises computation overhead, communication overhead and memory overhead, and the additional overhead comprises re-sharding overhead caused by strategy mismatch. The method comprises: performing aggregation on the computation overhead, the communication overhead, the memory overhead and the re-sharding overhead, to obtain performance data of the target model under the candidate distributed strategy.

4. The method of claim 1, wherein, When the candidate distributed strategy is a data parallel strategy, the method comprises: transforming single-device nodes corresponding to each operation operator in the original computation graph into data parallel nodes; wherein input tensors are split along a batch processing dimension to simulate allocation of each shard data to a corresponding data parallel node; and model weights are split in a full-shard data parallel manner to simulate storage of a corresponding first shard weight by the data parallel node; inserting a full-aggregation operator node in the original computation graph according to a data parallel configuration parameter, to obtain the distributed overhead computation graph.

5. The method of claim 1, wherein, When the candidate distributed strategy is a tensor parallel strategy, the method comprises: transforming single-device nodes corresponding to each operation operator in the original computation graph into tensor parallel nodes; wherein a size of an input tensor remains unchanged, and model weights are split along a tensor parallel dimension to simulate storage of a corresponding second shard weight by the tensor parallel node; inserting, according to the tensor parallel configuration parameter, a full aggregation operator node and a full reduction operator node into the original computation graph, to obtain the distributed overhead computation graph.

6. The method according to any one of claims 1 to 5, characterized in that, The candidate distributed strategy is determined in any of the following ways: The candidate distributed strategy is generated in response to a strategy configuration operation; The candidate distributed strategy is generated in response to a strategy selection operation; The candidate distributed strategy is obtained in response to a modification operation on an initial distributed strategy; Hardware constraint data is input into a strategy generator for generation of a distributed strategy, to obtain the candidate distributed strategy.

7. The method of claim 1, wherein, The method further comprises: determining a repeated module in the target model; wherein the repeated module has marking information; splitting the repeated module from the target model based on the marking information of the repeated module; constructing a partial computation subgraph of the repeated module; performing performance calculation according to the number of the repeated modules and sub-computation performance data corresponding to the partial computation subgraph, to determine overall computation performance data of the target model.

8. The method of claim 7, wherein, The repeated module is identified according to model configuration data of the target model.

9. The method of claim 7, wherein, The performance calculation according to the number of the repeated modules and the sub-computation performance data corresponding to the partial computation subgraph, to determine overall computation performance data of the target model, comprises: configuring a performance calculation repetition number based on the number of the repeated modules; wherein the performance calculation repetition number is equal to the number of the repeated modules minus 1; repeating the sub-computation performance data according to the performance calculation repetition number, to restore the overall computation performance data of the target model.

10. The method of claim 7, wherein, The sub-computation performance data corresponding to the partial computation subgraph is determined in the following ways: calculating operator performance data of each computation operator in the partial computation subgraph; summarizing the operator performance data of each computation operator to obtain the sub-computation performance data.

11. A model performance estimation apparatus characterized by comprising: The apparatus comprises: a distributed strategy determination module configured to determine model configuration data of a target model and a candidate distributed strategy; wherein the target model corresponds to a single-card deployment state and a multi-card deployment state in a pre-silicon simulator verification stage; the single-card deployment state corresponds to an original computation graph; an overhead computation graph generation module configured to perform data splitting based on the model configuration data and strategy configuration data of the candidate distributed strategy, to construct distributed nodes for multi-card parallel execution based on single-card nodes in the original computation graph, to obtain a distributed overhead computation graph corresponding to the multi-card deployment state; wherein a node of the distributed overhead computation graph is used to represent basic overhead of an operation operator, and an edge of the distributed overhead computation graph is used to represent additional overhead between connected operation operators; a distributed strategy estimation module configured to perform performance estimation on the candidate distributed strategy according to the distributed overhead computation graph, to obtain strategy performance data of the target model under the candidate distributed strategy.

12. A computer device, comprising: An apparatus comprising a memory, an artificial intelligence chip, and a computer program stored on the memory and executable on the artificial intelligence chip, the artificial intelligence chip implementing the method according to any one of claims 1 to 10 when executing the computer program.

Images