Optimal parallel strategy for non-uniform heterogeneous chip and searching method and device of optimal parallel strategy

Abstract

The invention relates to an optimal parallel strategy for a non-uniform heterogeneous chip and a searching method and device of the optimal parallel strategy. The method comprises the steps that a first candidate set is obtained according to data parallel dimension constraints, heterogeneous chip number constraints and tensor parallel dimension constraints, and the first candidate set comprises data parallel dimension combinations, tensor parallel dimension combinations and assembly line parallel dimension combinations of all kinds of heterogeneous chips; according to the mapping constraint, the search space constraint and the video memory constraint, model layer number distribution is carried out, a parallel strategy candidate set is determined, and the parallel strategy candidate set comprises data parallel dimensions, tensor parallel dimensions, assembly line parallel dimensions and model layer number combinations of all kinds of heterogeneous chips; and for each combination in the parallel strategy candidate set, calculating single iteration time, and selecting the combination with the lowest single iteration time as an optimal parallel strategy. Compared with the prior art, the method has the advantages that load distribution and scheduling of kilocard-level large-scale heterogeneous chips are achieved, and the optimal assembly line execution efficiency is achieved.

Description

Technical Field

[0001] The present invention relates to the field of distributed training technology, and in particular to an optimal parallel strategy for non-uniform heterogeneous chips and a search method and device thereof. Background Art

[0002] Currently, heterogeneous cluster training faces challenges primarily in bandwidth, computing power, and video memory. Specifically, compared to interconnections between homogeneous chips, the heterogeneous interconnection bandwidth of many new computing chips is lower. Furthermore, there are significant differences in video memory and computing power between new chips. Due to the varying processing capabilities of heterogeneous chips, existing parallel strategies designed for homogeneous clusters struggle to fully utilize the computing power of heterogeneous chips in heterogeneous scenarios.

[0003] In addition to pipeline parallelism, other parallel strategies, such as data parallelism and tensor parallelism, require that chips within the same parallel dimension have similar computing power and can be interconnected at high speeds. However, existing pipeline parallelism strategies also have certain limitations. On the one hand, in heterogeneous distributed hybrid training systems, hardware has different characteristics. How the system can evenly or unevenly divide tasks according to model characteristics and the real-time training status of the system when computing power is unevenly distributed to ensure maximum utilization of computing power has an important impact on the efficient training of large-scale heterogeneous clusters. However, the existing pipeline parallelization strategy targets a smaller scale, mainly concentrated in the hundreds of cards level. At a large scale, the search space for heterogeneous pipeline parallelization is larger, and the existing pipeline parallelization strategy is difficult to search for the optimal heterogeneous parallelization strategy at a scale of more than 1,000 cards; on the other hand, the existing heterogeneous pipeline parallelization strategy is mostly aimed at the heterogeneous interconnection between multiple different models of chips from a single manufacturer (such as NVIDIA). However, due to the huge differences in computing power and interconnection bandwidth of new chips from different manufacturers, the existing pipeline parallelization strategy is difficult to achieve efficient load distribution and scheduling in complex scenarios with multiple manufacturers and multiple new computing power chips.

[0004] In addition, the use of parameter servers is another method to solve the problems of heterogeneous parallel training. It mainly divides different types of heterogeneous chips into different groups. The main problem of this method is that the training is difficult to converge quickly, and there is an imbalance in computing power between different groups and the synchronization overhead between groups is large, resulting in poor execution efficiency. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies and provide an optimal parallel strategy for non-uniform heterogeneous chips and its search method and device, which can search for the optimal heterogeneous parallel strategy in heterogeneous clusters with a scale of more than 1,000 calories, and realize efficient and accurate chip load distribution and scheduling in complex new computing power chip scenarios, while achieving optimal pipeline execution efficiency.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, a method for searching an optimal parallel strategy for non-uniform heterogeneous chips is provided, and the method is implemented based on preset mapping constraints, search space constraints, data parallel dimension constraints, heterogeneous chip quantity constraints, tensor parallel dimension constraints and memory constraints, and includes the following steps: S101, according to the data parallel dimension constraints, the heterogeneous chip quantity constraints and the tensor parallel dimension constraints, obtaining a first candidate set, the first candidate set including combinations of data parallel dimensions, tensor parallel dimensions and pipeline parallel dimensions of all types of heterogeneous chips; S102, based on the first candidate set, allocating the number of model layers according to the mapping constraints, the search space constraints and the memory constraints, and determining a parallel strategy candidate set, the parallel strategy candidate set including combinations of data parallel dimensions, tensor parallel dimensions, pipeline parallel dimensions and model layer numbers of all types of heterogeneous chips; S103, calculating the single iteration time for each combination in the parallel strategy candidate set, and selecting the combination with the lowest single iteration time as the optimal parallel strategy.

[0008] As a preferred technical solution, in the optimal parallel strategy search method: the mapping constraint includes that each neural network layer of the model to be trained is mapped to a heterogeneous chip for execution; the search space constraint includes that, among all heterogeneous chips, the number of model layers of each pipeline stage corresponding to the same type of chip is the same, and the number of model layers of the pipeline stage corresponding to the heterogeneous chip is the same or different; the data parallel dimension constraint includes that the micro-batch size of each pipeline stage is 1, and thus each pipeline stage uses the same data parallel dimension; the heterogeneous chip quantity constraint includes that the total number of heterogeneous chips is The number is equal to the sum of the total number of heterogeneous chips in each pipeline stage, and the total number of heterogeneous chips in each pipeline stage is equal to the product of the tensor parallel dimension and the data parallel dimension of the corresponding pipeline stage; the tensor parallel dimension constraint includes that the tensor parallel dimension is an exponential power of 2, and the tensor parallel dimension is less than the maximum tensor parallel dimension independently specified for each heterogeneous chip; the video memory constraint includes that the chips in the pipeline are arranged from large to small according to the video memory, and the sum of the video memory occupied by the model, optimizer state and activation value of the first pipeline stage of each heterogeneous chip is less than the maximum video memory of the corresponding chip.

[0009] As an optimal technical solution, the optimal parallel strategy search method is also implemented based on preset computing card constraints, which are used to constrain the distribution of model layers; the computing card constraints include that different pipeline stages containing the same computing card use the same tensor parallel dimension.

[0010] As an optimal technical solution, the S101 specifically includes: S111, selecting a global candidate data parallel dimension set according to the data parallel dimension constraint, and then calculating the number of micro-batches; S112, for a given candidate data parallel dimension, selecting a candidate tensor parallel dimension of a single chip according to the tensor parallel dimension constraint, and then calculating the number of pipeline stages of a single chip according to the heterogeneous chip quantity constraint, and then combining the types of heterogeneous chips to obtain candidate combinations of multiple numbers of pipeline stages and tensor parallel dimensions under the given candidate data parallel dimension; S113, based on the candidate data parallel dimension set and the candidate combinations of multiple numbers of pipeline stages and tensor parallel dimensions, obtaining a candidate set of data parallel dimensions, tensor parallel dimensions and pipeline parallel dimensions of all types of heterogeneous chips under all candidate data parallel dimensions, that is, the first candidate set.

[0011] As a preferred technical solution, the number of micro-batches is the quotient of the training batch size and the data parallel dimension.

[0012] As an optimal technical solution, the S102 specifically includes: obtaining a candidate set of model layer allocation schemes based on the mapping constraints and the search space constraints; pruning the model layer allocation schemes using a heuristic method; performing water level balancing based on the time required for each heterogeneous chip to perform reverse calculations before processing one layer of the model under the current tensor parallel dimension in the candidate combinations of the number of pipeline stages and the tensor parallel dimension, and obtaining a preliminary set of parallel strategy candidates by fine-tuning until the total number of model layers is met; and removing parallel strategy candidates that do not meet the constraints from the preliminary set of parallel strategy candidates based on the video memory constraints to obtain a final set of parallel strategy candidates.

[0013] As a preferred technical solution, the video memory constraint is specifically: number of model layers × (video memory of the parameters and optimizer state of the model layer under the current data parallel dimension and tensor parallel dimension + number of warm-up micro-batches in the current pipeline stage × video memory of the activation value of processing 1 sample of the model layer under the current tensor parallel dimension) < maximum video memory of the chip.

[0014] As a preferred technical solution, in S103, the single iteration time is: the sum of the global forward and reverse calculation time, the global update optimizer parameter time, the pipeline bubble time, and the pipeline communication time.

[0015] According to a second aspect of the present invention, there is provided an optimal parallel strategy search device for non-uniform heterogeneous chips, wherein mapping constraints, search space constraints, data parallel dimension constraints, heterogeneous chip quantity constraints, tensor parallel dimension constraints, and video memory constraints are preset in the device, and the device includes a first candidate set acquisition module, a parallel strategy candidate set determination module, and an optimal parallel strategy selection module; the first candidate set acquisition module is used to acquire a first candidate set based on the data parallel dimension constraints, the heterogeneous chip quantity constraints, and the tensor parallel dimension constraints, the first candidate set including combinations of data parallel dimensions, tensor parallel dimensions, and pipeline parallel dimensions for all types of heterogeneous chips; the parallel strategy candidate set determination module is used to allocate the number of model layers based on the first candidate set according to the mapping constraints, the search space constraints, and the video memory constraints, and determine a parallel strategy candidate set, the parallel strategy candidate set including combinations of data parallel dimensions, tensor parallel dimensions, pipeline parallel dimensions, and model layer numbers for all types of heterogeneous chips; the optimal parallel strategy selection module is used to calculate a single iteration time for each combination in the parallel strategy candidate set, and select the combination with the lowest single iteration time as the optimal parallel strategy.

[0016] According to a third aspect of the present invention, an optimal parallel strategy for non-uniform heterogeneous chips is provided, and the optimal parallel strategy is obtained based on the described method, and the optimal parallel strategy is a combination of data parallel dimension, tensor parallel dimension, pipeline parallel dimension and number of model layers, wherein the data parallel dimension is an integer, the tensor parallel dimension, the pipeline parallel dimension and the number of model layers are all arrays, and the dimension of each array is limited by the type of heterogeneous chip.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. The optimal parallel strategy search method provided by the present invention is used for heterogeneous chips. This method is implemented based on multiple preset constraints. These constraints ensure that the optimal parallel strategy includes the optimal model segmentation scheme, the optimal parallel dimension, and the optimal resource mapping. This allows different types of heterogeneous chips on the pipeline to correspond to different parallel strategies, thereby searching for the optimal heterogeneous parallel strategy in heterogeneous clusters with a scale of more than 1,000 calories.

[0019] 2. The present invention fully considers the basic information of the heterogeneous cluster, such as the information of the model to be trained, the specifications of the heterogeneous chips, the cluster configuration, the network configuration, the heterogeneous interconnection configuration, etc., to ensure that the heterogeneous chips and the loads of the corresponding pipeline stages are matched with each other to achieve the optimal pipeline execution efficiency;

[0020] 3. The present invention obtains a candidate set of model layer allocation schemes based on mapping constraints and search space constraints, and uses a heuristic method to prune the model layer allocation schemes to achieve optimal model segmentation, ensuring that the segmentation method and the chip computing power match each other. At the same time, the communication volume between different model slices and the interconnection bandwidth between chips match each other, which can effectively simplify the search space and reduce the search complexity. At the same time, based on the candidate combination of the number of pipeline stages and the tensor parallel dimension, the time required for each heterogeneous chip to process the forward and reverse calculations of one layer of the model under the current tensor parallel dimension is obtained to obtain a preliminary set of parallel strategy candidates, and screen them according to the memory constraints to achieve fine-grained scheduling according to hardware characteristics, optimize the forward and reverse calculations during the training process, ensure that the computing resources of the heterogeneous chips are fully utilized, and improve the computing utilization rate of the heterogeneous chips;

[0021] 4. The present invention eliminates the need to synchronize parameters or gradients of different heterogeneous chips through a parameter server, which can reduce synchronization overhead. At the same time, through fine-grained pipeline partitioning and the selection of parallel strategies, it can effectively solve the problems of load imbalance in traditional parallel strategies and inefficient convergence in asynchronous parameter servers.

[0022] 5. In the optimal parallel strategy provided by the present invention, the data parallel dimension is an integer, the tensor parallel dimension, the pipeline parallel dimension and the number of model layers are all arrays, and the dimension of each array is limited by the type of heterogeneous chip. When this parallel strategy is used for training large-scale clusters of non-uniform heterogeneous chips, the training efficiency and load balancing of distributed training can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic flow chart of the method provided by the present invention;

[0024] Figure 2 Schematic diagram of the optimal parallel strategy in an embodiment of the present invention;

[0025] Figure 3 Schematic diagram of the structure of a search device in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In the context of the present invention, a training batch (Mini-Batch) refers to a group of samples processed by the model in a single training step, which is the most basic batch unit in deep learning; a micro-batch (Micro-Batch) is a subset of Mini-Batch, which is usually used for finer-grained control and optimization. In some distributed training frameworks, Micro-Batch is used to further decompose Mini-Batch to improve the efficiency of pipeline parallel processing. In a heterogeneous pipeline parallel system, different stages of different micro-batches (Micro-Batch) of the same training batch (Mini-Batch) can be pipelined in parallel (PipelineParallelization).

[0027] In the context of this invention, pipeline parallelism (PP) is used for simplicity; tensor parallelism (TP) is used for simplicity; and data parallelism (DP) is used for simplicity. Furthermore, DP size represents the data parallel dimension, TP size represents the tensor parallel dimension, and PP size represents the number of pipeline stages, i.e., the pipeline parallel dimension.

[0028] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0029] Example

[0030] like Figure 1 As shown, this embodiment provides an optimal parallel strategy search method for non-uniform heterogeneous chips. The method is implemented based on preset mapping constraints, search space constraints, data parallel dimension constraints, heterogeneous chip quantity constraints, tensor parallel dimension constraints, and memory constraints. It can obtain the optimal parallel strategy, including the optimal partitioning scheme, optimal parallel dimension, and optimal resource mapping, based on basic information of the heterogeneous cluster, such as model information, heterogeneous chip specifications, cluster configuration, network configuration, and heterogeneous interconnection configuration. In addition, the search method fully considers the specifications of the heterogeneous chips, such as memory size and computing information, to ensure that the loads of the heterogeneous chips and the corresponding pipeline stages are matched to each other, achieving optimal pipeline execution efficiency. It can also be efficiently implemented on a variety of new heterogeneous chips at the kilocalorie level.

[0031] The specific contents of each constraint are:

[0032] Constraint 1: Mapping Constraint

[0033] This constraint requires that each neural network layer of the model to be trained be mapped to a heterogeneous chip for execution. The relationship between the total number of layers in the model and the number of neural network layers at each stage is as follows:

[0034] The total number of neural network layers in the model (layer number) = sum(the number of neural network layers in each pipeline stage)

[0035] Constraint 2: Search Space Constraint

[0036] This constraint requires that all heterogeneous chips have the same number of model layers in each pipeline stage. However, heterogeneous chips can have different or identical pipeline stage model layers. This constraint effectively reduces the search space.

[0037] Constraint 3: Data Parallel Dimension Constraint

[0038] This constraint requires that the mini-batch size of each pipeline stage is 1, and thus each pipeline stage uses the same data parallel dimension (DP Size). The setting of this constraint mainly considers the search space size, the memory requirements of the activation value, and the proportion of vacuoles in the pipeline parallelism. From this constraint, we can get:

[0039] Data parallel dimension × number of mini-batches = training batch size

[0040] Constraint 4: Heterogeneous Chip Quantity Constraint

[0041] That is, the parallel strategy is constrained by the number of heterogeneous chips. This constraint requires that the total number of heterogeneous chips is equal to the sum of the total number of heterogeneous chips in each pipeline stage, and the total number of heterogeneous chips in each pipeline stage is equal to the product of the tensor parallel dimension and the data parallel dimension of the corresponding pipeline stage, that is:

[0042] Total number of heterogeneous chips = sum(total number of heterogeneous chips in each pipeline stage) = sum(tensor parallel dimension of each pipeline stage × data parallel dimension)

[0043] Constraint 5: Compute Card Constraint

[0044] This constraint requires that different pipeline stages using the same compute card use the same tensor parallelism dimension. Specifically, the optimal tensor parallelism dimension is the same for the same compute card; however, the optimal parallelism strategy for different stages using different compute cards may differ, and therefore different tensor parallelism dimensions can be used.

[0045] Constraint 6: Tensor Parallel Dimension Constraint

[0046] This constraint requires that the tensor parallel dimension be a power of 2 and less than the maximum tensor parallel dimension (TP_MAX) independently specified for each heterogeneous chip. Because the number of attention heads in large language models is generally a power of 2, and TP requires efficient communication, the TP should be smaller than the number of compute cards in a single machine, single NUMA, or single PCIE switch, depending on hardware topology and other factors. NUMA stands for Non-Uniform Memory Access, and PCIE refers to the high-speed serial computer expansion bus standard.

[0047] Constraint 7: Video Memory Constraint

[0048] This constraint requires that the chips in the pipeline be arranged in descending order of video memory, and that the sum of the video memory occupied by the model, optimizer state, and activation values of the first pipeline stage of each heterogeneous chip is less than the maximum available video memory of the corresponding chip. This constraint is set mainly to take into account the large number of preheating micro-batches in the early pipeline stages, which results in a large amount of activation values occupying video memory. It can be expressed as:

[0049] Number of layers × (VRAM of model layer parameters and optimizer state at current DP Size and TP Size + number of warmup micro-batches at this pipeline stage × VRAM of activation value of processing 1 sample at model layer at current TP Size) < Maximum available VRAM on chip

[0050] Exemplarily, based on the aforementioned constraints, the given parameters required for the search method provided in this embodiment include:

[0051] (1) The number of each chip;

[0052] (2) The size of the training batch for each iteration (Mini-Batch Size);

[0053] (3) the total number of layers in the model;

[0054] (4) The timetable required for the forward and reverse calculations of one layer of the model when each chip processes one sample at TP = 1 / 2 / 4 / 8 (some chips have 16 cards per machine, and 16 is also possible) and DP = 1;

[0055] (5) The timetable required for the optimizer step of the model layer when each chip is given a specific DP Size and TP Size;

[0056] (6) Communication and other loss time caused by each PP assembly line;

[0057] (7) Maximum available video memory for each chip;

[0058] (8) Table of memory usage of optimizer and parameters when DP Size and TP Size are given for a layer model. For example, if ZeRO1 is used to optimize the split optimizer state, the DP Size here can be regarded as ZeRO1Size in ZeRO1.

[0059] (9) When a layer model is given a TP Size, it processes 1 sample to occupy the activation value and occupies the video memory table.

[0060] The optimal parallel strategy search method provided in this embodiment is a phased heuristic search method. Its core concept is to first perform discrete enumeration on chip resource partitioning and pipeline configuration, then balance the distribution of model layers, and finally select the optimal configuration based on the predicted time per iteration. Based on the aforementioned constraints and given parameters, the specific implementation process of this method is as follows:

[0061] Step S101: Obtain a first candidate set based on the data parallel dimension constraint, the number of heterogeneous chips constraint, and the tensor parallel dimension constraint. The first candidate set includes combinations of data parallel dimensions, tensor parallel dimensions, and pipeline parallel dimensions for all types of heterogeneous chips. Specifically:

[0062] In step S111, based on constraint three, i.e., the data parallel dimension constraint, a global set of candidate data parallel dimensions (DP Size) is selected so that the training batch size divided by the data parallel dimension is an integer, and the number of micro batches (Micro Num) is calculated:

[0063] Micro Num=Mini-Batch Size÷DP Size.

[0064] Step S112 : Given the data parallel dimension, determine the number of pipeline stages and candidate tensor parallel dimensions for a single chip.

[0065] For each chip (assuming the number of chips i is N (i) ), requiring the resources on the chip to be divided into several pipeline stages and equipped with a tensor parallel setting. For a given candidate data parallel dimension, according to constraint six, the tensor parallel dimension constraint, the candidate tensor parallel dimension for a single chip is selected:

[0066] Candidate tensor parallel dimensions ∈ {1, 2, 4, ..., TP_MAX};

[0067] Based on constraint four, which is the constraint on the number of heterogeneous chips, the number of pipeline stages for a single chip is calculated:

[0068] The number of pipeline stages for a single chip = the total number of chips for a single chip ÷ (data parallel dimension × tensor parallel dimension);

[0069] The candidate combinations of the number of pipeline stages and tensor parallel dimensions of a single chip are denoted as: cj = [(p1,t1),...,(pi,ti),...,(pn,tn)], where n is the type of heterogeneous chip and j is the sampling number.

[0070] Combined with the types of heterogeneous chips, we can obtain candidate combinations of the number of pipeline stages and tensor parallel dimensions under a given candidate data parallel dimension di, which are denoted as:

[0071] Di=[(di,c1),...,(di,cj),...,(di,cJ)], where J is the number of samples.

[0072] In step S113, based on the candidate data parallel dimension set and the candidate combinations of the number of pipeline stages and tensor parallel dimensions, a candidate set of data parallel dimensions, tensor parallel dimensions, and pipeline parallel dimensions for all types of heterogeneous chips under all candidate data parallel dimensions is obtained, i.e., a first candidate set.

[0073] Based on steps S111 and S112, a first candidate set can be obtained, which is recorded as:

[0074] D=D1∪D2∪...∪DK, where K is the number of all candidate data parallel dimensions.

[0075] Step S102: Allocate model layers to each pipeline stage. Based on the first candidate set, allocate model layers according to mapping constraints, search space constraints, and memory constraints, and determine a parallel strategy candidate set. The parallel strategy candidate set includes all types of heterogeneous chip data parallel dimensions, tensor parallel dimensions, pipeline parallel dimensions, and model layer combinations. Specifically:

[0076] First, all model layer allocation schemes that satisfy constraint 1, i.e., the mapping constraint, and constraint 2, i.e., the search space constraint, are selected to obtain a candidate set of model layer allocation schemes.

[0077] Secondly, a heuristic method is used to prune the model layer allocation scheme;

[0078] Next, we perform water level balancing based on the time required for each heterogeneous chip in the (pi, ti) combination to perform reverse calculations before processing one layer of the model at the current ti, so that the calculation time of each pipeline stage of each chip is as close as possible. Then, we fine-tune the process until the total number of model layers is met, and thus obtain a preliminary set of parallel strategy candidates.

[0079] Finally, based on constraint seven, the video memory constraint, we remove the parallel strategy candidates that do not meet constraint seven from the preliminary set of parallel strategy candidates to obtain the final set of parallel strategy candidates. We can denote (data parallel dimension, tensor parallel dimension, pipeline parallel dimension, number of model layers) as a valid parallel strategy. The parallel strategy candidate set D is expressed as:

[0080] D=[(d1,t1,p1,l1),...,(di,ti,pi,li),...,(dn,tn1,pn,ln)],

[0081] Among them, di is an integer, ti, pi, and li are arrays, and the size of the array is the type of heterogeneous chip.

[0082] Step S103 : For each combination in the parallel strategy candidate set, that is, each candidate parallel strategy, a single iteration time is calculated, and the combination with the lowest single iteration time is selected as the optimal parallel strategy.

[0083] Specifically, the time for each prediction iteration is expressed as:

[0084] Total prediction time = prediction global forward and backward calculation time + global update optimizer parameter time + pipeline bubble time + pipeline communication time,

[0085] According to the above calculation formula, the parallel strategy (di, ti, pi, li) with the lowest total prediction time is selected.

[0086] In some other embodiments, the predicted time for each iteration may also include other existing loss time.

[0087] In the heterogeneous Kallikrein scenario, the training throughput is doubled compared to the industry-leading homogeneous distributed training framework (such as Megatron-LM) without any optimization.

[0088] In summary, the optimal parallel strategy search method for heterogeneous chips provided in this embodiment has at least the following advantages:

[0089] Heterogeneity: Compared with homogeneous distributed pre-training frameworks, this method supports newly manufactured heterogeneous chips, especially domestically produced heterogeneous chips.

[0090] Large-scale: Compared with other heterogeneous pipeline parallel strategies, this solution supports large-scale (kilocalorie level) efficient parallel strategy search and online scheduling for multiple (more than two) heterogeneous chips.

[0091] Efficiency: Compared to parameter server approaches, this solution eliminates the need to synchronize parameters or gradients across heterogeneous chips through a parameter server, reducing synchronization overhead. Furthermore, through fine-grained pipeline partitioning and parallel strategy selection, it effectively addresses the load imbalance inherent in traditional parallel strategies and the inefficient convergence of asynchronous parameter servers.

[0092] Furthermore, by comprehensively considering the fault tolerance cost and implementation cost, this method can be used in combination with the parameter server solution to further improve the training scale while solving the above-mentioned problems in the parameter server, and realize efficient distributed training of newly manufactured heterogeneous chips at a higher scale (on the order of 10,000 to 100,000 cards).

[0093] Furthermore, if Figure 2 As shown, the above-mentioned search method can be used to obtain the optimal parallel strategy for non-uniform heterogeneous chips. The optimal parallel strategy includes the optimal model segmentation scheme, the optimal parallel dimension and the optimal resource mapping. It is a combination of data parallel dimension, tensor parallel dimension, pipeline parallel dimension and number of model layers. Among them, the data parallel dimension is an integer, the tensor parallel dimension, pipeline parallel dimension and number of model layers are all arrays, and the dimension of each array is restricted by the type of heterogeneous chip.

[0094] Furthermore, if Figure 3 As shown, this embodiment also provides an optimal parallel strategy search device for non-uniform heterogeneous chips, in which mapping constraints, search space constraints, data parallel dimension constraints, heterogeneous chip quantity constraints, tensor parallel dimension constraints and video memory constraints are preset, including a first candidate set acquisition module 301, a parallel strategy candidate set determination module 302 and an optimal parallel strategy selection module 303; the first candidate set acquisition module 301 is used to obtain a first candidate set based on the data parallel dimension constraints, the heterogeneous chip quantity constraints and the tensor parallel dimension constraints, and the first candidate set includes a combination of data parallel dimensions, tensor parallel dimensions and pipeline parallel dimensions of all types of heterogeneous chips; the parallel strategy candidate set determination module 302 is used to allocate the number of model layers based on the first candidate set according to the mapping constraints, search space constraints and video memory constraints, and determine the parallel strategy candidate set, and the parallel strategy candidate set includes a combination of data parallel dimensions, tensor parallel dimensions, pipeline parallel dimensions and model layers of all types of heterogeneous chips; the optimal parallel strategy selection module 303 is used to calculate the single iteration time for each combination in the parallel strategy candidate set, and select the combination with the lowest single iteration time as the optimal parallel strategy. The specific operation process of each module is basically the same as the implementation process of the aforementioned method steps, and will not be repeated here.

[0095] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. An optimal parallel strategy search method for heterogeneous chips, characterized in that: The optimal parallel strategy search method is implemented based on preset mapping constraints, search space constraints, data parallel dimension constraints, heterogeneous chip quantity constraints, tensor parallel dimension constraints, and video memory constraints, and includes the following steps: S101, obtaining a first candidate set according to the data parallel dimension constraint, the heterogeneous chip quantity constraint, and the tensor parallel dimension constraint, where the first candidate set includes combinations of data parallel dimensions, tensor parallel dimensions, and pipeline parallel dimensions of all types of heterogeneous chips; S102: Based on the first candidate set, allocating the number of model layers according to the mapping constraint, the search space constraint, and the video memory constraint, and determining a parallel strategy candidate set, where the parallel strategy candidate set includes combinations of data parallel dimensions, tensor parallel dimensions, pipeline parallel dimensions, and number of model layers for all types of heterogeneous chips; S103 , calculating a single iteration time for each combination in the parallel strategy candidate set, and selecting a combination with the lowest single iteration time as the optimal parallel strategy.

2. The optimal parallel strategy search method for heterogeneous chips according to claim 1, characterized in that: In the optimal parallel strategy search method: The mapping constraints include that each neural network layer of the model to be trained is mapped to a heterogeneous chip for execution; The search space constraints include that, among all heterogeneous chips, the number of model layers of each pipeline stage corresponding to the same type of chips is the same, and the number of model layers of each pipeline stage corresponding to the heterogeneous chips is the same or different; The data parallel dimension constraint includes that the mini-batch size of each pipeline stage is 1, and thus each pipeline stage uses the same data parallel dimension; The constraint on the number of heterogeneous chips includes that the total number of heterogeneous chips is equal to the sum of the total number of heterogeneous chips in each pipeline stage, and the total number of heterogeneous chips in each pipeline stage is equal to the product of the tensor parallel dimension and the data parallel dimension of the corresponding pipeline stage; The tensor parallel dimension constraint includes that the tensor parallel dimension is a power of 2 and is less than the maximum tensor parallel dimension independently specified for each heterogeneous chip; The memory constraint includes arranging the chips in the pipeline from largest to smallest memory, and the sum of the memory occupied by the model, optimizer state and activation value of the first pipeline stage of each heterogeneous chip is less than the maximum memory of the corresponding chip.

3. The optimal parallel strategy search method for heterogeneous chips according to claim 1, characterized in that: The optimal parallel strategy search method is also implemented based on preset calculation card constraints, and the calculation card constraints are used to constrain the allocation of model layers; The computing card constraint includes that different pipeline stages involving the same computing card use the same tensor parallel dimension.

4. The optimal parallel strategy search method for heterogeneous chips according to claim 1, characterized in that: The S101 specifically includes: S111, selecting a global candidate data parallel dimension set according to the data parallel dimension constraint, and then calculating the number of micro-batches; S112: For a given candidate data parallel dimension, select a candidate tensor parallel dimension for a single chip type based on the tensor parallel dimension constraint, then calculate the number of pipeline stages for the single chip type based on the heterogeneous chip quantity constraint, and then, based on the type of heterogeneous chips, obtain candidate combinations of multiple pipeline stage quantities and tensor parallel dimensions for the given candidate data parallel dimension. S113, based on the candidate data parallel dimension set and the candidate combinations of the multiple pipeline stage numbers and tensor parallel dimensions, obtain a candidate set of data parallel dimensions, tensor parallel dimensions, and pipeline parallel dimensions for all types of heterogeneous chips under all candidate data parallel dimensions, that is, a first candidate set.

5. The optimal parallel strategy search method for heterogeneous chips according to claim 4, characterized in that: The number of mini-batches is the quotient of the training batch size and the data parallel dimension.

6. The optimal parallel strategy search method for heterogeneous chips according to claim 4, characterized in that: The S102 specifically includes: Obtaining a candidate set of model layer number allocation schemes according to the mapping constraint and the search space constraint; Pruning the model layer allocation scheme using a heuristic method; Based on the candidate combinations of the number of pipeline stages and tensor parallel dimensions, the time required for each heterogeneous chip to process the reverse calculation of one layer of the model under the current tensor parallel dimension is balanced, and fine-tuned until the total number of model layers is met, thereby obtaining a preliminary set of parallel strategy candidates; According to the video memory constraint, parallel strategy candidates that do not meet the constraint conditions are removed from the preliminary parallel strategy candidate set to obtain a final parallel strategy candidate set.

7. The optimal parallel strategy search method for heterogeneous chips according to claim 1, characterized in that: The video memory constraints are specifically: Number of model layers × (VRAM usage of parameters and optimizer state of a model layer in the current data and tensor parallel dimensions + number of warmup micro-batches in the current pipeline stage × VRAM usage of activation values for processing one sample of a model layer in the current tensor parallel dimensions) < chip maximum VRAM.

8. The optimal parallel strategy search method for heterogeneous chips according to claim 1, characterized in that: In S103 , the single iteration time is the sum of the global forward and reverse calculation time, the global optimizer parameter update time, the pipeline bubble time, and the pipeline communication time.

9. An optimal parallel strategy search device for heterogeneous chips, characterized in that: The device is pre-set with mapping constraints, search space constraints, data parallel dimension constraints, heterogeneous chip quantity constraints, tensor parallel dimension constraints and video memory constraints, and includes a first candidate set acquisition module, a parallel strategy candidate set determination module and an optimal parallel strategy selection module; The first candidate set acquisition module is configured to acquire a first candidate set according to the data parallel dimension constraint, the heterogeneous chip quantity constraint, and the tensor parallel dimension constraint, where the first candidate set includes combinations of data parallel dimensions, tensor parallel dimensions, and pipeline parallel dimensions of all types of heterogeneous chips; The parallel strategy candidate set determination module is configured to allocate the number of model layers based on the first candidate set according to the mapping constraint, the search space constraint, and the video memory constraint, and determine a parallel strategy candidate set, wherein the parallel strategy candidate set includes data parallel dimensions, tensor parallel dimensions, pipeline parallel dimensions, and model layer combinations of all types of heterogeneous chips; The optimal parallel strategy selection module is used to calculate the single iteration time for each combination in the parallel strategy candidate set, and select the combination with the lowest single iteration time as the optimal parallel strategy.

10. An optimal parallel strategy for heterogeneous chips, characterized in that: The optimal parallel strategy is obtained based on the method described in any one of claims 1-8, and the optimal parallel strategy is a combination of data parallel dimension, tensor parallel dimension, pipeline parallel dimension and number of model layers, wherein the data parallel dimension is an integer, the tensor parallel dimension, the pipeline parallel dimension and the number of model layers are all arrays, and the dimension of each array is limited by the type of heterogeneous chip.

Images