Model segmentation

Abstract

The present application provides a model segmentation method, a device and a storage medium. Segmentation parameters in different segmentation dimensions can be generated for a target model, and the generated segmentation parameters are filtered according to preset filtering rules, such that the remaining segmentation parameters in the segmentation dimensions are significantly reduced, and thus the number of segmentation strategies generated by means of combining the remaining segmentation parameters can also be significantly reduced. Candidate segmentation strategies that meet verification conditions are searched for among the segmentation strategies obtained by means of combination. After multi-level filtering operations, low-quality segmentation strategies can be rationally discarded, thereby making the number of retained candidate segmentation strategies more streamlined. The performance of the target model under each candidate segmentation strategy is evaluated, such that an appropriate target segmentation strategy can be finally determined to serve as the basis for distributed deployment of the target model.

Description

Technical Field of Model Partitioning

[0001] This application relates to the field of artificial intelligence technology, and in particular to model partitioning. Background Art

[0002] With the development of large language models (LLMs), the scale of large language models is constantly increasing. Therefore, the demand for distributed computing is also growing. The computing clusters used to host large language models often need to include thousands of GPUs to support the distributed training and distributed inference processes of large language models.

[0003] In the process of distributed deployment, the selection of partitioning parameters is very important and will affect the performance of the training process or the inference process. Currently, it is necessary to rely on algorithm engineers to determine the partitioning parameters based on experience, which is not only time-consuming and laborious, but also may lead to poor partitioning effects due to unreasonable partitioning parameters, thereby affecting the model performance. Summary of the Invention

[0004] Multiple aspects of this application provide a model partitioning method, device, and storage medium to optimize the model partitioning effect.

[0005] An embodiment of this application provides a model partitioning method, the method includes: filtering the partitioning parameters corresponding to the target model in each partitioning dimension according to a preset filtering rule; searching for candidate partitioning strategies that meet the verification conditions from the partitioning strategies combined from the remaining partitioning parameters in each partitioning dimension; performing performance evaluation on the target model under each candidate partitioning strategy to determine the target partitioning strategy as the basis for distributed deployment of the target model.

[0006] In some embodiments, the method may further include: obtaining the model parameters corresponding to the target model; querying the hardware information corresponding to the target cluster, where the target cluster is used to host the target model; generating, based on the model parameters and the hardware information, the partitioning parameters of the target model in each partitioning dimension.

[0007] In some embodiments, the partitioning dimension includes a tensor parallelism dimension, and the partitioning parameter in the tensor parallelism dimension adopts a tensor parallelism degree. The preset filtering rule in the tensor parallelism dimension includes: the tensor parallelism degree is 1 or an even number greater than 1 and does not exceed the total number of GPUs in the target cluster; the tensor parallelism degree can be divided evenly by the total number of attention heads corresponding to the target model; and / or the tensor parallelism degree does not exceed the number of GPUs installed on a single node in the target cluster; where the target cluster is used to host the target model.

[0008] In some embodiments, the segmentation dimension includes a pipeline parallelism dimension, the segmentation parameter under the pipeline parallelism dimension adopts the pipeline parallelism degree, and the preset filtering rules under the pipeline parallelism dimension include: the pipeline parallelism degree is 1 or an even number greater than 1 and not exceeding the total number of GPUs in the target cluster; and / or the pipeline parallelism degree can be divided evenly by the total number of model layers corresponding to the target model; wherein, the target cluster is used to host the target model.

[0009] In some embodiments, the segmentation dimension includes a data parallelism dimension, the segmentation parameter under the data parallelism dimension adopts the data parallelism degree, and the preset filtering rules under the data parallelism dimension include: the data parallelism degree does not exceed the ratio between the total memory of the target cluster and the memory occupied by the target model; and / or, the segmentation dimension includes a single batch specification dimension, the segmentation parameter under the single batch specification dimension adopts the number of samples in a single batch, and the preset filtering rules under the single batch specification dimension include: the number of samples in a single batch does not exceed the empirical value supported by the target cluster and does not exceed the number of samples supported by the remaining memory, where the remaining memory is the estimated remaining memory in the target cluster after deploying the target model; wherein, the target cluster is used to host the target model.

[0010] In some embodiments, the candidate segmentation strategies include tensor parallelism degree, pipeline parallelism degree, data parallelism degree, and / or the number of samples in a single batch. Finding candidate segmentation strategies that meet the verification conditions includes: if the product of the tensor parallelism degree, pipeline parallelism degree, and data parallelism degree included in the candidate segmentation strategy is equal to the total number of GPUs that the target cluster can provide, then it is determined that the verification conditions are met; and / or, if the data parallelism degree included in the candidate segmentation strategy and the number of samples in a single batch can be divided evenly, then it is determined that the verification conditions are met; wherein, the target cluster is used to host the target model.

[0011] In some embodiments, if the number of candidate segmentation strategies is N, then the performance of the target model is evaluated under each candidate segmentation strategy to determine the target segmentation strategy, including: under the N candidate segmentation strategies, calculating the corresponding simulation evaluation results of the target model on the target cluster respectively; according to the simulation evaluation results, screening K candidate segmentation strategies to be verified from the N candidate segmentation strategies, where K < N; deploying the target model on the target cluster according to the K candidate segmentation strategies to be verified in a distributed manner to generate real machine evaluation results; according to the real machine evaluation results, determining the target segmentation strategy from the K candidate segmentation strategies to be verified.

[0012] In some embodiments, under the N candidate partitioning strategies, the simulated evaluation results corresponding to the target model on the target cluster are calculated respectively, including: under the first candidate partitioning strategy, the performance overhead corresponding to the backbone network in the target model on the target cluster is simulated and evaluated as the simulated evaluation result corresponding to the target model under the first candidate partitioning strategy; wherein, the performance overhead includes computing overhead, communication overhead and / or memory overhead, and the first candidate partitioning strategy is any one of the candidate partitioning strategies.

[0013] In some embodiments, the performance overhead corresponding to the backbone network in the target model on the target cluster is simulated and evaluated, including: using the unit sample received by a single model layer carried on the target process as the simulation evaluation unit, the unit performance overhead corresponding to the target process is simulated and evaluated, wherein the target process is any process used to carry the target model on the simulated target cluster; according to each partitioning parameter in the first candidate partitioning strategy, the simulation evaluation coefficient corresponding to the target process is determined; the product of the simulation evaluation coefficient and the unit performance overhead is calculated as the performance overhead corresponding to the target process; according to the performance overhead calculated for each process used to carry the backbone network, the performance overhead corresponding to the backbone network is estimated.

[0014] In some embodiments, the unit performance overhead corresponding to the target process is simulated and evaluated, including: for the inference stage, the performance overhead corresponding to the single-round end-to-end propagation process of the unit sample is simulated and evaluated as the unit performance overhead corresponding to the target process; for the training stage, the total performance overhead corresponding to the unit sample in the multi-round end-to-end propagation process required for training and other training-required links is simulated and evaluated as the unit performance overhead corresponding to the target process.

[0015] In some embodiments, according to the performance overhead calculated for each process used to carry the backbone network, the performance overhead corresponding to the backbone network is estimated, including: according to the process time consumed included in the performance overhead corresponding to each process, representative processes are selected; the performance overhead corresponding to the representative processes is used as the performance overhead corresponding to the backbone network.

[0016] In some embodiments, according to the simulation evaluation results, K candidate splitting strategies are screened from the N candidate splitting strategies, including: according to the memory overhead in the simulation evaluation results, screening out the candidate splitting strategies estimated to have memory overflow problems from the N candidate splitting strategies; based on the simulation evaluation results, calculating the evaluation indices corresponding to the remaining candidate splitting strategies respectively; sorting the remaining candidate splitting strategies according to the evaluation indices to screen out the K candidate splitting strategies to be verified; wherein, the evaluation index includes the sampling rate.

[0017] In some embodiments, on the target cluster, the target model is distributedly deployed according to the K candidate splitting strategies to generate in-machine evaluation results, including: calling a model execution framework, and respectively inputting the splitting parameters in the K candidate splitting strategies into the model execution framework; using the model execution framework to respectively distribute and deploy the target model to the target cluster according to the K candidate splitting strategies; performing in-machine evaluation operations respectively under the K candidate splitting strategies to generate in-machine evaluation results.

[0018] In some embodiments, the method may further include: extracting, from the called model execution framework, the configuration file generated when the target model is distributedly deployed to the target cluster according to the target splitting strategy; saving the configuration file.

[0019] An embodiment of the present application further provides a computing device, including a memory, a processor, and a communication component; the memory is used to store one or more computer instructions; the processor is coupled with the memory and the communication component and is used to execute one or more computer instructions to execute the foregoing model splitting method.

[0020] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which when executed by one or more processors, cause one or more processors to execute the foregoing model splitting method.

[0021] In the embodiment of the present application, before the target model is distributedly deployed, splitting parameters in different splitting dimensions can be generated for the target model, and the generated splitting parameters are filtered according to a preset filtering rule, which can effectively reduce the number of splitting parameters in each splitting dimension, and the number of splitting strategies generated by the combination of the remaining splitting parameters in each splitting dimension will also be significantly reduced. Moreover, the splitting strategies that meet the verification conditions are also searched from the combined splitting strategies. Candidate segmentation strategies. In this way, through multi-level filtering operations, low-quality segmentation strategies can be reasonably discarded, so that the number of candidate segmentation strategies retained is more streamlined. On this basis, the performance of the target model can be evaluated under each candidate segmentation strategy to finally determine the appropriate target segmentation strategy. Accordingly, the appropriate target segmentation strategy can be efficiently determined for the target model and used as the basis for distributed deployment of the target model, without relying on manual experience, thus effectively improving the model segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0023] FIG. 1 is a schematic flowchart of a model segmentation method provided by an exemplary embodiment of the present application;

[0024] FIG. 2 is a schematic logical diagram of a model segmentation method provided by an exemplary embodiment of the present application;

[0025] FIG. 3 is a schematic flowchart of another model segmentation method provided by an exemplary embodiment of the present application;

[0026] FIG. 4 is a schematic diagram of an exemplary model segmentation principle provided by an exemplary embodiment of the present application;

[0027] FIG. 5 is a schematic structural diagram of a computing device provided by another exemplary embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0029] Before starting to elaborate on the technical solutions provided by the embodiments of the present application, several technical concepts involved in the present application are briefly explained as follows.

[0030] Large model: It can refer to a deep learning model with a large number of model parameters, usually containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than one quadrillion model parameters. A large model can also be referred to as a foundation model. (Foundation Model), through pre-training of large models with large-scale unlabeled corpora, a pre-trained model with over hundreds of millions of parameters is produced. Such a model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models (LLMs), multi-modal pre-training models, etc. It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned with a small number of samples so that the large model can be applied to different tasks. For example, large models can be widely applied in fields such as natural language processing (NLP for short), computer vision, and speech processing. Specifically, they can be applied to tasks in the field of computer vision such as visual question answering (VQA for short), image captioning (IC for short), and image generation. They can also be widely applied to tasks in the field of natural language processing such as text-based sentiment classification, text summary generation, and machine translation. Description (Image Caption, IC for short), tasks in the field of computer vision such as image generation, and can also be widely applied to tasks in the field of natural language processing such as text-based sentiment classification, text summary generation, and machine translation.

[0031] Distributed deployment: It can be understood that after the model is split, each split part of the model is carried by a computing cluster. A typical cluster for distributed deployment is a GPU cluster. After completing the distributed deployment, when receiving an inference or training task, the task can be split into parallel subtasks to process the task in a distributed manner, thereby improving the task processing efficiency.

[0032] Model execution framework: It can be understood as a tool for deploying and executing models. The model execution framework usually includes a distributed initialization link. In this link, the model execution framework will split the model according to the input split parameters and deploy it to the parallel processes in the cluster in a distributed manner. There are many model execution frameworks in the current field, such as Megatron, etc. No more product examples are given here.

[0033] As introduced in the background technology, currently, professional personnel such as algorithm engineers need to input split parameters into the model execution framework according to experience, and the model execution framework will split and deploy the model accordingly in a distributed manner. However, this model splitting method relying on manual experience results in unsatisfactory model splitting effects, leading to poor model performance and high model training / inference costs.

[0034] To this end, an embodiment of the present application provides an innovative model slicing method, which proposes to automatically determine a suitable slicing strategy for the model before distributed deployment of the model, as the basis for distributed deployment of the model execution framework, without relying on manual experience any more.

[0035] The following will describe in detail the technical solutions provided by each embodiment of the present application with reference to the accompanying drawings.

[0036] FIG. 1 is a schematic flowchart of a model slicing method provided by an exemplary embodiment of the present application. This method can be executed by a data processing device, which can be implemented as software, hardware, or a combination of software and hardware, and the data processing device can be integrated in a computing device. Referring to FIG. 1, this method may include steps 100 to 102.

[0037] Step 100: Filter the slicing parameters corresponding to the target model in each slicing dimension according to a preset filtering rule.

[0038] Step 101: Search for candidate slicing strategies that meet the verification conditions from the slicing strategies combined from the remaining slicing parameters in each slicing dimension.

[0039] Step 102: Perform performance evaluation on the target model under each candidate slicing strategy to determine the target slicing strategy as the basis for distributed deployment of the target model.

[0040] The target model in this embodiment may be the large model mentioned above, or of course other models that need to be distributedly deployed. This embodiment does not limit the attributes such as the scale and function of the target model.

[0041] Referring to FIG. 1, in step 100, slicing parameters corresponding to the target model can be generated in each slicing dimension. This embodiment supports slicing the target model in a hybrid parallel manner. Therefore, the slicing dimensions in this embodiment can be multiple. The slicing dimensions in this embodiment may include but are not limited to tensor parallel dimension, pipeline parallel dimension, data parallel dimension, and sample dimension in a single batch, etc. In this embodiment, the slicing parameters adopted in different slicing dimensions can be: tensor parallel degree in the tensor parallel dimension, pipeline parallel degree in the pipeline parallel dimension, data parallel degree in the data parallel dimension, and the number of samples in a single batch in the sample dimension of a single batch. The following will explain the above several exemplary slicing dimensions as follows.

[0042] Tensor Parallel (TP) can be understood as splitting a single model layer in a model and deploying it to different processes respectively. Among them, the processes run on the nodes in the cluster.

[0043] Pipeline Parallel (PP) can be understood as splitting multiple model layers in a model into multiple groups of model layers and deploying different groups to different processes.

[0044] Data Parallel (DP) can be understood as multiple processes carrying the same part of the model, processing the same data in parallel.

[0045] The single batch size dimension (Batch Size, BS) can be understood as the number of samples included in a single batch input to the model. When BS is different, the computational load borne by the parallel processes will be different.

[0046] It should be understood that the above several splitting dimensions are only exemplary, and this embodiment is not limited thereto.

[0047] In this embodiment, multiple implementation manners can be adopted to generate the splitting parameters corresponding to each splitting dimension for the target model. Figure 2 is a logical schematic diagram of a model splitting method provided by an exemplary embodiment of the present application. Referring to Figure 2, in a preferred implementation manner: the model parameters corresponding to the target model can be obtained; the hardware information corresponding to the target cluster, which is used to host the target model, can be queried; based on the model parameters and the hardware information, the splitting parameters of the target model in each splitting dimension can be generated.

[0048] Among them, in this preferred implementation manner, the user can input the model parameters of the target model. The model parameters in this embodiment may include but are not limited to the number of model layers, latent variable dimension, number of multi-heads, dimension of each head, vocabulary size, data type, sequence length, and maximum BS, etc. It should be understood that these model parameters are only exemplary, and this embodiment is not limited thereto, and no exhaustive listing is made here.

[0049] Among them, referring to Figure 2, in this preferred implementation, the hardware information corresponding to the target cluster can be obtained through hardware awareness. The dimensions of hardware awareness can include but are not limited to storage space, communication connection, computing power, etc. In practical applications, a management program is usually deployed in the target cluster. Based on this, the management program in the target cluster can be used to automatically and real-time monitor the hardware information in the target cluster. In this preferred implementation, the hardware information of the target cluster can be queried from the management program in the target cluster. By obtaining the hardware information corresponding to the target cluster through hardware awareness, the accuracy of the hardware information can be effectively improved, and thus the relevant partitioning parameters can be calculated more accurately for the target model. The hardware information in this embodiment can include but are not limited to hardware type, topology structure, network communication bandwidth, number of computing nodes, number of GPU cards on the nodes, video card memory, computing power, communication bandwidth, etc. It should be understood that these hardware information are only exemplary, and this embodiment is not limited thereto, and no exhaustive list is made here. The communication bandwidth, number of computing nodes, number of GPU cards on the nodes, video card memory, computing power, communication bandwidth, etc. It should be understood that these hardware information are only exemplary, and this embodiment is not limited thereto, and no exhaustive list is made here.

[0050] On this basis, according to the model parameters corresponding to the target model and the hardware information corresponding to the target cluster, the partitioning parameters of the target model can be generated in each partitioning dimension. The inventors found during the research process that under the two aspects of model parameters and hardware information, the number of partitioning parameters that can be calculated in each partitioning dimension is limited, rather than infinite. Therefore, in this embodiment, all the partitioning parameters supported by the target cluster in each partitioning dimension can be calculated for the target model by enumeration. In this embodiment, the calculation rules during the enumeration of partitioning parameters according to model parameters and hardware information are not limited, and the calculation rules can be designed as needed. For example, if there are 8 GPUs in the target cluster and the number of layers of the target model is 16, then the tensor parallelism in the tensor parallel dimension can enumerate values such as 1, 2, 3, 4, 5, 6, 7, and 8. Of course, this is only an exemplary calculation rule, and this embodiment is not limited thereto.

[0051] It can be seen that the partitioning parameters of the target model generated in each partitioning dimension according to the model parameters corresponding to the target model and the hardware information corresponding to the target cluster are very comprehensive.

[0052] In this embodiment, in step 100, it is further proposed that the segmentation parameters corresponding to the target model under each segmentation dimension can be filtered respectively according to the preset filtering rules. Here, in this embodiment, by setting reasonable preset filtering rules, parameter pruning can be performed on the target model under each segmentation dimension to filter out unreasonable segmentation parameters under each segmentation dimension.

[0053] In this embodiment, preset filtering rules may be configured for different segmentation dimensions. When segmentation parameters are filtered according to the corresponding preset filtering rules under a single segmentation dimension, the segmentation parameters under other segmentation dimensions may be set to 1 by default to avoid over-filtering. Several exemplary preset filtering rules are provided below for the exemplary segmentation dimensions described above.

[0054] The preset filtering rules under the tensor parallel dimension TP may include but are not limited to: the tensor parallelism is 1 or an even number greater than 1 and does not exceed the total number of GPUs in the target cluster; the tensor parallelism is divisible by the total number of attention heads corresponding to the target model; or, the tensor parallelism does not exceed the number of GPUs installed on a single node in the target cluster.

[0055] For example, if the total number of GPUs available in the target cluster is 8, the values of the tensor parallelism that are screened out may include 1, 2, 4, 6, 8, and the other tensor parallelisms enumerated will be screened out. If the total number of attention heads in the target model (included in the model parameters) is 32, since 6 cannot meet the divisibility rule, the previously screened 6 will be screened out, and the values of the remaining tensor parallelism may include 1, 2, 4, 8. If the number of GPUs installed on a single node in the target cluster is 4, then since 8 is greater than 4, 8 will be screened out. This aspect of screening can effectively ensure that the tensor parallel process will not cross nodes, so as to avoid the need to transmit data across nodes between tensor parallel processes. So far, the values of the remaining tensor parallelism may include 1, 2, 4.

[0056] The preset filtering rules under the pipeline parallel dimension PP include but are not limited to: the pipeline parallelism is 1 or an even number greater than 1 and does not exceed the total number of GPUs in the target cluster; or, the pipeline parallelism can be divided by the total number of model layers corresponding to the target model.

[0057] For example, if the total number of model layers of the target model is 32, the values of the pipelining parallelism selected can include 1, 2, 4, 8, and other enumerated pipelining parallelisms will be filtered out. Through the filtering in this aspect, the number of model layers carried on each process of the pipelining parallelism can be made consistent, so that the workloads between processes are more balanced.

[0058] The preset filtering rules under the data parallelism dimension DP can include: the data parallelism does not exceed the ratio between the total memory of the target cluster and the memory occupied by the target model.

[0059] For example, if the memory occupied by the target model itself is 26GB, and there are 8 GPUs provided in the target cluster, and the memory capacity of each GPU is 14GB, then the total memory of the target cluster is 14 * 8 = 112GB O And 112 / 26 > 4. Therefore, the values of the data parallelism that can be selected can include 1, 2, 3, 4. That is, at most 4 copies of the target model are deployed in the target cluster.

[0060] The preset filtering rules under the single batch size dimension BS can include: the number of samples in a single batch does not exceed the empirical value supported by the target cluster and does not exceed the number of samples supported by the remaining memory, where the remaining memory is the memory remaining in the target cluster after estimating the deployment of the target model.

[0061] For example, if the model of the GPU in the target cluster is A10, according to experience, it can be determined that the BS experience it supports is 512. And if the model of the GPU in the target cluster is A20, and its computing power is 2 times that of A10, then it can be determined that the BS experience value it supports is 1024; Continuing with the total memory of 112GB of the target cluster in the previous example, since DP is set to 1 during filtering, the remaining memory in the target cluster after deploying the target model will be 112 - 26 = 84GB. And through estimation, the memory occupied by a single sample in a single batch is 32MB, then it can be calculated that 84GB / 34MB = 2752. Then the minimum value can be taken from 2752 and 1024, which is 1024, and the BS values exceeding 1024 enumerated will be filtered out. In addition, during the enumeration process, an enumeration step greater than 1 can be used, for example, an enumeration step of 16 can be used, and some BS values can also be filtered out.

[0062] It should be understood that the above preset filtering rules are only exemplary, and this embodiment is not limited thereto, and no more examples are given here.

[0063] It can be understood that in step 100, in this embodiment, by designing reasonable filtering rules, a layer of filtering is boldly performed on the segmentation parameters under each segmentation dimension. After the filtering operation, the number of remaining segmentation parameters of the target model under each segmentation dimension is already relatively small. This not only reduces the number of alternative segmentation parameters, but also ensures the accuracy of the remaining segmentation parameters.

[0064] On this basis, continuing to refer to FIG. 1, in step 101, a segmentation strategy can be generated by combining the remaining segmentation parameters under each segmentation dimension. As mentioned above, this embodiment supports model segmentation in a hybrid parallel manner. Therefore, the combination here can be understood as taking one segmentation parameter from each segmentation dimension to combine a segmentation strategy. For example, if 4 segmentation dimensions are used, then a segmentation strategy combined in step 101 can be understood as a quadruple, taking 1 segmentation parameter from each of the 4 segmentation dimensions to obtain the 4 elements in the quadruple. Based on the above-mentioned layer of filtering operation in step 100, the number of remaining segmentation parameters under each segmentation dimension is already relatively small. Therefore, in step 101, an enumeration method can be used to enumerate all the segmentation strategies that can be combined among the remaining segmentation parameters under each segmentation dimension.

[0065] In this embodiment, in step 101, it is further proposed to find the part of the segmentation strategies that meet the verification conditions from the combined segmentation strategies as candidate segmentation strategies.

[0066] In one exemplary search scheme: if the product of the tensor parallelism, pipeline parallelism, and data parallelism included in the candidate segmentation strategy is equal to the total number of GPUs that the target cluster can provide, then it is determined that the verification condition is met. This can ensure that all GPUs in the target cluster can be fully utilized.

[0067] And / or, if the data parallelism included in the candidate segmentation strategy and the number of samples in a single batch are divisible, then it is determined that the verification condition is met. This can ensure that the number of samples in a single batch can be evenly distributed to multiple parallel model replicas, making the workload between processes more balanced.

[0068] In this exemplary search scheme, two aspects of exemplary verification conditions are provided. It should be noted that the verification conditions in this embodiment are not limited to this.

[0069] In this exemplary search scheme, two aspects of exemplary verification conditions are provided. It should be noted that the verification conditions in this embodiment are not limited to this.

[0070] It can be understood that in this embodiment, in step 101, an additional layer of filtering is performed. After the filtering operation, various unreasonable segmentation strategies can be screened out, and the number of remaining candidate segmentation strategies will be more refined. This not only reduces the number of candidate segmentation strategies, but also ensures the accuracy of the candidate segmentation strategies.

[0071] After that, in this embodiment, in step 102, the performance of the target model can be evaluated under each candidate segmentation strategy to determine the target segmentation strategy, which serves as the basis for distributed deployment of the target model.

[0072] Among them, performance evaluation can be understood as evaluating the model performance that the target model can achieve when it is distributedly deployed on the target cluster according to a certain candidate segmentation strategy. In practical applications, the target segmentation strategy can be determined by selecting the optimal one from the performance evaluation results of the candidate segmentation strategies. In this way, when the target model is distributedly deployed according to the target segmentation strategy subsequently, the model performance of the target model can be effectively guaranteed.

[0073] In summary, in this embodiment, before the target model is distributedly deployed, segmentation parameters under different segmentation dimensions can be generated for the target model, and the generated segmentation parameters can be filtered according to preset filtering rules. This effectively reduces the number of segmentation parameters under each segmentation dimension, and the number of segmentation strategies generated by the combination of the remaining segmentation parameters under each segmentation dimension will also be significantly reduced. Moreover, candidate segmentation strategies that meet the verification conditions are searched from the combined segmentation strategies. In this way, through multi-level filtering operations, low-quality segmentation strategies can be reasonably discarded, thereby making the number of remaining candidate segmentation strategies more refined. On this basis, the performance of the target model can be evaluated under each candidate segmentation strategy to finally determine the appropriate target segmentation strategy. Accordingly, an appropriate target segmentation strategy for the target model can be efficiently determined and used as the basis for distributed deployment of the target model, without relying on manual experience, thus effectively improving the model segmentation effect.

[0074] FIG. 3 is a schematic flowchart of another model segmentation method provided by an exemplary embodiment of the present application. Referring to FIG. 3, the method may include steps 300 to 305.

[0075] Step 300: Filter the segmentation parameters corresponding to the target model under each segmentation dimension according to preset filtering rules.

[0076] Step 301: From the segmentation strategies combined from the remaining segmentation parameters under each segmentation dimension, search for N candidate segmentation strategies that meet the verification conditions.

[0077] Step 302: Under the N candidate segmentation strategies, calculate the corresponding simulation evaluation results of the target model on the target cluster respectively.

[0078] Step 303: According to the simulation evaluation results, screen K candidate segmentation strategies from the N candidate segmentation strategies, where K < N, and K and N are positive integers.

[0079] Step 304: On the target cluster, deploy the target model distributively according to the K candidate segmentation strategies to generate actual machine evaluation results.

[0080] Step 305: According to the actual machine evaluation results, determine the target segmentation strategy from the K candidate segmentation strategies.

[0081] Among them, Steps 300 - 301 can refer to the relevant descriptions in the foregoing embodiments and will not be repeated here. In this embodiment, an optional implementation manner for determining the target segmentation strategy from the candidate segmentation strategies can be provided based on Steps 302 - 305.

[0082] Referring to FIG. 3, in Step 302, under the N candidate segmentation strategies, the performance of the target model can be simulated and evaluated on the target cluster respectively to obtain the corresponding simulation evaluation results. As mentioned above, in this embodiment, after the two-layer filtering performed in Steps 300 and 301, the number of candidate segmentation strategies has been relatively small. Therefore, in Step 302, only the filtered candidate segmentation strategies need to be subjected to performance simulation evaluation, and the computational efficiency of the performance simulation evaluation operation in Step 302 will be greatly improved.

[0083] It should be understood that the simulation evaluation results generated by the performance model evaluation operation performed in Step 302 are theoretical values. Although their accuracy cannot reach complete accuracy, they can basically reflect the model segmentation effects that can be achieved by different candidate segmentation strategies. Therefore, in this embodiment, the simulation evaluation results are used to perform another layer of filtering on the candidate segmentation strategies.

[0084] In addition, the specific implementation manner of the performance simulation evaluation operation is not limited in this embodiment and will be described by examples later. It will not be elaborated here for the time being.

[0085] However, in this embodiment, the final target segmentation strategy is not directly determined based on the simulation evaluation results. Referring to FIG. 2, instead, a mechanism of combining theoretical search with real machine verification optimization is innovatively proposed.

[0086] According to this mechanism, in step 303, K candidate segmentation strategies to be verified can be screened from N candidate segmentation strategies according to the simulation evaluation results.

[0087] In an exemplary screening scheme: First, candidate segmentation strategies estimated to have memory overflow problems can be screened out from N candidate segmentation strategies according to the memory overhead in the simulation evaluation results. The inventor found during the research process that the simulation evaluation results usually include memory overhead. Therefore, this information is fully utilized in this exemplary screening scheme to estimate the candidate segmentation strategies with memory overflow problems and screen them out. In practical applications, the memory overhead records the memory amount required for a single batch of input, while the hardware information contains the actual memory amount. If the memory amount required for a single batch of input exceeds the remaining memory amount after the target model is simulated and deployed on the target cluster, it can be determined that there is a memory overflow problem. This is actually a layer of filtering. After this layer of filtering operation, some defective candidate segmentation strategies can be screened out.

[0088] Then, based on the simulation evaluation results, the evaluation index corresponding to each of the remaining candidate segmentation strategies can be calculated. That is to say, only the evaluation index corresponding to the candidate segmentation strategies remaining in the previous link. Among them, the evaluation index can include the sampling rate, etc. The sampling rate can be understood as the amount of data that the target model can process per unit time. Exemplarily, an exemplary scheme for calculating the sampling rate can be: determining the number of samples in a single batch from a certain remaining candidate segmentation strategy; determining the time required for the target model to perform calculations according to this number of samples from the simulation evaluation results corresponding to this candidate segmentation strategy; taking the ratio between the number of samples and the time as the sampling rate. Of course, in addition to the sampling rate, other types of evaluation indexes can also be used, and it is not limited thereto.

[0089] After that, the remaining candidate segmentation strategies can be sorted according to the evaluation index to screen out K candidate segmentation strategies to be verified. Referring to FIG. 2, in practical applications, the K candidate segmentation strategies with the highest evaluation index can be selected as the candidate segmentation strategies to be verified.

[0090] After this layer of filtering, the number of remaining segmentation strategies has reached K, which is already very small.

[0091] Continuing to refer to FIG. 3, it is proposed in step 304 that the target model can be distributedly deployed on the target cluster according to K to-be-verified splitting strategies to generate in-machine evaluation results. It should be understood that in step 304, in-machine verification tests are performed according to each to-be-verified splitting strategy. An exemplary in-machine evaluation scheme can be: calling a model execution framework and respectively inputting the splitting parameters in the K to-be-verified splitting strategies into the model execution framework; using the model execution framework to respectively distribute and deploy the target model to the target cluster according to the K to-be-verified splitting strategies; performing in-machine evaluation operations respectively under the K to-be-verified splitting strategies to generate in-machine evaluation results.

[0092] In practical applications, a batch processing program can be used to call a specific model execution framework (such as Megatron or TensorRT etc.) to complete the aforementioned distributed initialization process of the target model on the target cluster according to each to-be-verified splitting strategy and perform in-machine evaluation. Here, the model execution framework called can preferably be the model execution framework that may be used for online deployment of the target model in the future. Moreover, the inventors found in the research process that in the process of distributed initialization by the model execution framework, a large amount of internal processing logic needs to be performed based on the input splitting parameters to complete the distributed deployment, and a configuration file will be generated in this process. In this embodiment, it is further proposed that the configuration file can be extracted from the called model execution framework; and after determining the target splitting strategy, the configuration file corresponding to the target splitting strategy is retained. In this way, when the target model is to be deployed online in the future, the configuration file can be directly input into the corresponding model execution framework, which can eliminate a large amount of repetitive work in the distributed initialization process of the model execution framework and improve the efficiency of distributed initialization.

[0093] In this embodiment, when performing the physical machine evaluation operation, the evaluation mechanism adopted may include, but is not limited to: evaluating the processing time corresponding to a fixed BS, or evaluating the BS that can be supported by a fixed processing time. Additionally, in this embodiment, the same evaluation metrics as those used in the aforementioned simulation evaluation can be adopted during the physical machine evaluation. For example, the evaluation metric used during the physical machine evaluation can also be the sampling rate. Continuing with this exemplary evaluation metric, evaluating the processing time corresponding to a fixed BS can be understood as: detecting the processing time consumed by the target model when processing the fixed BS, calculating the ratio between the fixed BS and the consumed processing time, and taking this as the sampling rate of the target model. And evaluating the BS that can be supported by a fixed processing time can be understood as: detecting the actual number of BS processed by the target model within the fixed processing time, calculating the ratio between this number of BS and the fixed processing time, and taking this as the sampling rate of the target model. The essence of both mechanisms is to calculate the workload that the target model can process per unit time as the physical machine evaluation result of the target model.

[0094] In this way, in step 304, the physical machine evaluation results corresponding to the target model under the K candidate segmentation strategies can be generated.

[0095] On this basis, referring to Figure 2, in step 305, the target segmentation strategy can be determined from the K candidate segmentation strategies according to the physical machine evaluation results. In practical applications, the physical machine evaluation results generated by the K candidate segmentation strategies can be sorted, and the target segmentation strategy can be optimized from them.

[0096] In summary, in this embodiment, a mechanism of combining theoretical search with physical machine verification optimization is adopted to optimize the final target segmentation strategy from the N candidate segmentation strategies, rather than relying solely on theoretical calculations; moreover, during the physical machine verification optimization process, the model execution framework expected to be used during the subsequent online deployment of the target model can also be adopted, and the configuration file precipitated when the model execution framework performs physical machine deployment on the target model can be saved, so that it can be directly used by the model execution framework during the subsequent online deployment, thus saving a lot of repetitive work and improving the online deployment efficiency of the target model.

[0097] In the above or following embodiments, the specific implementation scheme of the simulation evaluation is not limited. The following provides a preferred implementation scheme. For ease of description, the first candidate segmentation strategy among the aforementioned N candidate segmentation strategies will be used as an example to expand and explain this preferred implementation scheme. It should be understood that the same simulation evaluation logic can be adopted under other candidate segmentation strategies to obtain the corresponding simulation evaluation results.

[0098] In this preferred implementation: Under the first candidate segmentation strategy, the backbone network in the target model can be simulated and evaluated for the corresponding performance overhead on the target cluster, which is used as the simulation evaluation result of the target model under the first candidate segmentation strategy. During the research process, the inventor found that the model may contain multiple parts, such as: the vectorized embedding part, the backbone network, and the prediction head projection head, etc. Among them, the performance overhead is most significant in the backbone network. Therefore, in this preferred implementation, it is proposed that during the simulation evaluation, the performance overhead corresponding to the backbone network in the target model can be focused on and used as the simulation evaluation result of the target model under the first candidate segmentation strategy.

[0099] On this basis, an exemplary simulation evaluation scheme for the performance overhead of the backbone network in the target model can be: taking the unit sample received by a single model layer carried on the target process as the simulation evaluation unit, and simulating and evaluating the unit performance overhead corresponding to the target process, where the target process is any process used to carry the target model on the simulated target cluster; determining the simulation evaluation coefficient corresponding to the target process according to each segmentation parameter in the first candidate segmentation strategy; calculating the product of the simulation evaluation coefficient and the unit performance overhead as the performance overhead corresponding to the target process; estimating the performance overhead corresponding to the backbone network according to the performance overhead calculated for each process used to carry the backbone network.

[0100] First, in this exemplary simulation evaluation scheme, the target model is simulated and segmented according to the first candidate segmentation strategy to generate multiple model parts, and the parallel processes used to carry the target model on the target cluster are also simulated. It should be understood that the number of parallel processes simulated on the target cluster can be determined based on the hardware information corresponding to the target cluster.

[0101] FIG. 4 is a schematic diagram of an exemplary model segmentation principle provided by an exemplary embodiment of the present application. Referring to FIG. 4, for example, if there are two nodes in the target cluster and 16 GPUs can be provided, one process can be started on each GPU (gO - gl5 in FIG. 4). And as mentioned above, in any candidate segmentation strategy in this embodiment, the total number of model parts to be segmented is adapted to the number of parallel processes that can be provided in the target cluster. Therefore, when performing simulated segmentation on the target model, 16 model parts can be simulated and segmented. Of course, under different candidate segmentation strategies, the parallel relationships among the 16 segmented model parts are different. The candidate segmentation strategy corresponding to FIG. 4 is

TP = 2, PP = 4, DP = 2, BS = 2

[0102] In this way, from the overall perspective of the target model, it will receive a batch of inputs. In FIG. 4, BS = 2, and the single - batch input will be distributed according to multiple segmentation dimensions to each process. From the perspective of the process, in the case of a single - batch input, the workloads that different processes need to handle may be different. Therefore, the performance overheads required by different processes may also be different. In this exemplary simulation evaluation scheme, it is proposed to use the unit samples received by a single model layer carried on the target process as the simulation evaluation unit to simulate and evaluate the unit performance overhead required by the target process. It should be understood that there are no specific rules limited in the process of simulating and evaluating the unit performance overhead. Since the unit performance overhead is affected by model parameters and hardware parameters, there must be a natural influence law between the model parameters, hardware parameters and the unit performance overhead. In practical applications, this influence law can be extracted through machine learning or manual summary as the rule in the process of simulating and evaluating the unit performance overhead. No exhaustive list of this influence law is made here. kind of influence law.

[0103] Then, in this exemplary simulation evaluation scheme, the simulation evaluation coefficient corresponding to the target process is also determined according to each segmentation parameter in the first candidate segmentation strategy. Among them, the simulation evaluation coefficient is used to represent the multiple corresponding to the unit performance overhead required on the target process. The simulation evaluation coefficient can generally be equal to (the number of model layers carried on the target process * BS) / DP. Specifically, it can be understood that: after a batch of samples (for example, BS = 2) are input into the target model, they are allocated once based on DP (for example, DP = 2). In this way, the BS allocated to each model layer becomes 1. If there are two model layers loaded on the target process, the BS allocated to the target process becomes 2 again. In this way, the simulation evaluation coefficient on the target process can be calculated as 2.

[0104] After that, in this exemplary simulation evaluation scheme, the product of the simulation evaluation coefficient and the unit performance overhead is calculated as the performance overhead corresponding to the target process. Continuing with the above example, the performance overhead corresponding to the target process is 2 (simulation evaluation coefficient) * unit performance overhead.

[0105] Finally, in this exemplary simulation evaluation scheme, the performance overhead corresponding to the backbone network can be estimated according to the performance overhead calculated for each process used to carry the backbone network. The inventor found during the research process that since the model will ultimately be distributed for calculation, in this link, the representative process can be selected according to the process time consumed included in the performance overhead corresponding to each process; the performance overhead corresponding to the representative process is used as the performance overhead corresponding to the backbone network. In practical applications, the process with the largest time consumption can be selected as the representative process.

[0106] In this way, through this exemplary simulation evaluation scheme, the performance overhead corresponding to the backbone network in the target model on the target cluster can be simulated and evaluated.

[0107] In addition, the inventor found during the research process that the performance overhead required in the model inference stage and the model training stage is different, which is mainly due to the different number of rounds of the end-to-end propagation process required in the inference stage and the training stage. In the inference stage, usually a single round of the end-to-end propagation process is performed to obtain the model output result. However, in the training stage, because there are two propagation directions, forward propagation and backward propagation, it may take multiple rounds of the end-to-end propagation process to complete one training. Therefore, in this exemplary simulation evaluation scheme, when simulating and evaluating the unit performance overhead corresponding to the target process, the inference stage and the training stage can be simulated and evaluated separately.

[0108] Specifically: for the inference stage, simulate and evaluate the performance overhead corresponding to the end-to-end propagation process of a single unit sample, which is used as the unit performance overhead corresponding to the target process; for the training stage, simulate and evaluate the total performance overhead corresponding to the multi-round end-to-end propagation process required for training and other links required for training of a single unit sample, which is used as the unit performance overhead corresponding to the target process. In this way, based on the unit performance overheads respectively simulated and evaluated for the inference stage and the training stage, the performance overheads required for the target model in the inference stage and the training stage can be calculated.

[0109] In the process of screening K candidate splitting strategies described in the foregoing embodiments, the performance overheads simulated and evaluated for the inference stage and the training stage can both be used as the simulation evaluation results, so as to more reasonably screen out K candidate splitting strategies.

[0110] Furthermore, various other events that may occur in the end-to-end propagation process can also be simulated and calculated, and the time consumption that these events may cause can be simulated and added to the foregoing unit performance overhead. Among them, these events that may occur can include but are not limited to pipeline bubble events and the preparation work that the subsequent process needs to do for the previous process, etc. The pipeline bubble event can be understood as that the processes in pipeline parallelism may wait for each other due to inconsistent work rhythms before and after. In this way, after simulating the time consumption caused by these other events, the accuracy of the simulated and evaluated unit performance overhead can be further improved, and then the accuracy of the performance overhead simulated and evaluated for the backbone network in the target model under the first candidate splitting strategy can be improved.

[0111] It should be understood that in this embodiment, other exemplary solutions can also be used to simulate and evaluate the performance overhead corresponding to the backbone network in the target model. For example, the BS value, the memory amount required, the communication amount required, and the communication overhead that each process can support under the condition of fixed time consumption can be simulated and evaluated. It is not limited to the above exemplary simulation and evaluation solutions. No more detailed examples are given here.

[0112] In this embodiment, the performance overheads concerned in the simulation and evaluation process can include but are not limited to computing overhead, communication overhead, and / or memory overhead, etc. Among them, the computing overhead and the communication overhead can be used as the basis for calculating the evaluation index in the foregoing embodiments, while the memory overhead can be used as the basis for evaluating the memory overflow problem in the foregoing.

[0113] In summary, in this embodiment, by simulating and evaluating the performance overhead required by the backbone network in the target model under different candidate segmentation strategies, the simulation evaluation results for the target model can be generated more accurately and efficiently. Thus, K candidate segmentation strategies to be verified can be more reasonably selected from N candidate segmentation strategies, and further, the final selected target segmentation strategy can be made more reasonable.

[0114] It should be noted that in some of the processes described in the above embodiments and the accompanying drawings, there are multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The operation numbers such as 301 and 302 are only used to distinguish different operations, and the numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different candidate segmentation strategies, etc., do not represent a sequence, and do not limit that "first" and "second" are different types.

[0115] FIG. 5 is a schematic structural diagram of a computing device provided in another exemplary embodiment of the present application. As shown in FIG. 5, the computing device includes: a memory 50, a processor 51, and a communication component 52.

[0116] The processor 51 is coupled to the memory 50 and the communication component 52 and is configured to execute a computer program in the memory 50 to: filter the segmentation parameters corresponding to the target model in each segmentation dimension according to a preset filtering rule; search for candidate segmentation strategies that meet the verification conditions from the segmentation strategies combined from the remaining segmentation parameters in each segmentation dimension; perform performance evaluation on the target model under each candidate segmentation strategy to determine a target segmentation strategy as a basis for distributed deployment of the target model. In an alternative embodiment, the processor 51 may further be configured to: obtain model parameters corresponding to the target model; query hardware information corresponding to a target cluster for hosting the target model; and generate segmentation parameters for the target model in each segmentation dimension based on the model parameters and the hardware information.

[0117]

[0118] ​In an alternative embodiment, the splitting dimension includes a tensor parallelism dimension. The splitting parameter under the tensor parallelism dimension uses the tensor parallelism degree. The preset filtering rules under the tensor parallelism dimension include: the tensor parallelism degree is 1 or an even number greater than 1 and does not exceed the total number of GPUs in the target cluster; the tensor parallelism degree can be divided evenly by the total number of attention heads corresponding to the target model; and / or the tensor parallelism degree does not exceed the number of GPUs installed on a single node in the target cluster; wherein the target cluster is used to host the target model.

[0119] In an alternative embodiment, the splitting dimension includes a pipeline parallelism dimension. The splitting parameter under the pipeline parallelism dimension uses the pipeline parallelism degree. The preset filtering rules under the pipeline parallelism dimension include: the pipeline parallelism degree is 1 or an even number greater than 1 and does not exceed the total number of GPUs in the target cluster; and / or the pipeline parallelism degree can be divided evenly by the total number of model layers corresponding to the target model; wherein the target cluster is used to host the target model.

[0120] In an alternative embodiment, the splitting dimension includes a data parallelism dimension. The splitting parameter under the data parallelism dimension uses the data parallelism degree. The preset filtering rules under the data parallelism dimension include: the data parallelism degree does not exceed the ratio between the total memory of the target cluster and the memory occupied by the target model; and / or, the splitting dimension includes a single batch size dimension. The splitting parameter under the single batch size dimension uses the number of samples in a single batch. The preset filtering rules under the single batch size dimension include: the number of samples in a single batch does not exceed the empirical value supported by the target cluster and does not exceed the number of samples supported by the remaining memory. The remaining memory is the estimated memory remaining in the target cluster after deploying the target model; wherein the target cluster is used to host the target model.

[0121] In an alternative embodiment, the candidate splitting strategy includes the tensor parallelism degree, the pipeline parallelism degree, the data parallelism degree, and / or the number of samples in a single batch. When the processor 51 searches for a candidate splitting strategy that meets the verification conditions, it can be used to: if the product of the tensor parallelism degree, the pipeline parallelism degree, and the data parallelism degree included in the candidate splitting strategy is equal to the total number of GPUs that the target cluster can provide, then it is determined that the verification conditions are met; and / or, if the data parallelism degree included in the candidate splitting strategy and the number of samples in a single batch can be divided evenly, then it is determined that the verification conditions are met; wherein the target cluster is used to host the target model.

[0122] In an alternative embodiment, if the number of candidate segmentation strategies is N, when the processor 51 evaluates the performance of the target model under each candidate segmentation strategy to determine the target segmentation strategy, it can be used to: calculate the corresponding simulated evaluation results of the target model on the target cluster under the N candidate segmentation strategies; screen K candidate segmentation strategies to be verified from the N candidate segmentation strategies according to the simulated evaluation results, where K < N; perform distributed deployment of the target model on the target cluster according to the K candidate segmentation strategies to be verified to generate actual machine evaluation results; determine the target segmentation strategy from the K candidate segmentation strategies to be verified according to the actual machine evaluation results.

[0123] In an alternative embodiment, when the processor 51 calculates the corresponding simulated evaluation results of the target model on the target cluster under the N candidate segmentation strategies, it can be used to: under the first candidate segmentation strategy, simulate and evaluate the performance overhead corresponding to the backbone network in the target model on the target cluster as the simulated evaluation result corresponding to the target model under the first candidate segmentation strategy; where the performance overhead includes computing overhead, communication overhead, and / or memory overhead, and the first candidate segmentation strategy is any candidate segmentation strategy.

[0124] In an alternative embodiment, when the processor 51 simulates and evaluates the performance overhead corresponding to the backbone network in the target model on the target cluster, it can be used to: use the unit sample received by a single model layer carried on the target process as the simulation evaluation unit to simulate and evaluate the unit performance overhead corresponding to the target process, where the target process is any process used to carry the target model on the simulated target cluster; determine the simulation evaluation coefficient corresponding to the target process according to each segmentation parameter in the first candidate segmentation strategy; calculate the product of the simulation evaluation coefficient and the unit performance overhead as the performance overhead corresponding to the target process; estimate the performance overhead corresponding to the backbone network according to the performance overhead calculated for each process used to carry the backbone network.

[0125] In an alternative embodiment, when the processor 51 simulates and evaluates the unit performance overhead corresponding to the target process, it can be used to: for the inference stage, simulate and evaluate the performance overhead corresponding to the single-round end-to-end propagation process of the unit sample as the unit performance overhead corresponding to the target process; for the training stage, simulate and evaluate the total performance overhead corresponding to the multi-round end-to-end propagation process and other training required links of the unit sample during training as the unit performance overhead corresponding to the target process.

[0126] In an alternative embodiment, when estimating the performance overhead corresponding to the backbone network according to the performance overhead calculated for each process used to carry the backbone network, the processor 51 can be used to: screen representative processes according to the process time consumed included in the performance overhead corresponding to each process; use the performance overhead corresponding to the representative processes as the performance overhead corresponding to the backbone network.

[0127] In an alternative embodiment, when screening K candidate partitioning strategies to be verified from the N candidate partitioning strategies according to the simulation evaluation results, the processor 51 can be used to: screen out the candidate partitioning strategies estimated to have memory overflow problems from the N candidate partitioning strategies according to the memory overhead in the simulation evaluation results; calculate the evaluation index corresponding to each of the remaining candidate partitioning strategies based on the simulation evaluation results; sort the remaining candidate partitioning strategies according to the evaluation index to screen out the K candidate partitioning strategies to be verified; wherein, the evaluation index includes a sampling rate.

[0128] In an alternative embodiment, when the processor 51 performs distributed deployment of the target model on the target cluster according to the K candidate partitioning strategies to be verified to generate an in-machine evaluation result, it can be used to: call a model execution framework and input the partitioning parameters in the K candidate partitioning strategies to the model execution framework respectively; use the model execution framework to distribute and deploy the target model to the target cluster according to the K candidate partitioning strategies respectively; perform in-machine evaluation operations respectively under the K candidate partitioning strategies to generate an in-machine evaluation result.

[0129] In an alternative embodiment, the processor 51 can also be used to: extract the configuration file generated when the target model is distributed and deployed to the target cluster according to the target partitioning strategy from the called model execution framework; save the configuration file.

[0130] Further, as shown in FIG. 5, the computing device further includes: a power supply component 53 and other components. Only some components are schematically shown in FIG. 5, which does not mean that the computing device only includes the components shown in FIG. 5.

[0131] It should be noted that for the technical details in the embodiments of the computing device above, reference can be made to the relevant descriptions in the foregoing method embodiments. To save space, they will not be repeated here, but this should not cause loss of the protection scope of this application.

[0132] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed, it can implement the steps performed in the above method embodiments.

[0133] The memory in FIG. 5 above is used to store a computer program and can be configured to store various other data to support operations on a computing platform. Examples of such data include instructions for any application or method for operating on a computing platform, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0134] The communication component in FIG. 5 above is configured to facilitate communication between the device where the communication component is located and other devices in a wired or wireless manner. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0135] The power supply component in FIG. 5 above provides power for various components of the device where the power supply component is located. The power supply component can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device where the power supply component is located.

[0136] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0137] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce means for implementing the functions specified in one or more flows of the flowchart and / or one or more blocks of the block diagram.

[0138] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in one or more flows of the flowchart and / or one or more blocks of the block diagram.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowchart and / or one or more blocks of the block diagram.

[0140] It should also be noted that the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including a....." does not exclude the presence of additional identical elements in the process, method, article, or device including the said element.

[0141] It should be noted that the user information involved in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0142] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

Claims 1. A model segmentation method, comprising: Filter the segmentation parameters corresponding to the target model in each segmentation dimension according to the preset filtering rules respectively; Search for candidate segmentation strategies that meet the verification conditions from the segmentation strategies combined from the remaining segmentation parameters in each segmentation dimension; perform performance evaluation on the target model under each candidate segmentation strategy to determine the target segmentation strategy as the basis for distributed deployment of the target model.

2. The method according to claim 1, further comprising: Obtain the model parameters corresponding to the target model; Query the hardware information corresponding to the target cluster, where the target cluster is used to host the target model; Generate segmentation parameters for the target model in each segmentation dimension based on the model parameters and the hardware information.

3. The method according to claim 1 or 2, wherein, The segmentation dimension includes the tensor parallelism dimension, and the segmentation parameter under the tensor parallelism dimension uses the tensor parallelism degree. The preset filtering rules under the tensor parallelism dimension include: the tensor parallelism degree is 1 or an even number greater than 1 and does not exceed the total number of GPUs in the target cluster; the tensor parallelism degree can be divisible by the total number of attention heads corresponding to the target model; and / or the tensor parallelism degree does not exceed the number of GPUs installed on a single node in the target cluster; wherein, the target cluster is used to host the target model.

4. The method according to claim 1 or 2, wherein The segmentation dimension includes the pipeline parallelism dimension, and the segmentation parameter under the pipeline parallelism dimension uses the pipeline parallelism degree. The preset filtering rules under the pipeline parallelism dimension include: the pipeline parallelism degree is 1 or an even number greater than 1 and does not exceed the total number of GPUs in the target cluster; and / or the pipeline parallelism degree can be divisible by the total number of model layers corresponding to the target model; wherein, the target cluster is used to host the target model.

5. The method according to claim 1 or 2, wherein The segmentation dimension includes the data parallelism dimension, and the segmentation parameter under the data parallelism dimension uses the data parallelism degree. The preset filtering rules under the data parallelism dimension include: the data parallelism degree does not exceed the ratio between the total memory of the target cluster and the memory occupied by the target model; and / or, the segmentation dimension includes the single batch specification dimension, and the segmentation parameter under the single batch specification dimension uses the number of samples in a single batch. The preset filtering rules under the single batch specification dimension include: the number of samples in a single batch does not exceed the empirical value supported by the target cluster and does not exceed the number of samples supported by the remaining memory, where the remaining memory is the estimated remaining memory after deploying the target model in the target cluster; wherein, the target cluster is used to host the target model.

6. The method according to claim 1 or 2, wherein The candidate partitioning strategy includes tensor parallelism, pipeline parallelism, data parallelism, and / or the number of samples in a single batch. Search for candidate partitioning strategies that meet the verification conditions, including: if the product of the tensor parallelism, pipeline parallelism, and data parallelism included in the candidate partitioning strategy is equal to the total number of GPUs that the target cluster can provide, then it is determined that the verification conditions are met; and / or, if the data parallelism included in the candidate partitioning strategy and the number of samples in a single batch are divisible, then it is determined that the verification conditions are met; where the target cluster is used to host the target model.

7. The method according to claim 2, wherein If the number of candidate partitioning strategies is N, then perform performance evaluation on the target model under each candidate partitioning strategy to determine the target partitioning strategy, including: under the N candidate partitioning strategies, respectively calculate the corresponding simulation evaluation results of the target model on the target cluster; according to the simulation evaluation results, screen K candidate partitioning strategies to be verified from the N candidate partitioning strategies, where K < N; perform distributed deployment of the target model on the target cluster according to the K candidate partitioning strategies to be verified to generate real machine evaluation results; according to the real machine evaluation results, determine the target partitioning strategy from the K candidate partitioning strategies to be verified.

8. The method according to claim 7, wherein Under the N candidate partitioning strategies, respectively calculate the corresponding simulation evaluation results of the target model on the target cluster, including: under the first candidate partitioning strategy, simulate and evaluate the performance overhead of the backbone network in the target model on the target cluster as the corresponding simulation evaluation result of the target model under the first candidate partitioning strategy; where the performance overhead includes computing overhead, communication overhead, and / or memory overhead, and the first candidate partitioning strategy is any candidate partitioning strategy.

9. The method according to claim 8, wherein, Simulate and evaluate the performance overhead of the backbone network in the target model on the target cluster, including: using the unit sample received by a single model layer hosted on the target process as the simulation evaluation unit, simulate and evaluate the unit performance overhead corresponding to the target process, where the target process is any process simulated on the target cluster for hosting the target model; according to each partitioning parameter in the first candidate partitioning strategy, determine the simulation evaluation coefficient corresponding to the target process; calculate the product of the simulation evaluation coefficient and the unit performance overhead as the performance overhead corresponding to the target process; Estimate the performance overhead corresponding to the backbone network according to the performance overhead calculated for each process used to host the backbone network.

10. The method according to claim 9, wherein Simulate and evaluate the unit performance overhead corresponding to the target process, including: for the inference phase, simulate and evaluate the performance overhead corresponding to the end-to-end propagation process of a single round of a single unit sample as the unit performance overhead corresponding to the target process; for the training phase, simulate and evaluate the total performance overhead corresponding to the multi-round end-to-end propagation process and other training-required links for a single unit sample as the unit performance overhead corresponding to the target process.

11. The method according to claim 9, wherein, Estimate the performance overhead corresponding to the backbone network according to the performance overhead calculated for each process used to carry the backbone network, including: screening representative processes according to the process time consumed in the performance overhead corresponding to each process; using the performance overhead corresponding to the representative processes as the performance overhead corresponding to the backbone network.

12. The method according to claim 7, wherein According to the simulation and evaluation results, screen K candidate segmentation strategies from the N candidate segmentation strategies, including: screening out candidate segmentation strategies estimated to have memory overflow problems from the N candidate segmentation strategies according to the memory overhead in the simulation and evaluation results; calculating the respective evaluation indices corresponding to the remaining candidate segmentation strategies based on the simulation and evaluation results; sorting the remaining candidate segmentation strategies according to the evaluation indices to screen out the K candidate segmentation strategies; where the evaluation index includes the sampling rate.

13. The method according to claim 7, wherein, Perform distributed deployment of the target model on the target cluster according to the K candidate segmentation strategies to generate real machine evaluation results, including: invoking the model execution framework and inputting the segmentation parameters in the K candidate segmentation strategies into the model execution framework respectively; using the model execution framework to distribute and deploy the target model to the target cluster according to the K candidate segmentation strategies respectively; performing real machine evaluation operations respectively under the K candidate segmentation strategies to generate real machine evaluation results.

14. The method according to claim 13 further comprises: Extract the configuration file generated when the target model is distributed and deployed to the target cluster according to the target segmentation strategy from the invoked model execution framework; Save the configuration file.

15. A computing device, including a memory, a processor, and a communication component; the memory is used to store one or more computer instructions; The processor is coupled to the memory and the communication component and is used to execute the one or more computer instructions to execute the model segmentation method according to any one of claims 1-14.

16. A computer-readable storage medium storing computer instructions, wherein When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the model segmentation method according to any one of claims 1-14.

Images