Model deployment method and electronic device

Abstract

The present disclosure provides a model deployment method and an electronic device. The model deployment method comprises: obtaining parameters of N computing devices of a model to be deployed, and determining parameters of a model performance evaluation function on the basis of the parameters of the N computing devices; determining at least one parallel grouping scheme, wherein in an ith parallel grouping scheme, the computing devices are divided into Mi groups, each group comprises N / Mi computing devices, the computing devices in each group are configured to carry out model parallel computing, Mi is the number of stages of pipeline parallelism, each group of computing devices is configured to perform pipeline parallel computing, N / Mi is a parameter splitting number of model parallelism, i≥1, and Mi>1; on the basis of each parallel grouping scheme, determining a model performance evaluation value corresponding to each parallel grouping scheme; and determining a parallel grouping scheme corresponding to a maximum model performance evaluation value as an optimal parallel grouping scheme, and deploying a data processing model on the basis of the optimal parallel grouping scheme. Embodiments of the present disclosure can optimize the performance of the data processing model.

Description

Model deployment method and electronic device Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a model deployment method, device, and electronic device. Background Art

[0002] With the explosive popularity of ChatGPT, large models have rapidly captured the attention of the contemporary internet industry, and demand for large language models has grown across various industries. Demand for text generation has also skyrocketed. However, text generation itself consumes significant computing power. How to maximize the deployment and acceleration of inference on large pre-trained language models within widely used device environments is a hot research topic.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a model deployment method, apparatus and electronic device for improving the performance of a data processing model by improving the structure of the data processing model and the deployment method on different computing devices.

[0005] According to a first aspect of an embodiment of the present disclosure, a model deployment method is provided, comprising: obtaining parameters of N computing devices of a model to be deployed, determining parameters of a model performance evaluation function according to the parameters of the N computing devices, wherein the model performance evaluation function is the product of a computing time proportion function, a pipeline parallel balancing rate function, and a throughput efficiency function; determining at least one parallel grouping scheme, wherein in the i-th parallel grouping scheme, the computing devices are divided into Mi groups, each group containing N / Mi computing devices, and the computing devices within each group are configured to perform model parallel computing, wherein Mi is the number of pipeline parallel stages, and the number of computing devices in each group is The time is set to perform pipeline parallel computing, N / Mi is the parameter segmentation number of model parallelism, i≥1, Mi>1; based on each of the parallel grouping schemes, after determining the independent variables of the computing time proportion function, determining the independent variables of the pipeline parallel balancing rate function, and determining the independent variables of the throughput efficiency function, the value of the model performance evaluation function corresponding to each of the parallel grouping schemes is determined, that is, the model performance evaluation value corresponding to each of the parallel grouping schemes; the parallel grouping scheme corresponding to the largest model performance evaluation value is determined as the optimal parallel grouping scheme, and the data processing model is deployed according to the optimal parallel grouping scheme.

[0006] In an exemplary embodiment of the present disclosure, deploying a data processing model according to the optimal parallel grouping scheme includes: if there is no P2P PCIe communication between the N computing devices, setting the N / Mi computing devices within each group as computing devices that do not share bandwidth with each other.

[0007] In an exemplary embodiment of the present disclosure, the method further includes setting the number of running instances of the data processing model to be equal to the number of stages in the parallel grouping scheme plus one.

[0008] In an exemplary embodiment of the present disclosure, the independent variables of the computing time proportion function are the number of pipeline parallel stages and the parameter splitting number of model parallelism under the parallel grouping scheme, the independent variable of the pipeline parallel balancing rate function is the number of pipeline parallel stages under the parallel grouping scheme, and the independent variable of the throughput efficiency function is the available memory of the key-value cache under a parallel grouping scheme under the parallel grouping scheme.

[0009] In an exemplary embodiment of the present disclosure, determining the value of the independent variable of the throughput efficiency function includes: determining the available memory of the key-value cache under a parallel grouping scheme under the parallel grouping scheme based on the number of running instances; and using the available memory of the key-value cache under a parallel grouping scheme under the parallel grouping scheme as the value of the independent variable of the throughput efficiency function.

[0010] In an exemplary embodiment of the present disclosure, it also includes: setting up communication between the computing devices through a one-dimensional byte tensor, and the one-dimensional byte tensor is used to record the data name of the data to be transmitted and the compressed package of the data to be transmitted.

[0011] In an exemplary embodiment of the present disclosure, the setting of communication between the computing devices through one-dimensional byte tensors includes: setting a transceiver module on each of the computing devices, and the transceiver module is used to compress and package the data to be transmitted to form the one-dimensional byte tensor when communication with other computing devices is required, and send the one-dimensional byte tensor to other computing devices, or decompress the one-dimensional byte tensor received from other computing devices to obtain data.

[0012] In an exemplary embodiment of the present disclosure, sending the one-dimensional byte tensor to other computing devices includes: in a send-receive type communication, sending the length data of the one-dimensional byte tensor to the other computing device; after sending the length data, sending the one-dimensional byte tensor to the other computing device.

[0013] In an exemplary embodiment of the present disclosure, the compressed package of the data to be transmitted includes at least one of a first byte tensor, a second byte tensor, and a third byte tensor. The first byte tensor is before the second byte tensor and / or the third byte tensor and is spliced ​​with the second byte tensor or the third byte tensor. The second byte tensor is before the third byte tensor and is spliced ​​with the third byte tensor. The first byte tensor is used to record data parameters of the data to be transmitted, and the data parameters include at least one of a variable name, shape, data type, storage device, and data length. The second byte tensor is used to record the data to be transmitted stored on the CPU, and the third byte tensor is used to record the data to be transmitted stored on the GPU.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, comprising: N computing devices; a memory; and a processor coupled to the memory, the processor being configured to execute the method described in any of the above embodiments based on instructions stored in the memory to deploy a data processing model on the N computing devices.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the model deployment method as described in any one of the above items is implemented.

[0016] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the steps of the method described in any of the above embodiments when executed by a processor.

[0017] The disclosed embodiment divides the data processing model into multiple groups according to the parameters of the computing devices, realizes pipeline parallelism between the groups, realizes model parallelism within the group, and deploys the data processing model on N computing devices according to the optimal parallel grouping strategy. This can reduce the communication time when the large model is running, while increasing the storage capacity of the key-value cache, effectively improving the computing speed and throughput of the large model, and optimizing the performance of the large model.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0020] FIG1 is a flowchart of a model deployment method in an exemplary embodiment of the present disclosure.

[0021] FIG2 is a schematic diagram of an optimal parallel grouping solution in one embodiment of the present disclosure.

[0022] FIG3 is a schematic diagram of communication after computing devices are grouped in one embodiment of the present disclosure.

[0023] FIG4 is a schematic diagram of a model task scheduling system in one embodiment of the present disclosure.

[0024] 5A and 5B are schematic diagrams of an improved model running process in one embodiment of the present disclosure.

[0025] FIG6 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0027] The accompanying drawings are merely schematic illustrations of the present disclosure. Identical reference numerals in the drawings denote identical or similar components, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0028] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0029] FIG1 is a flowchart of a model deployment method in an exemplary embodiment of the present disclosure.

[0030] 1 , a model deployment method 100 may include:

[0031] Step S1, obtaining parameters of N computing devices of the model to be deployed, and determining parameters of a model performance evaluation function based on the parameters of the N computing devices, wherein the model performance evaluation function is the product of a computing time proportion function, a pipeline parallel balancing rate function, and a throughput efficiency function;

[0032] Step S2: determining at least one parallel grouping scheme. In the i-th parallel grouping scheme, the computing devices are divided into Mi groups, each group contains N / Mi computing devices, and the computing devices within each group are configured to perform model parallel computing. Mi is the number of pipeline parallel stages. The computing devices in each group are configured to perform pipeline parallel computing. N / Mi is the parameter splitting ratio of the model parallelism, i≥1, Mi>1.

[0033] Step S3, based on each of the parallel grouping schemes, after determining the independent variables of the calculation time proportion function, determining the independent variables of the pipeline parallel balancing rate function, and determining the independent variables of the throughput efficiency function, determine the value of the model performance evaluation function corresponding to each of the parallel grouping schemes, that is, the model performance evaluation value corresponding to each of the parallel grouping schemes;

[0034] Step S4: determining the parallel grouping scheme corresponding to the largest model performance evaluation value as the optimal parallel grouping scheme, and deploying the data processing model according to the optimal parallel grouping scheme.

[0035] The disclosed embodiment divides the data processing model into multiple groups according to the parameters of the computing devices, realizes pipeline parallelism between the groups, realizes model parallelism within the group, and deploys the data processing model on N computing devices according to the optimal parallel grouping strategy. This can reduce the communication time when the large model is running, while increasing the storage capacity of the key-value cache, effectively improving the computing speed and throughput of the large model, and optimizing the performance of the large model.

[0036] Below, each step of the model deployment method 100 is described in detail.

[0037] In step S1, the parameters of N computing devices of the model to be deployed are obtained, and the parameters of the model performance evaluation function are determined according to the parameters of the N computing devices. The model performance evaluation function is the product of the computing time proportion function, the pipeline parallel balancing rate function, and the throughput efficiency function.

[0038] In the embodiments of the present disclosure, a computing device refers to a hardware device used to accelerate computing, which can process large-scale data and perform complex computing tasks, such as a GPU (Graphics Processing Unit), which can also be called a graphics card, or simply a card.

[0039] With the advancement of artificial intelligence technology, deep learning models easily exceed trillions of parameters. The traditional single-machine, single-GPU model is no longer sufficient for training these extremely large models. Therefore, it is necessary to run large models distributed across multiple GPUs on a single machine, or even multiple machines with multiple GPUs. Specifically, when running data processing models such as large-scale pre-trained language models (referred to as large models), the model is divided into multiple parts and deployed on multiple GPUs. This allows for the combined computing power of multiple computing devices to meet the computing power requirements.

[0040] When using multiple computing devices to run large models, it is generally necessary to consider partitioning the model based on the matching of hardware resources and data / model scale to improve the efficiency of running large models on multiple computing device systems and optimize the processing speed, memory usage and other performance of the large model.

[0041] Pipelining is a parallel computing method that improves the efficiency of instruction execution for large models. Specifically, it divides the instruction execution process into multiple stages and allows multiple instructions to execute different stages simultaneously within the same clock cycle, thereby achieving overlapping execution. Dividing data into multiple blocks does not mean dividing the model into multiple blocks. It is important to note that common parallelism modes include data parallelism, model parallelism, and pipelining. Data parallelism necessarily requires data partitioning. Model parallelism can or cannot partition data. Both pipelining and model parallelism involve model partitioning.

[0042] The current practice in the industry is to only split the model (stage) without splitting the data or with a small splitting intensity.

[0043] In traditional data processing, each instruction must be executed completely before the next one can begin, resulting in low processor utilization. Pipelining, on the other hand, divides the instruction processing process into several stages, each of which is executed by a dedicated hardware unit. Different instructions are connected in a pipeline fashion between stages, forming an instruction pipeline. Once an instruction enters the pipeline, it is processed sequentially through each stage and continues to advance until it is finally executed. This allows multiple instructions to be in different stages simultaneously, achieving instruction-level parallelism.

[0044] The advantage of pipeline parallelism is that it improves instruction execution throughput and efficiency. Because multiple instructions can be in different stages simultaneously, the model can execute different stages of multiple instructions within the same clock cycle, thereby improving computing device utilization and performance. However, a disadvantage is that since different computing devices execute different stages of different instructions separately, this can lead to imbalance and waste of computing power. For example, after stage i completes processing instruction a, it needs to send the processing result of instruction a to stage i+1 for further processing. However, stage i+1 is currently processing instruction b (assuming instruction b is more complex or the processing of stage i+1 is more complex than other stages). In this case, stage i can only wait for stage i+1 to become idle and stop accepting pending data for instruction c from stage i-1. As a result, all stages must wait for the slowest stage to complete processing before data can flow down, resulting in waiting time and wasted computing power. This situation is called pipeline balance loss.

[0045] Due to cost balancing limitations, conventional pipeline parallelism configurations available on the market can only run one or two pipelines simultaneously (i.e., one or two stages can execute instructions simultaneously). Regardless of the number of stages configured (each stage generally corresponds to a card or computing device), all stages must remain idle, waiting for the arrival of these one or two pipelines. This results in many computing cards running idle, wasting 87.5% of computing power and maximizing cost balancing. Therefore, pipeline parallelism is generally not considered an option for increasing computing power.

[0046] Model parallelism is another technique for processing large-scale deep learning models. It aims to distribute the model's computational and storage load across multiple devices or computing nodes to improve performance and scalability. In traditional deep learning models, all computation is performed on a single device or node, which can limit the model's scale and performance. Model parallelism, on the other hand, performs computations in parallel by partitioning the model parameters into multiple parts and assigning each part to a different device or node. This allows each device or node to handle only a portion of the model's computational load, improving the model's computational efficiency and throughput.

[0047] Model parallelism can be achieved in many ways, one of the most common ways is model sharding parallelism. In model sharding parallelism, different layers or modules of the model are divided into multiple parts and assigned to different devices or nodes for calculation. Each device or node is responsible for processing the calculation of the corresponding part of the model, and then passes the results to other devices or nodes for the next step of calculation. In this way, the computational load of the model can be distributed among multiple devices or nodes, thereby speeding up the calculation process of the overall model. The model parallelism of the embodiments of the present disclosure specifically refers to model sharding parallelism.

[0048] In model parallelism, the communication overhead between models is high, resulting in significant room for improvement in model processing speed. However, in pipeline parallelism, only small amounts of data are transmitted between stages, resulting in lower communication overhead.

[0049] Therefore, in the embodiment of the present disclosure, the model is divided into multiple stages, which are respectively set on multiple computing device groups. Each computing device group contains at least one computing device, and data processing is performed in parallel using pipelines between multiple stages, that is, after one stage is processed, the output data is transferred to the next stage for processing; within each stage, the parameters of different parts of the model are set on different computing devices, and data processing is performed using a model parallel strategy, that is, the calculation of the current stage is completed by multiple computing devices.

[0050] By setting up intra-group model parallelism, different computing devices in the same group can process computing tasks at different stages at the same time, reducing the balance loss caused by the imbalance of each stage of pipeline parallelism, and improving computing efficiency and throughput. At the same time, intra-group model parallelism enables the computing devices in each group to jointly process the computing load of part of the model, making full use of the computing power of multiple devices and further improving computing efficiency; by setting up inter-group pipeline parallelism, the communication overhead between computing devices in each group can be reduced, and the overall operation efficiency of the model can be improved.

[0051] Furthermore, each group of computing devices can be viewed as a relatively independent computing unit, making it easier to manage and schedule computing resources, improving system maintainability and scalability. If additional computing resources are needed, simply add more computing devices to each group, or simply add more groups and continue using pipeline parallelism and model parallelism without changing the architecture. This architectural design facilitates system expansion and upgrades, allowing for flexible resource allocation and adjustment based on demand.

[0052] However, not every grouping method can best achieve the above effect. Precise calculations are required to reasonably set the grouping method based on the actual situation of the computing device to fully realize the optimal effect of the above architecture.

[0053] In the embodiment of the present disclosure, the throughput of the model corresponding to each grouping method is determined by a model performance evaluation function, thereby determining which grouping method has the best performance.

[0054] In the embodiment of the present disclosure, the model performance evaluation function throughput is the product of the computing time proportion function c, the pipeline parallel balance rate function r, and the throughput efficiency function f, which is:

[0055] throughout=c(Mi,N / Mi)r(Mi)f(M2i)(1)

[0056] The independent variables of the time proportion function c are the number of pipeline parallel stages Mi and the parameter split ratio N / Mi of model parallelism; the independent variable of the pipeline parallel balance rate function r is the number of pipeline parallel stages Mi; and the independent variable of the throughput efficiency function f is the available memory of the key-value cache (KV cache) under a parallel grouping scheme.

[0057] Below, each function is introduced in detail.

[0058] Because pipeline parallelism suffers from imbalance, the ratio of the average computing power of each stage to the computing power of the slowest stage is defined as the pipeline balance rate (r). It is foreseeable that r is negatively correlated with the number of stages; the more stages, the greater the likelihood of imbalance. Therefore, a pipeline balance rate function r(Mi) is defined, whose value is negatively correlated with the number of pipeline stages Mi. This function measures the pipeline balance rate under the i-th parallel grouping method, that is, the degree of balance loss.

[0059] Since computing devices not only perform computation but also communication tasks, to measure the communication overhead under each parallel grouping scheme, we define the computation time fraction c, which describes the proportion of the total time spent running the computation kernel on a computing device. This roughly measures the ratio of the time available for computing tasks on the computing device, meaning that the sum of c and the communication time fraction is approximately 1. It is foreseeable that c is related to the parallel grouping scheme, the amount of communication between computing devices, and the communication speed. For example, in pipeline parallelism, when executing point-to-point send-recv communication, the communication volume is relatively low, accounting for approximately 34.1% of the communication time, with c=0.659. In model parallelism, when executing global reduce communication (allreduce), the communication volume is relatively high, accounting for approximately 77.8% of the communication time, with c=0.222. A global reduce operation reduces data (such as summing or averaging) across multiple nodes or processors and then broadcasts the result back to each node, thereby achieving global data synchronization and aggregation.

[0060] Therefore, c is strongly correlated with the parallel grouping method. We define the computation time share function c(Mi, N / Mi) to measure the computation time share (which reflects the communication overhead) for the i-th parallel grouping method. It's understandable that the computation time share function is also strongly correlated with the performance of the computing device. Higher-performance computing devices lead to faster communication speeds, lower communication overhead, and a higher computation time share.

[0061] Finally, the processing speed and throughput efficiency of the model are strongly related to the available memory of the KV cache (key-value cache).

[0062] The memory used by large models for inference tasks consists primarily of two parts: memory m1 allocated for model parameters and memory m2 allocated for the key-value cache. The key-value cache (KV cache) stores data while processing requests and is not released until the request is completed. The larger the memory allocated to the KV cache, the more KV cache space it can hold. This increases the probability of hitting cached key-value pairs when processing requests, allowing for more simultaneous requests, higher saturation, and faster performance. Model throughput is positively correlated with the amount of memory m2 allocated to the KV cache. Furthermore, model throughput is also positively correlated with the performance of the computing device.

[0063] Different parallel grouping methods result in different amounts of memory (m1) occupied by model parameters. Given a fixed total memory footprint, the amount of memory (m2) available for allocation to the key-value cache also varies. In other words, using the same computing device, different parallel grouping methods can yield different model throughput efficiencies.

[0064] Therefore, we define a throughput efficiency function f(m2i) to measure the throughput efficiency of the model corresponding to different parallel grouping schemes, where m2i is the memory that can be allocated to the key-value cache under the i-th parallel grouping method. It is understandable that the parameters of the throughput efficiency function f are related to the performance of the computing device.

[0065] In step S1, the parameters of N computing devices are obtained, and the parameters of the model performance evaluation function are determined based on the parameters of the N computing devices. Mainly, the parameters of the computing time proportion function c (Mi, N / Mi) and the parameters of the throughput efficiency function f (m2i) are determined based on the parameters of the computing devices.

[0066] In some embodiments, the relationship between the parameters of each function and the parameters of the computing device can be determined through actual testing. For example, under the same parallel grouping scheme, computing devices with different parameters are tested for computing time share, and then, based on the corresponding relationship between the parameters of the computing devices and the results of the computing time share test, the relationship between the parameters of the computing time share function and the parameter changes of the computing devices is fitted. When the parameters of the N computing devices of the model to be deployed are obtained, the relationship between the parameters of the computing time share function and the parameter changes of the computing devices is directly introduced to obtain the parameters of the computing time share function under the computing device.

[0067] Similarly, based on actual tests, the throughput efficiency of the throughput efficiency function on different computing devices under the same parallel grouping scheme can be determined, and the relationship between the parameters of the throughput efficiency function and the parameters of the computing device can be fitted. Therefore, when the parameters of the N computing devices of the model to be deployed are obtained, the relationship between the fitted parameters of the throughput efficiency function and the parameters of the computing device can be directly introduced to obtain the parameters of the throughput efficiency function under the computing device.

[0068] In step S2, at least one parallel grouping scheme is determined. In the i-th parallel grouping scheme, the computing devices are divided into Mi groups, each group contains N / Mi computing devices, and the computing devices within each group are configured to perform model parallel computing. Mi is the number of pipeline parallel stages, and the computing devices in each group are configured to perform pipeline parallel computing. N / Mi is the parameter segmentation number for model parallelism, i≥1, Mi>1.

[0069] Assuming N=8, when Mi≠1 (Mi equals 1, indicating no pipeline parallelism), there are three grouping schemes:

[0070] M1=2, N / M1=4;

[0071] M2=4, N / M2=2;

[0072] M3=8, N / M3=1.

[0073] When the computing device is determined, there is no need for a grouping scheme. The corresponding pipeline parallel efficiency r is different, the computing time proportion c is different, and the throughput efficiency f is different. That is, the independent variables of each function are different, the function values ​​are different, and the values ​​of the model performance evaluation function are also different.

[0074] For the throughput efficiency function, when calculating the memory that can be allocated to the key-value cache, it can be calculated based on the memory usage of the model parameters under various parallel methods.

[0075] When all N graphics cards are set to model parallelism, the model parameters are evenly divided into N parts. Data input is required on each computing device. However, the cached key-value (KV) value will only generate 1 / N of the KV cache when the parameters are calculated with 1 / N. Therefore, the memory consumption of each graphics card is (m1 + m2) / N.

[0076] When all N cards are set to pipeline parallelism, the parameter layers are divided into N stages and distributed to each of the N cards, with each stage containing multiple parameter layers. However, the first parameter layer, the embedding layer, is placed in the first stage, and the last parameter layer, the post-processing layer, is placed in the last stage, so the memory load is not balanced. Generally speaking, the parameter memory of the last and first layers is slightly larger than that of other layers, but within 3%. Therefore, the memory consumption of each card is (m1+m2) / N+δ(m1 / N), where δ≤3%. It can be assumed that the memory consumption of pipeline parallelism is the same as that of model parallelism.

[0077] Furthermore, memory is never left idle during inference. Aside from the memory m1 occupied by model parameters and a portion of total memory reserved for activation or other temporary memory usage during program execution, all remaining memory is used for the key-value cache. Typically, 10% of total memory is reserved. Therefore, for each of the aforementioned parallel grouping schemes, combined with pipeline parallelism and model parallelism, the memory usage is:

[0078] (m1+m2) / N=0.9M(2)

[0079] Among them, M is the total available memory of N computing devices, and m1 is the memory occupied by model parameters.

[0080] According to the above parallel grouping schemes, the memory m2 / N that can be allocated to the kv cache by each computing device (each card) is 0.9M-m1 / N. Sorting out formula (1), we have:

[0081] throughout=c(Mi,N / Mi)r(Mi)f(0.9M-m1 / N) (3)

[0082] Different grouping modes correspond to different model parameters, and m1 may be different to a certain extent. Therefore, when determining the parallel grouping mode in step S2, the memory m1 occupied by the model parameters corresponding to each parallel grouping mode may be determined at the same time.

[0083] In some embodiments, the mapping relationship between different grouping methods and the memory m1 occupied by the model parameters can be determined based on historical data and real-time test data, so that m1 can be quickly determined.

[0084] In step S3, based on each of the parallel grouping schemes, after determining the independent variables of the computing time proportion function, determining the independent variables of the pipeline parallel balancing rate function, and determining the independent variables of the throughput efficiency function, the value of the model performance evaluation function corresponding to each of the parallel grouping schemes is determined, that is, the model performance evaluation value corresponding to each of the parallel grouping schemes.

[0085] After determining the parameters of each function in step S1 and the independent variables of each function in step S2, Mi and N / Mi corresponding to different parallel grouping schemes can be substituted into formula (3) to evaluate the model performance evaluation values ​​corresponding to different parallel grouping schemes.

[0086] In experiments, when setting N=8 and running the llama2 70b model, the parallel grouping scheme (M2=4, N / M2=2) achieved the fastest processing speed on various graphics cards: a 65% speed increase on the 3090 graphics card, a 99% speed increase on the A100 graphics card, and a 36% speed increase on the A800 graphics card. This is primarily due to the fact that computation time accounts for c. The 3090 and A100 graphics cards utilize PCIe communication, while the A800 uses NVLink. This bandwidth is higher than PCIe, and the A800's computing speed is higher than that of the A100, which in turn is higher than that of the 3090. It can be said that the lower the bandwidth divided by the computing speed, the greater the improvement in model data speed achieved by the technical solutions of the disclosed embodiments.

[0087] However, (M3=8, N / M3=1) is not as good as (M2=4, N / M2=2) because the decrease from r(4) to r(8) is greater than the increase in the computational time from c(4,2) to c(8,1).

[0088] In step S4, the parallel grouping scheme corresponding to the maximum model performance evaluation value is determined as the optimal parallel grouping scheme, and the data processing model is deployed according to the optimal parallel grouping scheme.

[0089] FIG2 is a schematic diagram of an optimal parallel grouping solution in one embodiment of the present disclosure.

[0090] Referring to Figure 2, still referring to the above example, the 8 cards can be finally divided into 4 groups, namely the first computing device group 21, the second computing device group 22, the third computing device group 23, and the fourth computing device group 24. Each group runs in parallel according to the pipeline to achieve pipeline parallel communication; the two cards 2iA and 2iB in each group run in parallel through the model to achieve model parallel communication.

[0091] The optimal parallel grouping scheme only provides the number of groups to divide all cards into and the number of cards in each group. However, in actual execution, which cards are divided into the same group will affect the overall running speed of the model.

[0092] In one embodiment of the present disclosure, if there is no peer-to-peer (P2P) PCIe (Peripheral Component Interconnect Express) communication between the N computing devices, the N / Mi computing devices within each group are configured as non-shared bandwidth computing devices. This process, also known as rank shuffle, can significantly reduce model parallel communication time on certain GPU models.

[0093] On an unsupported PCIe system, if there are eight cards, the first four cards share bandwidth, and the last four cards share bandwidth. Therefore, communication between card 0 and card 1 takes longer than communication between card 0 and card 4. Divide the eight cards into two groups: cards 0, 1, 2, and 3 in the front group, and cards 4, 5, 6, and 7 in the back group. Communication within the same group takes longer than communication between groups. Leveraging this principle, when more than eight cards are involved in a communication, try to divide the participating cards equally between the front and back groups to minimize communication time.

[0094] FIG3 is a schematic diagram of communication after computing devices are grouped in one embodiment of the present disclosure.

[0095] Referring to Figure 3, for example, in the parallel grouping scheme with M2=4 and N / M2=2, if cards 0, 1, 2, and 3 share bandwidth, and cards 4, 5, 6, and 7 share bandwidth, and the order of the cards is shuffled from 0, 1, 2, 3, 4, 5, 6, and 7 to 0, 5, 7, 2, 1, 4, 6, and 3, and then grouped in the new order, then the parallel AllReduce and broadcast communications within the same group will change from the original grouping order of [0-1], [2-3], [4-5], and [6-7] to [0-5], [7-2], [1-4], and [6-3]. In other words, the communication within the four groups becomes the communication between the four groups, and theoretically the communication speed is doubled.

[0096] At the same time, the parallel Send-Recv communication starts from:

[0097] [0-2], [2-4], [4-6], [6-0], [1-3], [3-5], [5-7], [7-1] become:

[0098] [0-7], [7-1], [1-6], [6-0], [5-2], [2-4], [4-3], [3-5].

[0099] That is, 4 intra-group and 4 inter-group communications have become 8 inter-group communications, and the communication speed has been improved.

[0100] In the parallel grouping scheme with M2=4 and N / M2=2, the AllReduce communication time accounts for 46%. The theoretical upper limit of the running speed improvement of this scheme is 30%, and the actual improvement is about 20%.

[0101] It should be noted that when performing grouping in a disordered order, it is necessary to first detect whether the machine is using PCIe communication without P2P. If there is P2P PCIe communication, the order between cards cannot be disrupted and can only be grouped in order, such as [0-1], [2-3], [4-5], [6-7].

[0102] Here is an example of rearranging the order of cards to group them.

[0103] Divide the sort into Mi groups, each group has tp=N / Mi cards, and the cards in each group need to communicate with AllReduce. Each group contains tp / 2 cards that communicate with the previous group and tp / 2 cards that communicate with the next group (when tp=1, there is no AllReduce communication within the group, and this condition does not need to be met).

[0104] Since each card needs to perform send-recv communication with the next group (stage), the jth card needs to communicate with the j+tpth card, one belonging to the front group and one belonging to the back group.

[0105] Assuming the number of cards N = 8 and tp = 2, the order of the cards is shuffled to 0, 5, 7, 2, 1, 4, 6, 3. When tp is not 2, the order is uniformly shuffled to 0, 5, 2, 7, 4, 1, 6, 3. Of course, there are other shuffling methods, and it can be demonstrated that there is no universal shuffling method that is applicable to all parallel configurations. Those skilled in the art can configure the groups based on the actual number of cards and bandwidth sharing, and this will not be exhaustively listed here.

[0106] In order to further improve the model running speed, the embodiment of the present disclosure also improves the model running process, but this improvement will cause changes in the model parameters, which in turn affects the available memory that can be allocated to the key-value cache, affects the parameters and values ​​of the throughput efficiency function, and also affects the parameters of the calculation time proportion function.

[0107] Therefore, when deciding to make the model operation process improvements described below, it is necessary to determine the parameters of the computing time ratio, the parameters of the throughput efficiency function, and the available memory allocated to the key-value cache based on the improved model operation mode, and then determine the optimal parallel grouping scheme under the improved model operation mode.

[0108] In an embodiment of the present disclosure, the model running process improvement includes: setting the number of running instances of the data processing model equal to the number of stages in the parallel grouping scheme plus one.

[0109] The principle is introduced below.

[0110] FIG4 is a schematic diagram of a model task scheduling system in one embodiment of the present disclosure.

[0111] 4 , in an embodiment of the present disclosure, a model task scheduling system 400 is applied to perform task processing. The model task scheduling system 400 includes a waiting queue 41 , a ready queue 42 , a running instance queue 43 , a pause queue 44 , and a completion queue 45 .

[0112] 1) The waiting queue 41 (Waiting) is used to store pending requests. When inference begins, all incoming requests enter the waiting queue. During inference execution, the waiting queue continues to receive requests from the server.

[0113] 2) Ready queue 42 (Ready) stores requests that have been allocated KV cache memory. If sufficient memory is available, ready queue 42 prioritizes requests from paused queue 44 (paused) and allocates KV cache memory to these requests. If paused queue 44 is empty, ready queue 42 allocates KV cache memory to these requests by moving them from the waiting queue 41.

[0114] 3) The Run Instance Queue 43 (Runnings) is a queue of already running instances. The Runnings queue consists of multiple runs, each of which runs a microbatch, which contains multiple requests. The Run Instance Queue 43 maintains a microbatch ID to identify the run currently running on the first computing device. A run instance is defined as the number of task batches currently being processed in the pipeline at a given moment.

[0115] Each request is assembled on the first compute device (assuming it's card 0) by fetching it from the ready queue 42. After the current compute device completes a microbatch, it sends it to the next stage, which then executes the microbatch. The current card does not need to wait; it switches to the next running instance to process the next microbatch.

[0116] After a microbatch has been executed through all stages, it will eventually return to the first computing device. The running process of the first computing device will post-process the returned microbatch. For each request, it will either load the next batch of prefill data, generate a new decode token, or generate a token to indicate the end of the request.

[0117] If a request has finished, it will be moved from the running queue to the completed queue 45 (finished). The current running process moves a batch of completed requests to the completed queue 45. At this time, the running process is partially idle and will continue to grab requests from the ready queue 42 to fill the current microbatch.

[0118] If the number of tokens required to be calculated by the microbatch of the current run significantly exceeds that of the microbatches of other runs, several requests will be kicked out of the current run and moved to the head of the ready queue 42. The next run will start grabbing requests from the head of the ready queue 42.

[0119] 4) Paused queue 44 (Paused) is used to store paused requests. Each time a decode token is processed, a new token is generated. Therefore, memory may need to be allocated for this new token (or not, depending on whether the page allocated to the current request has been used up). If this memory allocation requirement cannot be met, that is, if there is insufficient memory, all memory for the current request is released, and the current request moves from running to pause queue 44, waiting for memory to be allocated again and placed in ready queue 42.

[0120] 5) Completion queue 45 (Finished) stores completed requests. The KVcache corresponding to this queue is not released directly; it is simply frozen. If a request with the same prefix token is encountered, it will be reused, eliminating the need to recalculate the KVcache. If the KVcache runs out of memory, some infrequently used requests will be released. (For example, "Today I want to cook" can reuse the "Today I" portion of "Today I want to go to the zoo").

[0121] By using the model task scheduling system 400 to complete task scheduling, multiple requests can be efficiently and orderly entered into each computing device for processing.

[0122] 5A and 5B are schematic diagrams of an improved model running process in one embodiment of the present disclosure.

[0123] First, communication and computation must be performed on separate streams (independent threads) so that they can overlap. Under this premise, the pipelined parallel operation process is shown in Figure 5A.

[0124] Refer to Figure 5A, which shows the chronological order of microbatch processing by each stage, with time plotted on the horizontal axis. Microbatch 0 moves sequentially from stage 0 to stage 1, stage 2, and stage 3, before returning to stage 0 for the next set of processing (renumbered). This return time is the communication time. Stage 0 must wait for stage 3 to return to microbatch 0 before continuing (renamed microbatch 4), meaning the communication time has elapsed. Each running instance executes a microbatch, and the boxes representing microbatches in the figure represent running instances.

[0125] Referring to Figure 5B, if the number of running instances is set to the number of stages + 1, calculations can be performed within the communication time. As shown in Figure 5B, although there are only four stages, five running instances are set to run sequentially. Therefore, when stage 3 returns micro-batch 0 to stage 0, during this communication time, stage 0 continues to process micro-batch 4 and does not stop calculation. After stage 0 processes micro-batch 4, it directly processes micro-batch 0 (renamed micro-batch 5), realizing the reuse of communication time and calculation time, effectively improving the model throughput. It should be noted that for the convenience of drawing, in Figure 5B, there is no gap between the same batch, but there is a gap between batches, but this gap is not shown. It is particularly important to point out here that four gaps can be covered with one micro-batch.

[0126] When the model parallel grouping method is fixed, the total KV cache memory allocated to each request remains essentially constant. Because runnable requests must be evenly distributed across running instances, the same batch of requests, previously divided into only four (number of stages) parts, now needs to be divided into five (number of stages + 1) parts. This reduces computational saturation.

[0127] At this time, the model performance evaluation function needs to be adaptively modified as follows:

[0128] throughout=c'(Mi,N / Mi)r(Mi)f((0.9M-m1 / N)*Mi / (Mi+1)) (4)

[0129] Among them, c'(Mi,N / Mi) is the calculation time share function re-determined based on the modified model. Since the overall communication time of the improved model is reduced, compared with the calculation time share function c(Mi,N / Mi) in formula (1), under the same independent variables, the value of c'(Mi,N / Mi) is greater than the value of c(Mi,N / Mi).

[0130] Furthermore, after this adjustment, each card must be configured to use a separate stream (thread) for communication, rather than a single stream for both communication and computation. This means that each card must have a separate receive stream and a separate send stream, and must cache received and sent tensors.

[0131] The method provided by the embodiment of the present disclosure divides a large data block into multiple parts, calculates each small part one by one, and can set stage+1 running instances to run simultaneously, making use of all computing devices. Compared with the pipeline parallel operation mode in the prior art that can only set one or two running instances to run simultaneously, it greatly improves the computing power utilization rate.

[0132] After determining to improve the model operation, the optimal parallel grouping scheme needs to be determined according to formula (4) in method 100. By optimizing and setting the number of running instances to the number of stages plus one, the overall performance of the model can be optimized.

[0133] In another embodiment of the present disclosure, after grouping, the model performance can be further optimized by improving the communication method between computing devices. The communication method set up in the embodiment of the present disclosure can be called a postal service, which can significantly shorten the communication time of the parallel pipeline.

[0134] In an embodiment of the present disclosure, computing devices are arranged to communicate via a one-dimensional byte tensor, and the one-dimensional byte tensor is used to record the data name of the data to be transmitted and the compressed package of the data to be transmitted.

[0135] In one embodiment, a transceiver module can be set on each computing device. The transceiver module is used to compress and package the data to be transmitted to form a one-dimensional byte tensor when communication with other computing devices is required, and send the one-dimensional byte tensor to other computing devices, or decompress the one-dimensional byte tensor received from other computing devices to obtain data.

[0136] In one embodiment, the compressed package of the data to be transmitted includes at least one of a first byte tensor, a second byte tensor, and a third byte tensor. The first byte tensor is before the second byte tensor and / or the third byte tensor and is spliced ​​with the second byte tensor or the third byte tensor. The second byte tensor is before the third byte tensor and is spliced ​​with the third byte tensor. The first byte tensor is used to record data parameters of the data to be transmitted, and the data parameters include at least one of a variable name, shape, data type, storage device, and data length. The second byte tensor is used to record the data to be transmitted stored on the CPU, and the third byte tensor is used to record the data to be transmitted stored on the GPU.

[0137] A single communication requires the receiver to know the shape of the data. However, unlike training, inference is different from training. The shape and type of the data sent each time are not fixed. In the inference scenario, only the sender knows the shape and type of the data. To solve this problem, the disclosed embodiment sets the data transmission process as follows: in send-receive communication, the length data of a one-dimensional byte tensor is sent to the other computing device; after sending the length data, the one-dimensional byte tensor is sent to the other computing device.

[0138] The post office system of the embodiment of the present disclosure is described in detail below. The functions of the "post office system" can be implemented by a transceiver module provided on each computing device.

[0139] (1) The post office system customizes communication and only communicates with 1-dimensional ByteTensor.

[0140] (2) For AllGather communication, AllReduce communication, and Broadcast communication, since the receiver knows the shape of the sender, only one communication is performed.

[0141] (3) For point-to-point Send-Recv communication, the first communication sends / receives / exchanges the length L of the ByteTensor. After the first communication is completed, the receiving party knows the shape L of the sender's ByteTensor, which is a Byte data type. At this time, the ByteTensor itself is sent / received / exchanged in the second communication.

[0142] (4) During the shipping process, a package maintains a name-to-data mapping. Data to be sent can be stored in the package, and received data is also stored in the package. The postal system only sends and receives packages, compressing them before sending and decompressing them immediately upon receipt.

[0143] (5) When the postal system packs a package, it first stores the variable name, shape, data type, storage device (CPU or CUDA), and data length of all the data in the package, and saves this meta information in front of the data to form a one-dimensional ByteTensor (i.e., the first byte tensor). Next, it forms a second byte tensor and / or a third byte tensor, where the second byte tensor is a tensor stored on the CPU and the third byte tensor is a tensor stored on the GPU. When both the second byte tensor and the third byte tensor exist, placing the second byte tensor in front of the third byte can reduce the copying when transferring data from the CPU to the GPU, thereby compressing the packaging time. Then, all byte tensors are converted into one-dimensional ByteTensors using a function that has no actual cost, such as view(), and then spliced ​​together and placed on the GPU, with the first byte tensor in front.

[0144] For data that is not a tensor, such as a custom class, a serialization process is required to convert it into a tensor. A deserialization process is also required to reconstruct the tensor and completely restore it to data.

[0145] (6) When the postal system unpacks the package, it converts the received ByteTensor into a name-to-data mapping. According to a specific protocol format, the first byte tensor in the one-dimensional byte tensor is first extracted, which is the variable name, shape, data type, storage device (CPU or CUDA), and data length. Then, based on the information in the first byte tensor, the subsequent data is converted into the original data. At the same time, a ByteTensor storage location is maintained for each piece of data, which is used to quickly replace data with the same name, data type, shape, and device but different content. (Each time it is run in the hidden state, the content changes, but everything else remains unchanged.) This quick replacement can avoid repackaging.

[0146] By using the postal system to package data and setting supported package names, it is possible to transfer arbitrary variables, avoid exposing the internal structure of serialization and deserialization methods, and avoid extensive changes when changing the transmitted content. Furthermore, it supports the direct use of package contents as function input, improving the ease of use and flexibility of packages.

[0147] In summary, the embodiments of the present disclosure determine the optimal parallel grouping scheme for computing devices, set up inter-group pipeline parallelism and intra-group model parallelism, and at the same time improve the model throughput by changing the model operation mode, compress the communication time of model parallelism by setting card rearrangement during grouping, and compress the communication time of pipeline parallelism by setting a post office system (transceiver module) on the card. Based on the hardware conditions of various computing devices, a data processing model with high throughput, small model parameter memory usage, and fast processing speed can be achieved, such as a large-scale pre-trained language model.

[0148] It is understandable that the data processing models referred to in the embodiments of the present disclosure include various artificial intelligence models, such as multiple neural network models (including multiple deep learning models), and the present disclosure does not impose any special restrictions on this.

[0149] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0150] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as "circuits," "modules," or "systems."

[0151] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. The electronic device includes: N computing devices; a memory; and a processor coupled to the memory, the processor being configured to execute the method of any of the above embodiments based on instructions stored in the memory to deploy a data processing model on the N computing devices.

[0152] The electronic device 600 according to this embodiment of the present invention is described below with reference to Figure 6. The electronic device 600 shown in Figure 6 is only an example and should not limit the functions and scope of use of the embodiment of the present invention.

[0153] As shown in Figure 6, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, the aforementioned at least one processing unit 610, the aforementioned at least one storage unit 620, and a bus 630 connecting various system components (including storage unit 620 and processing unit 610).

[0154] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section above. For example, the processing unit 610 can perform the method described in the embodiments of the present disclosure.

[0155] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0156] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0157] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0158] The electronic device 600 can also communicate with one or more external devices 700 (e.g., a keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, modem, etc.). This communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0159] Through the description of the above embodiments, it will be readily understood by those skilled in the art that the example embodiments described herein can be implemented via software or via a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard drive) or on a network and includes several instructions for enabling a computing device (such as a personal computer, server, terminal device, or network device) to execute the methods according to the embodiments of the present disclosure.

[0160] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section of this specification.

[0161] The program product for implementing the above-described method according to an embodiment of the present invention may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0162] The program product may utilize any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0163] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0164] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

Claims

1. A model deployment method, characterized in that: include: Obtain parameters of N computing devices of the model to be deployed, and determine parameters of a model performance evaluation function based on the parameters of the N computing devices, where the model performance evaluation function is the product of a computing time proportion function, a pipeline parallel balancing rate function, and a throughput efficiency function; Determine at least one parallel grouping scheme, where in the i-th parallel grouping scheme, the computing devices are divided into Mi groups, each group containing N / Mi computing devices, the computing devices within each group are configured to perform model parallel computing, Mi is the number of pipeline parallel stages, and the computing devices in each group are configured to perform pipeline parallel computing, N / Mi is the parameter splitting fraction of the model parallelism, i ≥ 1, Mi > 1; Based on each of the parallel grouping schemes, after determining the independent variables of the computing time proportion function, determining the independent variables of the pipeline parallel balancing rate function, and determining the independent variables of the throughput efficiency function, determine the value of the model performance evaluation function corresponding to each of the parallel grouping schemes, that is, the model performance evaluation value corresponding to each of the parallel grouping schemes; The parallel grouping scheme corresponding to the largest model performance evaluation value is determined as the optimal parallel grouping scheme, and the data processing model is deployed according to the optimal parallel grouping scheme.

2. The model deployment method according to claim 1, wherein: Deploying the data processing model according to the optimal parallel grouping scheme includes: If there is no P2P PCIe communication between the N computing devices, the N / Mi computing devices in each group are set as computing devices that do not share bandwidth with each other.

3. The model deployment method according to claim 1, wherein: Also includes: The number of running instances of the data processing model is set equal to the number of stages in the parallel grouping scheme plus one.

4. The model deployment method according to claim 1 or 3, wherein: The independent variables of the computing time proportion function are the number of pipeline parallel stages and the parameter splitting number of model parallelism under the parallel grouping scheme, the independent variable of the pipeline parallel balancing rate function is the number of pipeline parallel stages under the parallel grouping scheme, and the independent variable of the throughput efficiency function is the available memory of the key-value cache under a parallel grouping scheme under the parallel grouping scheme.

5. The model deployment method according to claim 3, wherein: Determining the value of the independent variable of the throughput efficiency function includes: Determining available memory for a key-value cache in a parallel grouping scheme according to the number of the running instances; The available memory of the key-value cache under one of the parallel grouping schemes is used as the value of the independent variable of the throughput efficiency function.

6. The model deployment method according to claim 1, wherein: Also includes: The computing devices are set to communicate via a one-dimensional byte tensor, where the one-dimensional byte tensor is used to record the data name of the data to be transmitted and the compressed package of the data to be transmitted.

7. The model deployment method according to claim 6, wherein: The setting of communicating between the computing devices via a one-dimensional byte tensor includes: A transceiver module is set on each of the computing devices. The transceiver module is used to compress and pack the data to be transmitted to form the one-dimensional byte tensor when communication with other computing devices is required, and send the one-dimensional byte tensor to other computing devices, or decompress the one-dimensional byte tensor received from other computing devices to obtain data.

8. The model deployment method according to claim 7, wherein: The sending the one-dimensional byte tensor to another computing device includes: In a send-receive type communication, sending the length data of the one-dimensional byte tensor to the other computing device; After sending the length data, the one-dimensional byte tensor is sent to the other computing device.

9. The model deployment method according to claim 6, wherein: The compressed package of the data to be transmitted includes at least one of a first byte tensor, a second byte tensor, and a third byte tensor. The first byte tensor is before the second byte tensor and / or the third byte tensor and is spliced ​​with the second byte tensor or the third byte tensor. The second byte tensor is before the third byte tensor and is spliced ​​with the third byte tensor. The first byte tensor is used to record data parameters of the data to be transmitted, and the data parameters include at least one of a variable name, a shape, a data type, a storage device, and a data length. The second byte tensor is used to record the data to be transmitted stored on the CPU, and the third byte tensor is used to record the data to be transmitted stored on the GPU.

10. An electronic device, characterized in that: include: N computing devices; Memory; as well as A processor coupled to the memory, the processor being configured to execute the method according to any one of claims 1 to 9 based on instructions stored in the memory to deploy the data processing model on the N computing devices.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images