Model deployment method and electronic device

Abstract

The present disclosure relates to the technical field of deep learning, and provides a model deployment method and an electronic device. The model deployment method comprises: on the basis of the computing time required to be occupied by each of a plurality of neural network layers of a target data processing model, sequentially allocating the plurality of neural network layers to N computing devices, so that the difference value between the total number of computing times corresponding to the neural network layers on the computing devices is not greater than a preset value, wherein N≥2; and running the target data processing model by means of the N computing devices so as to perform pipelining processing on a plurality of requests to be processed. The embodiments of the present disclosure can improve the balance of pipelining models.

Description

Model deployment method and electronic device Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a model deployment method and electronic device. Background Art

[0002] Pipelining is a parallel computing method that improves the efficiency of large-scale instruction execution. Specifically, it divides the instruction execution process into multiple stages and allows multiple instructions to execute different stages simultaneously within the same clock cycle, thereby achieving overlapping execution of instructions.

[0003] The advantage of pipeline parallelism is that it improves instruction execution throughput and efficiency. Because multiple instructions can be in different stages simultaneously, the model can execute different stages of multiple instructions within the same clock cycle, thereby improving computing device utilization and performance. However, the disadvantage is that since different computing devices execute different stages of different instructions separately, this can lead to imbalances and waste computing power.

[0004] The relevant technology is limited by the imbalance problem of pipeline parallelism. Only a small number of running instances can be set up to run simultaneously, and the data processing efficiency is low, which makes the widespread application of pipeline parallel architecture difficult.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0006] The purpose of the present disclosure is to provide a model deployment method and electronic device for solving the balance problem of pipeline parallel models and enhancing the data processing efficiency of pipeline parallel models.

[0007] According to a first aspect of an embodiment of the present disclosure, a model deployment method is provided, comprising: allocating the multiple neural network layers of a target data processing model to N computing devices in sequence according to the computing time required for each layer, so that the difference between the total computing time corresponding to the neural network layers on each computing device is no more than a preset value, N≥2; and running the target data processing model through the N computing devices to perform pipeline parallel processing on multiple requests to be processed.

[0008] In an exemplary embodiment of the present disclosure, the method of sequentially allocating the multiple neural network layers to N computing devices based on the computing time required for each layer of the target data processing model includes: determining the number of neural network layers corresponding to each type according to the type of the neural network layers of the target data processing model; taking the computing time occupied by a neural network layer of the type with the largest number of layers as a unit of measurement, and determining the computing time required for a layer of each neural network layer according to the unit of measurement; and sequentially allocating the multiple neural network layers to N computing devices so that the difference between the total computing time corresponding to the neural network layers on any two computing devices is not greater than a preset proportion of the unit of measurement.

[0009] In an exemplary embodiment of the present disclosure, allocating the multiple neural network layers in sequence to N computing devices includes: determining the computing time occupied by all neural network layers in the target data processing model based on the product of the computing time required for each layer in each neural network layer and the number of layers of each neural network layer; determining a baseline value of the total computing time corresponding to each computing device based on the ratio of the computing time occupied by all neural network layers to N; allocating the neural network layers in sequence to the 1st to Nth computing devices in the order of the baseline value and all neural network layers, and the number of neural network layers corresponding to each computing device is an integer; when the difference between the total computing times corresponding to the neural network layers on any two computing devices cannot be satisfied to be no greater than the preset value, adjusting the number of at least one neural network layer.

[0010] In an exemplary embodiment of the present disclosure, the target data processing model is a generative pre-trained model, and the multiple neural network layers include an embedding layer, multiple decoding layers, and a normalization layer arranged in sequence. The multiple neural network layers are sequentially distributed to N computing devices according to the computing time required for each layer of the multiple neural network layers of the target data processing model, including: deploying the embedding layer on the 1st computing device, deploying the normalization layer on the Nth computing device, deploying x decoding layers on the 1st computing device to the N-1th computing device, and deploying x-1 decoding layers on the Nth computing device, where x≥2.

[0011] In an exemplary embodiment of the present disclosure, the target data processing model is a generative pre-training model, and the target data processing model includes a first normalization layer, an embedding layer, multiple decoding layers, and a second normalization layer arranged in sequence. The multiple neural network layers of the target data processing model are sequentially distributed to N computing devices according to the computing time required for each layer. The method includes: deploying the embedding layer and the first normalization layer on the first computing device, and deploying the second normalization layer on the Nth computing device. The first normalization layer and the second normalization layer are set to alternately perform post-processing on the output data of the last decoding layer.

[0012] In an exemplary embodiment of the present disclosure, the proportion of tokens that need to be decoded in the target request to the total number of tokens is q, the proportion of computing time required by the normalization layer and the decoding layer is r, the proportion of the first normalization layer performing the post-processing task is p, and running the target data processing model through the N computing devices includes: when q>(1 / (1+r)), setting the last decoding layer to only decode the tokens that need to be decoded, and setting p<(1-(1-q) / qr).

[0013] In an exemplary embodiment of the present disclosure, running the target data processing model through the N computing devices to perform pipeline parallel processing on multiple pending requests includes: pre-filling the multiple pending requests to generate multiple pre-filled tokens, and at the same time, decoding the generated pre-filled tokens through multiple running instances; when an idle running instance appears, obtaining a first number of pre-filled tokens in sequence from multiple pre-filled tokens that have not been decoded and filling them into the idle running instance, and the first number is determined according to the number of pre-filled tokens currently being decoded.

[0014] In an exemplary embodiment of the present disclosure, the obtaining of a first number of pre-filled tokens in sequence from a plurality of pre-filled tokens that have never been decoded and filling them into the idle running instance includes: determining the maximum number A of pre-filled tokens that can be decoded simultaneously based on the available memory of the key-value cache, assuming that the number of pre-filled tokens currently being decoded is a; determining the first number as Aa, and obtaining Aa pre-filled tokens in sequence from a plurality of pre-filled tokens that have never been decoded and filling them into the idle running instance.

[0015] In an exemplary embodiment of the present disclosure, the obtaining of a first number of pre-filled tokens in sequence from a plurality of pre-filled tokens that have never been decoded and filling them into the idle running instance includes: determining the maximum number A of pre-filled tokens that can be decoded simultaneously based on the available memory of the key-value cache; determining the number B of pre-filled tokens decoded simultaneously corresponding to the maximum computing power utilization based on the computing power utilization corresponding to decoding different numbers of pre-filled tokens simultaneously; assuming that the number of pre-filled tokens currently being decoded is a, the first number is b, the difference between b and a is less than a preset value, and B≤a+b≤A; obtaining b pre-filled tokens in sequence from a plurality of pre-filled tokens that have never been decoded and filling them into the idle running instance.

[0016] In an exemplary embodiment of the present disclosure, the obtaining of a first number of pre-filled tokens in sequence from a plurality of pre-filled tokens that have never been decoded and filling them into the idle running instance includes: assuming that the number of pre-filled tokens currently being decoded is a, the number of pre-filled tokens entering the ready queue is c, and the number of running instances that can be run simultaneously in the target data processing model is R; determining the first number based on the rounded result of (a+c) / R; and obtaining the first number of pre-filled tokens in sequence from the ready queue and filling them into the idle running instance.

[0017] According to a second aspect of the present disclosure, an electronic device is provided, comprising: N computing devices; a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

[0018] According to a third aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements any one of the above methods when executed by a processor.

[0019] The disclosed embodiment improves the hierarchical setting scheme of the pipeline parallel model and distributes the neural network layers of the pipeline parallel model (target data processing model) to N computing devices according to the computing time required for each neural network layer. When the model is used to perform pipeline parallel processing on requests, the computing time of each computing device can be relatively uniform, thereby overcoming the imbalance problem of the pipeline parallel model, improving the data processing efficiency of the pipeline parallel model, and enabling the pipeline parallel model to be widely used.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0022] FIG1 is a flowchart of a model deployment method in an exemplary embodiment of the present disclosure.

[0023] FIG2 is a sub-flowchart of step S1 in one embodiment of the present disclosure.

[0024] FIG3 is a schematic diagram of the distribution of neural network layers on various computing devices in one embodiment of the present disclosure.

[0025] FIG4 is a schematic diagram of the distribution of neural network layers on various computing devices in another embodiment of the present disclosure.

[0026] FIG5 is a schematic diagram showing the distribution of neural network layers on various computing devices in yet another embodiment of the present disclosure.

[0027] 6A and 6B are schematic diagrams of data processing time in an embodiment of the present disclosure.

[0028] FIG7 is a sub-flowchart of step S2 in one embodiment of the present disclosure.

[0029] FIG8 is a schematic diagram of the token processing process of step 22 in one embodiment of the present disclosure.

[0030] FIG9 is a schematic diagram of a pipeline parallel scheduling system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0032] The accompanying drawings are merely schematic illustrations of the present disclosure. Identical reference numerals in the drawings denote identical or similar components, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0033] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0034] FIG1 is a flowchart of a model deployment method in an exemplary embodiment of the present disclosure.

[0035] 1 , a model deployment method 100 may include:

[0036] Step S1, sequentially allocating the plurality of neural network layers of the target data processing model to N computing devices according to the computing time required for each layer, so that the difference between the total computing time corresponding to the neural network layers on each computing device is no greater than a preset value, N ≥ 2;

[0037] Step S2: Run the target data processing model through the N computing devices to perform pipeline parallel processing on multiple requests to be processed.

[0038] In the embodiments of the present disclosure, a computing device refers to a hardware device used to accelerate computing, which can process large-scale data and perform complex computing tasks, such as a GPU (Graphics Processing Unit), which can also be called a graphics card, or simply a card.

[0039] The pipeline parallel model is a model for parallel computing that breaks down computing tasks into multiple stages, each of which performs a specific task. Data flows between these stages, allowing different tasks to be executed in parallel on different processors or cores.

[0040] In a pipeline parallel model, multiple neural network layers are sequentially divided into multiple stages, each assigned to a computing device. Each task must pass through each stage (computing device) one or more times for processing. Each computing device sends its results to the next computing device, receiving its tasks from the previous one. Different processing stages may use different processing methods and require different amounts of time, leading to imbalances and wasted computing power. For example, after stage i completes processing instruction a, it needs to send the results of instruction a to stage i+1 for further processing. However, stage i+1 is currently processing instruction b (assuming its processing is more complex than the other stages). In this situation, stage i must wait for stage i+1 to become idle and stop accepting pending data for instruction c from stage i-1. Consequently, all stages must wait for the slowest stage to complete processing before processing data can flow down, resulting in waiting times and wasted computing power. This situation is known as pipeline balance loss.

[0041] For example, the generative pre-trained Transformer (GPT) model consists of an embedding layer, n decoding layers, and a normalization layer, where n is typically 32 or 80. When running on a machine with 8 GPUs (for example, 8 GPUs), these n decoding layers are evenly divided into 8 stages, with the embedding layer placed in the first stage, before all decoding layers, and the normalization layer placed in the last stage, after all decoding layers.

[0042] The embedding layer converts input discrete features or symbols (tokens) into continuous vector representations. These vectors contain semantic information about the tokens, and the input tokens are mapped to an initial token embedding vector space. In GPT, the embedding layer converts input words or tokens into vector representations, enabling the model to understand and process them.

[0043] Decoder layers are the core of GPT and consist of multiple stacked decoder layers. Each decoder layer typically consists of multiple self-attention sublayers and feedforward neural network sublayers. The self-attention mechanism allows the model to interact with information within the input sequence, thereby better understanding contextual relationships. The feedforward neural network facilitates nonlinear transformation and modeling of the input. The decoder layer is used to model the input token sequence and generate the output sequence.

[0044] Normalization layers are used to make input data have an appropriate scale and range, accelerating the training process and reducing the vanishing gradient problem during training. In GPT, normalization is usually achieved through layer normalization to make model training more stable and achieve better generalization performance.

[0045] In the GPT model, the input data is vectorized by the embedding layer, then passes through n decoder layers for feature extraction and sequence generation, and finally passes through the normalization layer to stabilize the model training process.

[0046] During training or inference, the embedding layer generally takes only a short time (about 5%-10% of the time of a decoding layer), but the final normalization layer takes half the time of a decoding layer to one decoding layer. The specific time ratio r is:

[0047] (1)

[0048] Among them, time(final norm) is the computing time occupied by the normalization layer, and time(decode layer) is the computing time occupied by the decoding layer.

[0049] Vocabulary size refers to the number of different words or tokens that a model can handle. In a pipeline-parallel model, the vocab size of each stage may be different, depending on the vocabulary size of the input data processed by that part. In the entire model, the total vocab size is equal to the sum of the vocab sizes of all parts. The vocab size of the same model is the same, and they are only used in the first layer of embedding (converting from token to vector) and the last layer to calculate logits (calculating the probability of the next word to appear). The vocab size of different models can be different. Commonly used ones are all English vocabulary, with a vocab size of 32,000, and the first version of Chinese vocabulary, with a vocab size of 50,048. Others are based on the number of tokens learned by the training model, which may be 100,000 or 200,000.

[0050] Hidden size refers to the number of hidden units. In a pipeline-parallel model, since the model is split into multiple parts, each part may have its own hidden layer. The hidden layer size of each part may be different because different parts may process different types of information or have different parameter sizes.

[0051] For general generative pre-training models, the r value is roughly between 1 and 0.5. At this point, the final stage takes up approximately one decoding layer's worth of computation time compared to the other stages. This causes the other seven cards to wait for the final card's half-decode layer to go from one decoding layer to another before continuing to pass data, wasting computing power and reducing the pipeline-parallel model's data processing capabilities.

[0052] The disclosed embodiment improves the hierarchical setting scheme of the pipeline parallel model and distributes the neural network layers of the pipeline parallel model (target data processing model) to N computing devices according to the computing time required for each neural network layer. When the model is used to perform pipeline parallel processing on requests, the computing time of each computing device can be relatively uniform, thereby overcoming the imbalance problem of the pipeline parallel model, improving the data processing efficiency of the pipeline parallel model, and enabling the pipeline parallel model to be widely used.

[0053] Below, each step of the model deployment method 100 is described in detail.

[0054] In step S1, the multiple neural network layers of the target data processing model are sequentially allocated to N computing devices according to the computing time required for each layer, so that the difference between the total computing time corresponding to the neural network layers on each computing device is no more than a preset value, N≥2.

[0055] In the disclosed embodiments, the target data processing model is a pipeline-parallel model. The neural network layers of the target data processing model are divided into multiple categories, each category including one or more neural network layers. The types of neural network layers include, but are not limited to, the aforementioned embedding layers, decoding layers, normalization layers, or other convolutional layers, pooling layers, etc. The disclosed embodiments do not limit the specific types of layers in the pipeline-parallel model.

[0056] Since the pipeline parallel model includes multiple neural network layers of multiple types, the computing time required for neural network layers of the same type is roughly the same. Therefore, the computing time required for each neural network layer can be determined according to the type of the neural network layer.

[0057] In one embodiment, based on the types of neural network layers of the target data processing model, the number of neural network layers corresponding to each type is determined, the computing time occupied by the neural network layer of the largest type is used as a unit of measurement, and the computing time required for other layers is determined based on the unit of measurement.

[0058] For example, in a generative pre-training model, there is one embedding layer, n decoding layers, and one normalization layer. The decoding layer has the largest number of layers. The computational time required for one decoding layer can be set to 1. According to formula (1), the computational time required for the normalization layer is determined to be r. Similarly, referring to formula (1), the computational time required for the embedding layer is determined to be s. Thus, the total computational time occupied by the M neural network layers is (M-2+r+s) units.

[0059] Since step S1 is used to measure the time contribution of each layer in order to allocate the time to each layer, the use of measurement units to calculate the above computation time does not require consideration of hardware factors. It can be calculated based on preset parameters such as vocabulary size and hidden layer size, which is relatively simple. In other embodiments of the present disclosure, the actual computation time of each neural network layer can also be determined through experiments or other calculation methods.

[0060] It is understandable that although the embodiments of the present disclosure are explained using a generative pre-training model as an example, the methods of the embodiments of the present disclosure can be applied to the deployment of pipeline parallel models including but not limited to generative pre-training models, such as pipeline models, dataflow models, recurrent neural networks (RNN), convolutional neural networks (CNN), tensor pipeline models, etc.

[0061] For different pipeline parallel models, the unit of measurement can be determined based on the type of neural network layer and the computational time required for each type of neural network layer. For some types of neural network layers, the computational time required can be determined based on the ratio of their vocabulary size to the hidden layer size, as referenced in formula (1).

[0062] In one embodiment, the distribution of the neural network layers on each computing device can be determined based on the ratio of the total computing time of the neural network layers of the target data processing model to the number of computing devices N, so that after multiple neural network layers are sequentially distributed to N computing devices, the difference between the total computing times corresponding to the neural network layers on any two computing devices is no more than a preset value.

[0063] The setting of this preset value is used to limit the imbalance of pipeline parallelism. The ideal value is zero (indicating that the pipeline is completely balanced and the calculation time corresponding to each computing device is exactly the same). It can be fine-tuned according to actual conditions.

[0064] The specific value of the preset value can be determined based on the calculation method of the calculation time. If the calculation time is calculated in absolute time units such as milliseconds or picoseconds, the preset value can be set as an absolute time value. If the calculation time is calculated in the above-mentioned measurement units, the preset value can be, for example, a preset ratio of one measurement unit, such as 0.1.

[0065] FIG2 is a sub-flowchart of step S1 in one embodiment of the present disclosure.

[0066] Referring to FIG2 , in one embodiment, step S1 includes:

[0067] Step S11, determining the computing time occupied by all neural network layers in the target data processing model according to the product of the computing time required for each layer in each neural network layer and the number of layers of each neural network layer;

[0068] Step S12, determining a baseline value of the total computing time corresponding to each computing device based on the ratio of the computing time occupied by all neural network layers to N;

[0069] Step S13, allocating the neural network layers to the 1st to Nth computing devices in sequence according to the reference value and the order of all neural network layers, wherein the number of neural network layers corresponding to each computing device is an integer;

[0070] Step S14: When the difference between the total computing times corresponding to the neural network layers on any two computing devices cannot be greater than a preset value, adjust the number of layers of at least one neural network layer.

[0071] Ideally, the ratio of the total computation time of the neural network layers of the target data processing model to the number of computing devices N is equal to the total computation time of the neural network layers. For example, if the total computation time of 100 neural network layers (including 5 types) is 110 units, distributed across 10 computing devices, the baseline total computation time per device is 11, meaning each device receives 11 units of computation time. Generally speaking, the first layer requires no more than 0.1 units of time. According to current industry designs, the last layer takes 0.5 to 1 unit of time, with a maximum of no more than 4 units of time. The neural network layers on each computing device can be assembled one by one based on the computation time taken by each layer from 1 to 100. For example, layers 1 to 9 can be assigned to the first computing device, layers 11 to 20 to the second computing device, and so on. This ensures that the total computation time of the neural network layers on each computing device is approximately 11 units, and the number of neural network layers on each computing device is an integer. If each layer takes 11 units of time, then layers 1 to 11 are involved. If embedding counts as one layer, and embedding only takes 0.1 time, then layers 1-12 are considered. Layers must be arranged consecutively. If the first computation device is 1-9, then the second device must start at 10, not 11. If the first layer is 1-11, then the second layer should be 12-22.

[0072] However, in some cases, since the number of neural network layers on each computing device must be an integer, it is easy to make it impossible to form a neural network layer allocation that meets the above conditions. In this case, the number of neural network layers can be adjusted according to the above-mentioned limiting conditions, such as modifying the number of layers of one or some types of neural network layers, thereby adjusting the computing time occupied by all neural network layers and adjusting the benchmark value of the total computing time corresponding to each computing device, and redistributing the neural network layers in sequence to the 1st to Nth computing devices until the difference between the total computing time corresponding to the neural network layers on any two computing devices is not greater than the preset value.

[0073] For example, according to the above example, the total computing time occupied by all M neural network layers of a generative pre-trained model is (M-2+r+s) units. Assume r=1, s=0.1, M=82 (including 80 decode layers), N=8, then the total computing time occupied by 82 neural network layers is 81.1. The baseline value of the total computing time corresponding to each computing device is 81.1 / 8=10.1375. Each computing device should be allocated approximately 10 units of computing time (equivalent to 10 decode layers).

[0074] According to the rule that the embedding layer is set on the first computing device and the normalization layer is set on the eighth computing device, the time s=0.1 occupied by the embedding layer can be ignored. Therefore, 10 decoding layers are set on the 1st to 7th computing devices, and 10 decoding layers and one normalization layer are set on the 8th computing device. The total computing time on the 1st to 8th computing devices is 10.1, 10, 10, 10, 10, 10, 10, 10, 11 respectively. The difference between the total computing time of the last computing device and the total computing time of the other computing devices is 1, which is much larger than the preset ratio of one unit of measurement, that is, 1*0.1=0.1.

[0075] At this point, you can adjust the number of decode layers to achieve balance.

[0076] In one embodiment of the present disclosure, when the target data processing model is a generative pre-trained model, and the multiple neural network layers include an embedding layer, multiple decoding layers, and a normalization layer arranged in sequence, the number of decoding layers can be adjusted, the embedding layer is deployed on the first computing device, the normalization layer is deployed on the Nth computing device, x decoding layers are deployed on the first computing device to the N-1th computing device, and x-1 decoding layers are deployed on the Nth computing device, where x≥2.

[0077] FIG3 is a schematic diagram of the distribution of neural network layers on various computing devices in one embodiment of the present disclosure.

[0078] Referring to Figure 3, in an example, assume N=8, x=10. In this example, 10 decoding layers 30 can be deployed on the first computing device 31 to the seventh computing device 37, and 9 decoding layers 30 can be deployed on the eighth computing device 38. The total number of decoding layers 30 is adjusted from the original 80 to 79. In addition, the embedding layer 301 is deployed on the first computing device and the normalization layer 40 is deployed on the eighth computing device. The total computing time on the first to eighth computing devices is 10.1, 10, 10, 10, 10, 10, 10, 10, 10, respectively. The difference in the total computing time between any two layers is less than or equal to 0.1, achieving basic balance.

[0079] Experiments have found that, according to the embodiment shown in Figure 3, reducing one decoding layer has minimal impact on the accuracy of the model's final logits (logical values), but can increase processing speed by 6%-12%, improving Model Flops Utilization (MFU). Logits are the raw values ​​output by the model before softmax or sigmoid activation. These values ​​are typically floating-point numbers, and their accuracy depends on the data type used (e.g., 32-bit floating-point, 16-bit floating-point, etc.) and the numerical stability of the computation. Higher accuracy means a more precise representation of the model's output, but it can also increase computational overhead.

[0080] FIG4 is a schematic diagram of the distribution of neural network layers on various computing devices in another embodiment of the present disclosure.

[0081] Referring to Figure 4, in another example, assume N=8, x=11. In this example, 11 decoding layers 40 can be deployed on the 1st computing device 41 to the 7th computing device 47, and 10 decoding layers 40 can be deployed on the 8th computing device 48. The total number of decoding layers 40 is adjusted from the original 80 to 87, and the total computing time on the 1st to 8th computing devices is 11.1, 11, 11, 11, 11, 11, 11, 11, respectively. The difference in the total computing time between any two layers is less than or equal to 0.1, achieving basic balance.

[0082] In the embodiment shown in FIG4 , it is equivalent to adding a decoding layer to each computing device during the time when the 1st to 7th computing devices wait for the 8th computing device. This increases the processing accuracy (the increase in the number of decoding layers increases the processing accuracy) without increasing the overall processing time (the overall processing time is limited by the last computing device), thereby improving the computing power utilization.

[0083] In another embodiment of the present disclosure, when the target data processing model is a generative pre-training model and the multiple neural network layers include an embedding layer, a decoding layer, and a normalization layer, the number and setting of the normalization layer can also be adjusted to achieve pipeline balancing. The previous solution is to adjust the number of decoding layers, and the latter solution is to adjust the position of the normalization layer. It should be pointed out that although the normalization layer will be used multiple times in the model, there is only one and only one normalization layer at the end. The number cannot be adjusted, but the position can be adjusted.

[0084] The methods shown in the embodiments of Figures 3 and 4 can be referred to as unbalanced allocation. It's important to note that with unbalanced allocation, the number of layers must be reduced (or increased) from the start of model training. That is, if a model with 79 decode layers is used for inference, then the model must also be trained with 79 decode layers.

[0085] In one embodiment, a target data processing model is set to include a first normalization layer, an embedding layer, multiple decoding layers, and a second normalization layer arranged in sequence, the embedding layer and the first normalization layer are deployed on the first computing device, and the second normalization layer is deployed on the Nth computing device. The first normalization layer and the second normalization layer are set to alternately perform post-processing on the output data of the last decoding layer.

[0086] FIG5 is a schematic diagram showing the distribution of neural network layers on various computing devices in yet another embodiment of the present disclosure.

[0087] 5 , in yet another embodiment, N=8, an embedding layer 501 and a first normalization layer 503 are deployed on the first computing device 51, and a second normalization layer 504 is deployed on the eighth computing device 58, and the first normalization layer 503 and the second normalization layer 504 are configured to alternately perform post-processing on the output data of the last decoding layer 502.

[0088] Assume that the proportion of post-processing performed by the first normalization layer 503 is p (p < 1), and the proportion of post-processing performed by the second normalization layer 504 is 1-p. Then the total computing time corresponding to the first computing device is pr+x1, the total computing time corresponding to the eighth computing device is (1-p)r+x8, and the total computing time corresponding to the second to seventh computing devices is x2 to x7, respectively, where x1 to x8 are the number of decoding layers on the first to eighth computing devices, respectively.

[0089] Since r<1, p<1, 1-p<1, if p is set reasonably, even if x1~x8 are set equal, the difference between the total computing time corresponding to the neural network layers on each computing device can be no more than the preset value, thus achieving pipeline balance.

[0090] In addition to achieving pipeline balancing, the configuration shown in FIG5 can also reduce the amount of communication between the first computing device and the Nth computing device, thereby improving data processing efficiency.

[0091] If post-processing is performed on the normalization layer of the Nth compute device, the output logits (logical values) need to be communicated to the first compute device. Logits refer to the values ​​output by the model without the softmax or sigmoid activation function. They represent the model's raw scores or scores for each category. Assuming m is the batch size of the data processed by the model, and vocab_size is the size of the model's vocabulary (i.e., the number of categories output by the model), then when communicating the final logits to the first stage, m * vocab_size float data types need to be transmitted. This is because each sample corresponds to a logits vector of length vocab_size, and there are m samples in a batch, so a total of m * vocab_size elements of data need to be transmitted.

[0092] If, however, the hidden states of the last decoding layer are communicated to the first computing device before the normalization layer on the Nth computing device, and then normalized on the first computing device, only m * hidden_size float values ​​need to be communicated. This is because before normalization on the first computing device, the hidden state has a dimension of [batch_size, hidden_size], where batch_size represents the batch size and hidden_size represents the dimension of the hidden state. During communication, each hidden state contains hidden_size elements, and there are m examples in a batch, so a total of m * hidden_size elements of data need to be transmitted.

[0093] In a typical pipeline-parallel model, vocab_size might be 30,000, 50,000, or 100,000, and hidden_size might be 4096 or 8192. Therefore, transmitting the output of the last decoding layer to the first computing device for normalization can reduce communication traffic by 5 to 10 times compared to normalizing the output of the last decoding layer on the last computing device before transmitting the result to the first computing device. This reduction in communication traffic is extremely beneficial for pipeline balancing.

[0094] However, if the first computing device is made to undertake all normalization tasks, the first computing device will replace the Nth computing device and become the slowest link. There will also be the problem of unbalanced pipeline, causing other computing devices to wait and wasting computing resources.

[0095] Unlike related technologies, in the embodiment shown in FIG5 , a first normalization layer 503 is provided on the first computing device to undertake a certain proportion p of post-processing tasks. The first normalization layer 503 can be a batch normalization layer or a layer normalization layer, which is used to normalize the hidden state transmitted from the last decoding layer in the model. Batch normalization is commonly used in convolutional neural networks, while layer normalization is commonly used in architectures such as recurrent neural networks. In GPT, normalization is usually achieved through layer normalization. Therefore, when the target data processing model is a generative pre-training model, both the first normalization layer 503 and the second normalization layer 504 can be layer normalization layers.

[0096] In addition to modifying the model structure, one embodiment of the present disclosure can also implement a code modification to assign a certain proportion p of post-processing tasks to the first computing device, without changing the model structure, by setting only a normalization layer on the last computing device (as in the existing architecture). For example, code can be added to the forward propagation function of the first stage to calculate and apply normalization, without explicitly adding a normalization layer. It should be noted that since post-processing cannot be split, the last layer does not only process a certain proportion p of tokens each time, leaving the rest to the first layer. Rather, in the simplest case, following a certain rhythm, the last and first layers perform post-processing at even-numbered execution steps. This is because while the first layer processes the 2kth microbatch in even-numbered steps, the last layer is combing through the 2k-3th microbatch in even-numbered steps (assuming a stage count of 4). Therefore, a microbatch is post-processed only once, and the first and last layers are post-processed in the same time period. That is, they are both very fast at odd steps (no post-processing), and very slow at even steps (both post-processing), achieving a balanced pipeline effect.

[0097] Subsequent analysis will explore the more complex case of alternating post-processing. For example, as shown in Figure 6B, post-processing is performed on the first step of every three steps on the first layer, and on the first two steps of every three steps on the last layer. This also achieves pipeline balance. This setup, where the first computing device takes on a certain proportion p of post-processing tasks, is called alternative post-processing.

[0098] In addition to modifying the model structure, the embodiment of the present disclosure can also achieve flow balancing by modifying the data processing process of the model.

[0099] In step S2, the target data processing model is run through the N computing devices to perform pipelined parallel processing on multiple requests to be processed.

[0100] In one embodiment, in step S2, it is possible to set the last decoding layer to only perform decoding processing on the tokens that need to be decoded. This method can be implemented during the model application (inference) process.

[0101] There are two stages in the inference process: prefill and decode. The prefill stage is used to read and understand the user's question, after which the model can answer word by word. During the prefill stage, it is only necessary to perform embedding processing on the request at the early stage of each decoding layer (Decode Layer) to convert it into tokens, and store the key-value memory (kv cache) of each token. There is no need to generate new words during the reading process. For example, for the user's question "The weather is sunny, today I want", 7 tokens are obtained in the prefill stage: "天气", "晴朗", ",", "今天", "我", "想". After "天气晴朗", the next word must be ",", and there is no need for the model to generate "吗?" to form "天气晴朗吗?".

[0102] Each decoding layer usually contains a self-attention mechanism and a feed-forward neural network. The self-attention mechanism allows the model to interact information within the input sequence, thereby better understanding the context relationship. The feed-forward neural network helps to perform non-linear transformation and modeling on the input. At the beginning of a decoding layer is the attention mechanism (Attention), where key-value (kv, key-value) is calculated, rotary embedding is performed, and the calculated key-value is cached (cached to the kv cache). Next, the feed-forward neural network processes the kv for decoding.

[0103] However, in the embodiments of the present disclosure, first, it is determined whether a token needs to be decoded. If it needs to be decoded, the token is decoded using a feed-forward neural network according to the normal process and output to the normalization layer; if it does not need to be decoded, no decoding process is performed, and the token is directly output to the normalization layer. This setting can be called "Shrink Tail". If it does not need to be decoded, after calculating all the kv cache, that is, after the kv cache is calculated in the last decoding layer, it is directly discarded and no subsequent calculation of this token is performed (no attention, lm head, normalization, and subsequent normalization layer calculations in the last decoding layer). Note that for Shrink Tail, it means that the tail is no longer calculated, rather than directly output to the last layer.

[0104] For example, for the request "The weather is sunny, today I want", the model only calculates what the next word of "want" is, and tokens such as "weather", "sunny", ",", "today", and "I" do not need to be decoded any further at this step and can be directly output (to the normalization layer). If no further decoding is required, they do not participate in subsequent calculations after the kv cache is calculated.

[0105] This can be achieved by setting the decoding layer to reduce q after calculating qkv and rotary embedding so that it only calculates the tokens that need to be decoded. Here, q is the proportion of tokens that need to be decoded among all tokens, and kv (key-value) is the key value of the tokens.

[0106] By calculating the key value (kv, key-value) and performing rotary embedding in the last decoding layer of the last computing device, and only calculating the tokens that need to be decoded, the data processing time can be greatly reduced, and the computing time of the last computing device can be shortened. When a normalization layer is provided in the last computing device, the pipeline can be balanced as much as possible.

[0107] In the case where there are many tokens in prefill, the Shrink Tail operation reduces the computational amount of the last decoding layer and the normalization layer in the last layer by a large margin. As long as the number of tokens in prefill is more than the number of tokens to be decoded, the pipeline can be balanced, and the balance is the best when the number of tokens in prefill is about the same as the number of tokens to be decoded.

[0108] For example, when the number of prefilled tokens is similar to the number of decoded tokens, assuming the vocab size is approximately equal to 12*hidden size, enabling shrinking can achieve the most balanced pipeline. This is because after decoding only the tokens that need to be decoded, the amount of data received by the normalization layer is reduced, and the computational complexity and time of the normalization layer are similar to those of half a decoding layer. After shrinking, the computation time of the last decoding layer is almost half (actually slightly more than half, about one-twelfth more, due to the QKV matrix multiplication calculation). Together, this is approximately equal to the computation time of one decoding layer. In existing technologies where all decoding layers are evenly distributed across computing devices, the computation time of the last computing device is comparable to that of the others, achieving pipeline balance.

[0109] In step S1 of the present disclosure, when deploying the model, if shrinking is enabled, each layer needs to be allocated to N computing devices based on the computation time of the last decoding layer and the computation time of the normalization layer after shrinking. In the above example, after considering the computation time of the last decoding layer and the computation time of the normalization layer after shrinking, the final layer allocation result is the same as the existing layer allocation result (i.e., the decoding layers are evenly distributed across the computing devices, the embedding layer is set on the first computing device, and the normalization layer is set on the last computing device).

[0110] In one embodiment of the present disclosure, the above-mentioned tail-shrinking operation and the alternating post-processing operation may be combined to further achieve flow balancing.

[0111] In any case, simply turning on shrinkage or performing alternating post-processing can reduce computation time without any cost. However, turning on shrinkage and then alternating post-processing may not necessarily save time.

[0112] Next, we quantify the change in computation time when both shrinking and alternating post-processing are enabled.

[0113] For the convenience of calculation, the computation time of the decoding layer is used as the unit of measurement. First, according to formula (1), the computation time ratio of the normalization layer can be obtained as r.

[0114] Suppose the proportion of tokens that need to be decoded to all tokens is q, and the proportion of tokens that do not need to be decoded to all tokens is 1-q. Assume that on average each stage (stage / computing device) needs to deploy n decoding layers.

[0115] The total calculation time of the last stage after shrinking is turned on is:

[0116] q*(1+r)+(n-1) (2)

[0117] Where n-1 is the computation time of n-1 normal decoding layers, 1+r is the total computation time of the last decoding layer and the normalization layer, and q(1+r) means that after shrinking is enabled, only a proportion q of the tokens need to be decoded, and the total computation time of the last decoding layer and the normalization layer needs to be considered for this proportion.

[0118] The total computation time of other stages that only deploy n decoding layers is n. The total computation time of the last stage that exceeds the other stages is:

[0119] max{ q*(1+r)-1,0} (3)

[0120] When both winsoring and alternating post-processing are set, if the first and last stages are set to each take on half of the post-processing tasks, the sum of the time it takes for the last stage to perform the two alternating steps is:

[0121] [q*(1+r)+(n-1)] +[ q+(n-1)] (4)

[0122] Where q+(n-1) represents the case where no post-processing is performed, and the last stage does not need to go through the normalization layer, and there is no computation time r.

[0123] The sum of the time it takes for the first stage to alternately execute two steps is:

[0124] n+(n+q*r) (5)

[0125] Since the first stage itself has n decoding layers, the computation time is n when no post-processing is performed. When post-processing is performed, the processing time is increased by r. However, since only q proportion of tokens need to be post-processed, the increased processing time is qr. That is, the total processing time of the first stage when post-processing is performed is n + qr.

[0126] Observing formulas (4) and (5), when performing both shrinking and alternating post-processing, the normalized computation time q*r is moved from the last stage to the first stage. The extra computation time of the slowest stage per 2 steps compared to other stages that only deploy the decoding layer is:

[0127] max{ 2(q-1)+qr,qr}=qr (6)

[0128] Because q is less than 1, and after shrinking, the first and last stages perform post-processing alternately. However, the last decoding layer of the last stage can be shrunk a bit, while the first stage cannot, so the slowdown becomes the first stage. To make shrinking profitable, you need to have:

[0129] qr<2*max{q*(1+r)-1,0} (7)

[0130] Because qr>0, formula (7) is split into two parts: qr<2(q-1+qr) and 0<2(q-1+qr). Therefore, when qr>0, it is only necessary to satisfy qr<2(q-1+qr). Therefore, we can deduce:

[0131] q>2 / (2+r) (8)

[0132] That is, when the proportion q of tokens that need to be decoded to all tokens is greater than 2 / (2+r), ​​setting the first stage and the last stage to perform alternating post-processing evenly after turning on tail shrinking can still further balance the pipeline.

[0133] Next, consider the case where q is less than 2 / (2+r). Assume that the first stage in every K steps performs L post-processing, L ≤ K, then the movement ratio p = L / K ≤ 1, and the average calculation time of the last stage is:

[0134] p*[q*(1+r)+(n-1)]+(1-p)*[q+(n-1)]=q+qpr+(n-1) (9)

[0135] The average calculation time of the first stage is:

[0136] n+p*q*r (10)

[0137] The total amount of computation time that the last stage takes longer than other stages is:

[0138] max{pqr+(q-1),pqr}=pqr (11)

[0139] To make profits from shrinking the tail, you need

[0140] pqr <max{q*(1+r)-1,0}≦q-1+qr (12)

[0141] Need to meet

[0142] q>1 / (1+r), p<1-(1-q) / qr (13)

[0143] That is, in the embodiment of the present disclosure, if the first computing device is set to undertake post-processing tasks of proportion p, when q>(1 / (1+r)), the last decoding layer can be set to only decode the tokens that need to be decoded, and p<(1-(1-q) / qr) can be set at the same time.

[0144] 6A and 6B are schematic diagrams of data processing time in an embodiment of the present disclosure.

[0145] Referring to FIG6A , only winsoring is set.

[0146] Assuming q = 2 / 3, r = 1, and N = 4, the data processing time flow is shown in Figure 6A. The computation time for task 61 is the same when it passes through the first, second, and third computing devices, all n. The computation time on the fourth computing device is longer, calculated as n + 1 / 3 according to formula (2). The other computing devices need to wait for the fourth computing device to complete processing before processing the next task, resulting in waiting time between tasks processed by the other computing devices.

[0147] The total time to complete the three tasks 61, 62, and 63 is based on the slowest stage, that is, 3*(n+1 / 3)=3n+1.

[0148] Referring to FIG6B , winsoring and alternating post-processing are set simultaneously.

[0149] Similarly, let q = 2 / 3, r = 1, and N = 4. According to Formula (13), p must be less than 1 / 2, so let p = 1 / 3. The first stage performs post-processing once every three steps. The computational time for the first stage to perform post-processing is n + qr = n + 2 / 3, and the computational time without post-processing is n. The computational time for the last stage to perform post-processing is n + 1 / 3 according to Formula (2), and the computational time without post-processing is q + (n - 1) = n - 1 / 3. The computational time for the remaining stages is n.

[0150] The total time to execute tasks 61, 62, and 63 is limited by the slowest stage, which is either the first or the last stage. The total time it takes for the first stage to execute the three tasks is 2n + n + 2 / 3 = 3n + 2 / 3, and the total time it takes for the last stage to execute the three tasks is (n + 1 / 3) * 2 + n - 1 / 3 = 3n + 1 / 3. In this case, the first stage takes 1 / 3 longer than the last stage, and the slowest stage is the first stage, so the total time to execute the three tasks is 3n + 2 / 3.

[0151] Compared to only turning on shrinking, turning on shrinking and alternating post-processing at the same time, the total time to perform three tasks can be reduced by 1 / 3 of the decoding layer time.

[0152] It can be seen that when the above conditions are met, turning on both winsoring and alternating post-processing and setting the p-value reasonably can improve the processing efficiency of the model.

[0153] In addition to adjusting the processing mechanism of each neural network layer of the model, in step S2, the flow balance can also be achieved by adjusting the number of task processing.

[0154] FIG7 is a sub-flowchart of step S2 in one embodiment of the present disclosure.

[0155] Referring to FIG. 7 , in one embodiment, step S2 may include:

[0156] Step S21 , pre-filling a plurality of pending requests to generate a plurality of pre-filled tokens, and decoding the generated pre-filled tokens through a plurality of running instances;

[0157] Step S22: When an idle running instance appears, a first number of pre-filled tokens are sequentially obtained from a plurality of pre-filled tokens that have not been decoded and filled into the idle running instance, where the first number is determined according to the number of pre-filled tokens currently being decoded.

[0158] For the inference process of the model, processing a request is divided into two stages, namely the pre-filling stage and the decoding stage.

[0159] During the prefill phase, the key-values ​​(kvs) of existing tokens need to be calculated and cached. This occurs for a new request, during which time the attention between each word in the request may not have been calculated. Prefill can be very lengthy. For example, if a user asks the model a long question, such as a 500-word essay asking about typos, the query might contain over 400 tokens.

[0160] During the decoding phase, the probability of the next token needs to be calculated. This occurs during the inference process of a request, which is short, but each request requires repeated decoding and word-by-word inference.

[0161] For example, if a user asks "The weather is fine, I want to play today", the model's reasoning process may be: "The weather is fine, I want to play today" -> "go" -> "zoo" -> "play" -> "."

[0162] The prefill phase involves 7 tokens: "weather", "sunny", ",", "today", "I", and "think". The decode phase has 4 steps, each of which generates only one token, namely:

[0163] 1. Based on “The weather is fine, today I want to”, the next token is inferred / decoded as “go”;

[0164] 2. Based on "The weather is nice, I want to go today", the next token is inferred / decoded as "zoo";

[0165] 3. Based on the sentence “The weather is nice, I want to go to the zoo today”, the next token is inferred / decoded as “play”;

[0166] 4. Based on the sentence “The weather is nice, I want to go to the zoo today”, the next token is inferred / decoded as “.”

[0167] Except for the last step, each inference / decoding requires returning the generated token to the first stage and concatenating it with the generated token for the next inference / decoding.

[0168] To handle incoming requests, the current approach is to first decode the generated token while simultaneously accumulating prefills. When the prefills accumulate to a certain level, all decoding processes are paused, and the current prefills are completed (generating tokens). This converts all prefill requests into decoding tasks, and then continues decoding the generated tokens. This approach is very inefficient.

[0169] In the embodiment of the present disclosure, in order to improve data processing efficiency, it is set to execute prefill and decode simultaneously, that is, in step S21, multiple pending requests are prefilled to generate multiple prefilled tokens, and at the same time, the generated prefilled tokens are decoded through multiple running instances.

[0170] However, this raises the question of when an overly long prefill request is calculated by the model. Due to KV cache memory limitations, a new, overly long prefill request may not be immediately included in the calculation. However, if other requests are added, there may not be enough free memory to add the overly long prefill request, causing the request to be delayed indefinitely. If other requests are blocked, computing power utilization will decrease in the current step and even in the future (until a certain number of requests are completed and enough memory is freed up to accommodate the overly long prefill request).

[0171] In order to solve this problem, the present disclosure sets up step S22. When an idle running instance occurs, a first number of pre-filled tokens are obtained in sequence from multiple pre-filled tokens that have not been decoded and filled into the idle running instance. The first number is determined according to the number of pre-filled tokens currently being decoded.

[0172] The following describes in detail the principle by which step S22 can improve computing power.

[0173] In one embodiment, in step S22, the maximum number A of pre-filled tokens that can be decoded simultaneously can be determined based on the available memory of the key-value cache, and the number of pre-filled tokens currently being decoded is set to a; the first number is determined to be Aa, and Aa pre-filled tokens are obtained in sequence from multiple pre-filled tokens that have not been decoded and filled into the idle running instance.

[0174] In this embodiment, the system first calculates how much key-value cache (kv cache) can be allocated to support the decoding process, and then uses the kv cache to determine a token saturation value A. Assuming that each inference step has a tokens to be decoded, (Aa) uncached tokens are taken from the prefilled tokens each time.

[0175] FIG8 is a schematic diagram of the token processing process of step 22 in one embodiment of the present disclosure.

[0176] Referring to FIG8 , the number of tokens corresponding to the multiple requests 81 to 87 is different, and the width represents the number of tokens.

[0177] Since A is determined based on system performance and is a fixed value, it can be pre-calculated and used directly when retrieving the token.

[0178] In the first step (1), when there are no tokens being decoded, the number of tokens represented by width A is taken from the prefilled token set generated after the prefill phase in the order of token generation, and placed in the idle running instance for decoding to fully utilize computing power. In order to intuitively represent the proportion of the key-value cache occupied by the token being decoded (the part corresponding to a1, a2, and a3 below), Figure 8 uniformly represents the tokens calculated in all running instances as decoding tokens (decode), without distinguishing between running instances.

[0179] In the second step (2), when the token represented by width a1 is being decoded, the number of tokens represented by width A-a1 is taken from the remaining tokens in the pre-filled token set in the order of token generation, and placed into the idle running instance for decoding, so that the number of tokens being decoded is still equal to the saturation value A.

[0180] In the third step (3), when the token represented by width a2 is being decoded, the number of tokens represented by width A-a2 is taken from the remaining tokens in the pre-filled token set in the order of token generation, and put into the idle running instance for decoding, so that the number of tokens being decoded is still equal to the saturation value A.

[0181] In the fourth step (4), when the token represented by the width of a3 is being decoded, the number of tokens represented by the width of A-a3 is taken from the remaining tokens in the pre-filled token set in the order of token generation, and placed in the idle running instance for decoding, so that the number of tokens being decoded is still equal to the saturation value A.

[0182] Among them, a1 corresponds to the token generated after processing the first request 81, a2 corresponds to the token generated after processing the first request 81, the second request 82, and the third request 83, and a3 corresponds to the token generated after processing the first request 81, the second request 82, the third request 83, and the fourth request 84.

[0183] By sequentially obtaining pre-filled tokens that can saturate the key-value cache from multiple pre-filled tokens that have not been decoded and filling them into idle running instances based on the number of pre-filled tokens currently being decoded, computing power can be fully utilized and request processing efficiency can be improved.

[0184] However, it can be observed that in the embodiment shown in FIG8 , the number of tokens taken in each step is different, which easily causes the running load of each filling to be different, and the flow balancing effect is not ideal.

[0185] In another embodiment, in step S22, the maximum number A of pre-filled tokens that can be decoded simultaneously can be determined based on the available memory of the key-value cache, and then the number B of pre-filled tokens decoded simultaneously corresponding to the maximum computing power utilization can be determined based on the computing power utilization corresponding to decoding different numbers of pre-filled tokens simultaneously; assuming that the number of pre-filled tokens currently being decoded is a, the first number is b, the difference between b and a is less than a preset value, and B≤a+b≤A; b pre-filled tokens are obtained in sequence from multiple pre-filled tokens that have not been decoded and filled into the idle running instance.

[0186] This method is an improvement on the one shown in Figure 8. When the workload is not fully loaded, the loading speed b can be controlled to ensure that the number of tokens a calculated in each step is roughly equal to the loading speed b. However, the sum of a and b after loading does not exceed the saturation value at most, nor fall below a value that would otherwise reduce computing power utilization. By controlling the loading speed, we can avoid excessive loading that could lead to the eviction of requests that have already computed a large portion of the workload later, and we can also maintain a relatively high computing power utilization.

[0187] Both of these approaches can keep the number of tokens being calculated in the parallel pipeline at a comparable level in the last few steps, which helps to make the calculation time of each pipeline comparable.

[0188] However, it is also possible to make the water flow more balanced.

[0189] In another embodiment of the present disclosure, in step S22, the number of pre-filled tokens currently being decoded can be set as a, the number of pre-filled tokens entering the ready queue can be set as c, the number of running instances that can run simultaneously in the target data processing model can be set as R, and the first number is determined based on the rounded result of (a+c) / R, and then the first number of pre-filled tokens are obtained in sequence from the ready queue and filled into the idle running instance.

[0190] In order to understand this embodiment, the pipeline parallel scheduling system is first introduced.

[0191] FIG9 is a schematic diagram of a pipeline parallel scheduling system according to an embodiment of the present disclosure.

[0192] 9 , in an embodiment of the present disclosure, a model task scheduling system 900 is applied to process tasks. The model task scheduling system 900 includes a waiting queue 91 , a ready queue 92 , a running instance queue 93 , a pause queue 94 , and a completion queue 95 .

[0193] 1) The waiting queue 91 (Waiting) is used to store pending requests. When inference begins, all incoming requests enter the waiting queue. During inference execution, the waiting queue continues to receive requests from the server.

[0194] 2) Ready queue 92 (Ready) stores requests that have been allocated KV cache memory. If sufficient memory is available, ready queue 92 prioritizes requests from paused queue 94 (paused) and allocates KV cache memory to these requests. If paused queue 94 is empty, ready queue 92 allocates KV cache memory to these requests by moving them from wait queue 91.

[0195] 3) Run instance queue 93 (Runnings) is a queue of already running instances. The Runnings queue consists of multiple runs, each of which runs a microbatch, which contains multiple requests. Run instance queue 93 maintains a microbatch ID to identify the run currently running on the first computing device. A run instance here can be understood as the number of task batches being processed in the pipeline at a given moment.

[0196] Each request is assembled on the first compute device (assuming it's card 0) by fetching it from the ready queue 92. After the current compute device completes a microbatch, it sends it to the next stage, which then executes the microbatch. The current card does not need to wait; it switches to the next running instance to process the next microbatch.

[0197] After a microbatch has been executed through all stages, it will eventually return to the first computing device. The running process of the first computing device will post-process the returned microbatch. For each request, it will either load the next batch of prefill data, generate a new decode token, or generate a token to indicate the end of the request.

[0198] If a request has finished, it will be moved from the running queue to the completed queue 95 (finished). The current running process moves a batch of completed requests to the completed queue 95. At this time, the running process is partially idle and will continue to grab requests from the ready queue 92 to fill the current microbatch.

[0199] If the number of tokens required to be calculated by the microbatch of the current run significantly exceeds that of the microbatches of other runs, several requests will be kicked out of the current run and moved to the head of the ready queue 92. The next run will start grabbing requests from the head of the ready queue 92.

[0200] 4) Paused queue 94 (Paused) is used to store paused requests. Each time a decode token is processed, a new token is generated. Therefore, memory may need to be allocated for this new token (or not, depending on whether the page allocated to the current request has been used up). If this memory allocation requirement cannot be met, that is, if there is insufficient memory, all memory for the current request is released, and the current request moves from running to pause queue 94, waiting for memory to be allocated again and placed in ready queue 92.

[0201] 5) Completion queue 95 (Finished) stores completed requests. The key-value cache corresponding to this queue is not released directly; it is simply frozen. If a request with the same prefix token is encountered, it is reused, eliminating the need to recalculate the key-value cache. If the key-value cache runs low on memory, some infrequently used requests are released. (For example, "Today I want to cook" can reuse the "Today I" portion of "Today I want to go to the zoo").

[0202] By using the model task scheduling system 900 to complete task scheduling, multiple requests can be efficiently and orderly entered into each computing device for processing.

[0203] Next, let’s return to step S22.

[0204] After the first split, whether prefill or decode, ready or running will contain a certain number of tokens. Let this number be m. The value of m changes with the number of requests. For example, if there are no requests from the next moment, m will decrease. In other words, m is not a fixed value. Of course, the embodiment shown in Figure 8 can ensure that m remains constant when there are sufficient requests, that is, n times the saturation value, where n is the number of cards. Improvements to Figure 8 that control the filling speed can also ensure that m does not fluctuate significantly even if there are not enough requests.

[0205] When m is not a constant value, if all tokens are taken out of ready and put into running, it will be very unbalanced. For example, ready receives 4 requests in 4 steps, and each request has 200, 10, 20, and 10 tokens respectively. If all tokens are taken out of ready and put into running, the first pipeline will process 200 tokens, while the other pipelines will only process 20 tokens at most, which is very unbalanced. Three cards spend more than 90% of their time waiting for the pipeline with the most tokens. In addition, at this time, it is completely unknown how many tokens the subsequent requests will have, and it is impossible to process them according to the set number of tokens. If the number of tokens is set too low, the subsequent requests with a large number of tokens cannot be processed in time. If the number of tokens is set too high, there will be no subsequent requests and computing power will be wasted.

[0206] Therefore, in the embodiment of the present disclosure, the number of pre-filled tokens currently being decoded can be set as a, the number of pre-filled tokens entering the ready queue can be set as c, the number of running instances that can run simultaneously in the target data processing model can be set as R, and the first number can be determined based on the rounded result of (a+c) / R, and then the first number of pre-filled tokens can be obtained in sequence from the ready queue and filled into the idle running instance.

[0207] That is, each time only the average number of tokens of the sum of the current ready and running tokens is added to the idle running.

[0208] The following example illustrates this.

[0209] Assume that ready receives 4 requests in 4 steps, and each request has 200, 10, 20, and 10 tokens respectively, and a total of 4 running are set.

[0210] In the first step, the first request comes in with 200 tokens. The number of tokens in ready and runnings[0] to runnings[3] are: Ready: 200, runnings: 0, 0, 0, 0. At this time, a = 0, c = 200, r = 4, (a + c) / R = 50. 50 tokens are taken from ready and put into the idle running instance, which is runnings[0]. The number of tokens in runnings[0] and ready changes as follows: runnings[0]: 0 → 50, ready: 200 → 150.

[0211] In the second step, the second request comes in with 10 tokens. The number of tokens in ready and runnings[0] to runnings[3] are Ready:150+10, runnings:50,0,0,0. At this time, a=50, c=160, r=4, (a+c) / R=52.5, rounded to 53. 53 tokens are taken from ready and put into the idle running instance, which is runnings[1] in order. Then the number of tokens in runnings[1] and ready changes as follows: runnings[1]:0→53, ready:160→107.

[0212] And so on, the following is briefly expressed:

[0213] In the third step, the third request comes in with 20 tokens:

[0214] Before getting token from ready: Ready:107+20, runnings:50,53,0,0,

[0215] After getting token from ready: runnings[2]:0→58, ready:127→69

[0216] Step 4: The fourth request comes in with 10 tokens:

[0217] Before getting token from ready: Ready:69+10, runnings:50,53,58,0

[0218] After getting token from ready: runnings[3]:0→60, ready:79→19

[0219] In the fifth step, no requests come in, and runnings[0] completes processing the 50 tokens taken out in the first step and becomes an idle running instance:

[0220] Before getting token from ready: Ready:19, runnings:0,53,58,60

[0221] a=171, c=19, r=4, (a+c) / R=47.5, rounded to 48. However, there are not 48 tokens in ready at this time, so all tokens in ready can be retrieved:

[0222] After getting token from ready: runnings[0]:0→19, ready:19→0

[0223] In the sixth step, no requests come in. Only runnings[1] completes processing the 53 tokens taken out in the second step and becomes an idle running instance. The number of tokens in ready and runnings is:

[0224] Ready:0, runnings:19,0,58,60

[0225] In step 7, no requests come in. Only runnings[2] completes processing the 58 tokens taken out in step 3 and becomes an idle running instance. The number of tokens in ready and runnings is:

[0226] Ready:0, runnings:19,0,0,60

[0227] In step 8, no requests come in. Only runnings[3] completes the processing of the 60 tokens taken out in step 3 and becomes an idle running instance. It should be noted that in this step, all 200 tokens of the first request and all 10 tokens of the second request are processed, and two tokens are generated and retained in runnings[3]. The number of tokens in ready and runnings is:

[0228] Ready:0, runnings:19,0,0,2

[0229] In step 9, no requests come in. Only runnings[0] completes the processing of the 19 tokens taken out in step 5 and becomes an idle running instance. It should be noted that in this step, all 20 tokens of the third request and all 10 tokens of the fourth request are processed, and two tokens are generated and retained in runnings[0]. The number of tokens in ready and runnings is:

[0230] Ready:0, runnings:2,0,0,2

[0231] After one round

[0232] Ready:0, runnings:0,0,0,4, runnings[3]:4->1, ready:0->3

[0233] Ready:3, runnings:0,0,0,1, runnings[0]:0->1, ready:3->2

[0234] Ready:2, runnings:1,0,0,1, runnings[1]:0->1, ready:2->1

[0235] Ready:1, runnings:1,1,0,1, runnings[2]:0->1, ready:1->0

[0236] Finally stabilized at

[0237] Ready:0, runnings:1,1,1,1

[0238] Until 4 requests are completed one after another.

[0239] Even with a constant influx of requests, this solution ensures that adjacent pipelines (running concurrently) maintain a consistent number of tokens. This solution transforms a sudden surge or plummet in token requests into multiple, slowly increasing or decreasing steps, reducing the difference between adjacent pipelines to a fraction of their original value. This solution, combined with the embodiment shown in Figure 8 and the embodiment for controlling the loading rate, can, under normal circumstances, ensure that the difference in token counts between adjacent pipelines does not exceed 2%, and achieves a completely identical number of tokens under saturation conditions.

[0240] In summary, the embodiments of the present disclosure can achieve that the total amount of computing time corresponding to each computing device is basically the same by adjusting the number of neural network layers, adjusting the distribution of neural network layers on each computing device, and adjusting the scheduling method of the target data processing model during operation, eliminating the phenomenon that some computing devices perform processing time and space on other computing devices, and can fully utilize the computing power of each computing device to improve the overall processing efficiency of the target data processing model.

[0241] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0242] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. The electronic device includes: N computing devices; a memory; and a processor coupled to the memory, the processor being configured to execute the method described in any of the above embodiments based on instructions stored in the memory to deploy a data processing model on the N computing devices.

[0243] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as "circuits," "modules," or "systems."

[0244] Through the description of the above embodiments, it will be readily understood by those skilled in the art that the example embodiments described herein can be implemented via software or via a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard drive) or on a network and includes several instructions for enabling a computing device (such as a personal computer, server, terminal device, or network device) to execute the methods according to the embodiments of the present disclosure.

[0245] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section of this specification.

[0246] The program product for implementing the above-described method according to an embodiment of the present invention may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0247] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0248] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

Claims

1. A model deployment method, characterized in that: include: Allocate the plurality of neural network layers to N computing devices in sequence according to the computing time required for each layer of the plurality of neural network layers of the target data processing model, so that the difference between the total computing times corresponding to the neural network layers on each computing device is no greater than a preset value, N ≥ 2; The target data processing model is run through the N computing devices to perform pipeline parallel processing on multiple requests to be processed.

2. The model deployment method according to claim 1, wherein: The step of sequentially allocating the plurality of neural network layers to N computing devices according to the computing time required for each layer of the plurality of neural network layers of the target data processing model comprises: Determining the number of neural network layers corresponding to each type of neural network layer according to the type of the target data processing model; The computation time occupied by a neural network layer of the type with the largest number of layers is used as a unit of measurement, and the computation time required for a layer of each type of neural network layer is determined based on the unit of measurement; The multiple neural network layers are sequentially distributed to N computing devices so that the difference between the total computing times corresponding to the neural network layers on any two computing devices is no more than a preset ratio of one of the measurement units.

3. The model deployment method according to claim 1, wherein: Distributing the plurality of neural network layers sequentially to the N computing devices includes: Determine the computing time occupied by all neural network layers in the target data processing model according to the product of the computing time required for each layer in each neural network layer and the number of layers of each neural network layer; Determine a baseline value of the total computing time corresponding to each computing device based on the ratio of the computing time occupied by all neural network layers to N; Allocating the neural network layers to the first to Nth computing devices in sequence according to the reference value and the order of all the neural network layers, wherein the number of neural network layers corresponding to each computing device is an integer; When the difference between the total computing times corresponding to the neural network layers on any two computing devices cannot be greater than the preset value, the number of layers of at least one neural network layer is adjusted.

4. The model deployment method according to any one of claims 1 to 3, wherein: The target data processing model is a generative pre-trained model, the multiple neural network layers include an embedding layer, multiple decoding layers, and a normalization layer arranged in sequence, and the multiple neural network layers are sequentially distributed to N computing devices according to the computing time required for each layer of the multiple neural network layers of the target data processing model, including: The embedding layer is deployed on the first computing device, the normalization layer is deployed on the Nth computing device, x decoding layers are deployed on the first computing device to the N-1th computing device, and x-1 decoding layers are deployed on the Nth computing device, where x≥2.

5. The model deployment method according to any one of claims 1 to 3, wherein: The target data processing model is a generative pre-trained model, and the target data processing model includes a first normalization layer, an embedding layer, multiple decoding layers, and a second normalization layer arranged in sequence. The multiple neural network layers of the target data processing model are sequentially distributed to N computing devices according to the computing time required for each layer. The method includes: The embedding layer and the first normalization layer are deployed on the first computing device, and the second normalization layer is deployed on the Nth computing device. The first normalization layer and the second normalization layer are configured to alternately perform post-processing on the output data of the last decoding layer.

6. The model deployment method according to claim 1, wherein: Running the target data processing model through the N computing devices includes: setting the last decoding layer to only decode tokens that need to be decoded.

7. The model deployment method according to claim 5, wherein: The ratio of tokens that need to be decoded in the target request to the total number of tokens is q, the ratio of computing time required by the normalization layer and the decoding layer is r, the ratio of the first normalization layer performing the post-processing task is p, and running the target data processing model through the N computing devices includes: When q>(1 / (1+r)), the last decoding layer is set to decode only the tokens that need to be decoded, and p is set to <(1-(1-q) / qr).

8. The model deployment method according to claim 1, wherein: The step of running the target data processing model on the N computing devices to perform pipeline parallel processing on the multiple pending requests includes: Pre-filling the plurality of pending requests to generate a plurality of pre-filled tokens, and decoding the generated pre-filled tokens through a plurality of running instances; When an idle running instance occurs, a first number of pre-filled tokens are sequentially obtained from a plurality of pre-filled tokens that have not been decoded and filled into the idle running instance, where the first number is determined according to the number of pre-filled tokens currently being decoded.

9. The model deployment method according to claim 8, wherein: The step of sequentially acquiring a first number of pre-filled tokens from a plurality of pre-filled tokens that have not been decoded and filling the pre-filled tokens into the idle running instance comprises: Determine the maximum number A of pre-filled tokens that can be decoded simultaneously based on the available memory of the key-value cache. Let the number of pre-filled tokens currently being decoded be a. The first number is determined to be Aa, and Aa pre-filled tokens are sequentially obtained from a plurality of pre-filled tokens that have not been decoded and filled into the idle running instance.

10. The model deployment method according to claim 8, wherein: The step of sequentially acquiring a first number of pre-filled tokens from a plurality of pre-filled tokens that have not been decoded and filling the pre-filled tokens into the idle running instance comprises: Determine the maximum number A of pre-filled tokens that can be decoded simultaneously based on the available memory of the key-value cache; Determine the number of pre-filled tokens that can be decoded simultaneously corresponding to the maximum computing power utilization, based on the computing power utilization corresponding to the simultaneous decoding of different numbers of pre-filled tokens. Assume that the number of pre-filled tokens currently being decoded is a, the first number is b, the difference between b and a is less than a preset value, and B≤a+b≤A; b pre-filled tokens are sequentially obtained from a plurality of pre-filled tokens that have not been decoded and filled into the idle running instance.

11. The model deployment method according to claim 8, wherein: The step of sequentially acquiring a first number of pre-filled tokens from a plurality of pre-filled tokens that have not been decoded and filling the pre-filled tokens into the idle running instance comprises: Assume that the number of pre-filled tokens currently being decoded is a, the number of pre-filled tokens entering the ready queue is c, and the number of running instances that can run simultaneously in the target data processing model is R; Determine the first quantity according to the rounded result of (a+c) / R; A first number of pre-filled tokens are sequentially obtained from the ready queue and filled into the idle running instance.

12. An electronic device, characterized in that: include: N computing devices; Memory; as well as A processor coupled to the memory, the processor being configured to execute the method according to any one of claims 1 to 11 based on instructions stored in the memory to deploy the data processing model on the N computing devices.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images