Data processing method and device and electronic equipment

Abstract

The invention provides a data processing method and device and electronic equipment, and relates to the technical field of large models. The method comprises the following steps: receiving and processing a preheating request to obtain a corresponding relationship between the token number and processing performance data; in response to the request arrival message, determining the number of tokens of the to-be-processed request; determining a plurality of processing modes according to the number of tokens of the to-be-processed request and the processing performance data, wherein the token execution time from the first step to the Ni-th step in the i-th processing mode is monotonically increased and then monotonically decreased; processing time corresponding to each processing mode is calculated according to the processing performance data, the processing mode with the minimum processing time is determined as a target processing mode, and the processing time comprises token execution time and bubble time of each step; and processing the token of the to-be-processed request according to the target processing mode. According to the embodiment of the invention, the response speed of the large language model to the input information can be improved.

Description

Background Art

[0002] At present, the demand for language model reasoning is increasing day by day, including the need for long text input such as paper reading summaries, article error checking, and polishing.

[0003] The time from when the server receives the long text input to when the model responds for the first time (outputs the first token) is called TTFT (time to first token). During this time, the model needs to read and understand the content input by the user, a process called prefill. Therefore, TTFT is the prefill time, which is also the user's waiting time. In order to improve the user experience, it is necessary to compress TTFT as much as possible to reduce the user's waiting time.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a data processing method, device and electronic device for improving the response efficiency of a large language model to user requests and shortening user waiting time.

[0006] According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, comprising: receiving a preheating request and processing it to obtain a corresponding relationship between the number of tokens and processing performance data; responding to a request arrival message to determine the number of tokens of the request to be processed; determining a plurality of processing methods according to the number of tokens of the request to be processed and the processing performance data, the i-th processing method is to divide the tokens of the request to be processed into Ni steps for processing, the batch size corresponding to the j-th step in the i-th processing method is aij, the token execution time from the 1st step to the Ni-th step in the i-th processing method satisfies the condition of first monotonically increasing and then monotonically decreasing, the batch size is the number of tokens corresponding to a batch, i≥1, Ni≥1, 0≤j≤Ni-1; calculating the processing time corresponding to each of the processing methods according to the processing performance data, determining the processing method with the smallest processing time as the target processing method, the processing time including the token execution time and bubble time of each step; processing the tokens of the request to be processed according to the target processing method.

[0007] In an exemplary embodiment of the present disclosure, receiving and processing a preheating request to obtain a corresponding relationship between the number of tokens and processing performance data includes:

[0008] Processing the first preheating request in multiple batches, wherein the batch size of each batch in the first preheating request is not greater than the batch size of the previous batch;

[0009] Processing the second preheating request in multiple batches, wherein the batch size of each batch in the second preheating request includes multiple preset batch sizes;

[0010] Recording the absolute execution time of each token in each batch in processing the second warm-up request, and a first corresponding relationship between the absolute execution time of each token and the batch size of the batch to which the token belongs;

[0011] Determine the absolute time multiplier corresponding to each preset batch size according to the first corresponding relationship, the absolute time multiplier is a multiple of the average token execution time of a token to the preset basic token execution time under the preset batch size, the preset basic token execution time corresponds to the minimum value of multiple preset batch sizes, and the absolute time multiplier is used to represent the processing efficiency corresponding to different batch sizes in the target model;

[0012] A token execution time function is determined according to the first corresponding relationship and the absolute time multiplier, and the token execution time function is used to represent the relative execution time of the target model for each token with a serial number without the influence of the batch size.

[0013] In an exemplary embodiment of the present disclosure, determining the absolute time multiplier corresponding to each preset batch size according to the first corresponding relationship includes:

[0014] Determine the first execution time of the first token corresponding to the end of the kth batch processing according to the absolute execution time of the tokens of the kth batch, determine the second execution time of the first token corresponding to the start of the k+1th batch processing according to the absolute execution time of the tokens of the k+1th batch, and determine the relative time multiplier corresponding to the batch size of the k+1th batch according to the ratio of the second execution time to the first execution time, the batch size of the kth batch is different from that of the k+1th batch, 1≤k≤M-1, M is the number of preset batch sizes;

[0015] Assume that the absolute time multiplier corresponding to the batch size of the first batch is 1, determine the absolute time multiplier corresponding to each batch size according to the relative time multiplier corresponding to each batch size, and obtain the absolute time multiplier corresponding to each preset batch size.

[0016] In an exemplary embodiment of the present disclosure, determining a token execution time function according to the first corresponding relationship and the absolute time multiplier includes:

[0017] Determine the relative execution time corresponding to each token according to the ratio of the absolute execution time of each token to the absolute time multiplier corresponding to the batch to which it belongs;

[0018] The token execution time function is formed by fitting regression according to the relative execution time of each token.

[0019] In an exemplary embodiment of the present disclosure, calculating the processing time corresponding to each of the processing modes according to the processing performance data includes:

[0020] Determine the absolute time multiplier corresponding to the batch size aij according to the absolute time multiplier corresponding to each preset batch size, where the batch size aij is the batch size of the jth step in the i-th processing method;

[0021] Determine the average relative execution time corresponding to each token in the j-th step according to the batch size of the first j steps in the i-th processing mode and the token execution time function;

[0022] Determine the token execution time of the j-th step according to the product of the average relative execution time corresponding to each token, the batch size aij of the j-th step, and the absolute time multiplier corresponding to the batch size aij;

[0023] The processing time corresponding to the ith processing method is determined according to the sum of the token execution times corresponding to each step in the ith processing method.

[0024] In an exemplary embodiment of the present disclosure, determining multiple processing methods according to the number of tokens of the request to be processed and the processing performance data includes:

[0025] In the i-th processing method, the batch sizes of the first Ni-1 steps are equal, the batch size of the Ni-th step is less than or equal to the batch size of the first Ni-1 steps, and the sum of the batch sizes of the Ni steps is equal to the number of tokens of the pending requests.

[0026] In an exemplary embodiment of the present disclosure, determining multiple processing methods according to the number of tokens of the request to be processed and the processing performance data includes:

[0027] In the i-th processing method, the batch sizes from the 2nd step to the Ni-th step are equal, the batch size of the 1st step is less than or equal to the batch size from the 2nd step to the Ni-th step, and the sum of the batch sizes of the Ni steps is equal to the number of tokens of the pending requests.

[0028] In an exemplary embodiment of the present disclosure, determining multiple processing methods according to the number of tokens of the request to be processed and the processing performance data includes:

[0029] In the i-th processing method, the maximum difference between the batch sizes of each step is 1, the batch size of the j+1-th step is greater than or equal to the batch size of the j-th step, and the sum of the batch sizes of Ni steps is equal to the number of tokens of the pending requests.

[0030] In an exemplary embodiment of the present disclosure, determining multiple processing methods according to the number of tokens of the request to be processed and the processing performance data includes:

[0031] In the i-th processing method, the batch size of the Ni-th step is the smallest, the token execution time of the j-th step does not exceed the token execution time of the j+1-th step, the batch size of the j-th step is the sum of half the batch size of the j+1-th step and the calculation batch size, the processing time corresponding to the calculation batch size is half the processing time of the j+1-th step, and the sum of the batch sizes of the Ni steps is equal to the number of tokens of the pending requests.

[0032] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing device, including:

[0033] a preheating module, configured to receive and process a preheating request to obtain a corresponding relationship between the number of tokens and processing performance data;

[0034] A request pre-processing module, configured to respond to a request arrival message and determine the number of tokens for a pending request;

[0035] A processing method generation module is configured to determine multiple processing methods according to the number of tokens of the pending request and the processing performance data, wherein the i-th processing method is to divide the tokens of the pending request into Ni steps for processing, the batch size corresponding to the j-th step in the i-th processing method is aij, and the token execution time from the 1st step to the Ni-th step in the i-th processing method satisfies a monotonically increasing and then a monotonically decreasing trend, and the batch size is the number of tokens corresponding to a batch, i≥1, Ni≥1, 0≤j≤Ni-1;

[0036] a processing method evaluation module, configured to calculate the processing time corresponding to each processing method according to the processing performance data, and determine the processing method with the smallest processing time as the target processing method, wherein the processing time includes the token execution time and the bubble time of each step;

[0037] The request processing module is configured to process the token of the request to be processed according to the target processing method.

[0038] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

[0039] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the data processing method as described in any one of the above items is implemented.

[0040] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of any of the above methods when executed by a processor.

[0041] The disclosed embodiment provides a plurality of processing methods for the request to be processed by obtaining model processing performance data based on a preheating request, pre-calculates the processing time of each processing method based on the model processing performance data, and finally uses the processing method with the shortest processing time to process the request to be processed. This can effectively improve the processing efficiency of the large language model, shorten the response time to the request to be processed, shorten the user waiting time, and improve the user experience.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0044] Figure 1 is a flow chart of a data processing method in an exemplary embodiment of the present disclosure.

[0045] Figure 2 Schematic diagram of bubble time in one embodiment of the present disclosure.

[0046] Figure 3 It is a sub-flow chart of step S1 in the embodiment of the present disclosure.

[0047] Figure 4 It is a schematic diagram of calculating the relative time multiplier in one embodiment of the present disclosure.

[0048] Figure 5 Schematic diagram of absolute time multipliers corresponding to various preset batch sizes in an exemplary embodiment of the present disclosure.

[0049] Figure 6 It is a sub-flowchart of step S4 in the exemplary embodiment of the present disclosure.

[0050] Figure 7 is a block diagram of a data processing device in an exemplary embodiment of the present disclosure.

[0051] Figure 8 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0053] In addition, the accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0054] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0055] Figure 1 is a flow chart of a data processing method in an exemplary embodiment of the present disclosure.

[0056] refer to Figure 1 , the data processing method 100 may include:

[0057] Step S1, receiving and processing a preheating request to obtain a corresponding relationship between the number of tokens and processing performance data;

[0058] Step S2, responding to the request arrival message, determining the number of tokens of the request to be processed;

[0059] Step S3, determining multiple processing methods according to the number of tokens of the pending request and the processing performance data, the i-th processing method is to divide the tokens of the pending request into Ni steps for processing, the batch size corresponding to the j-th step in the i-th processing method is aij, the token execution time from the 1st step to the Ni-th step in the i-th processing method satisfies the requirement of first monotonically increasing and then monotonically decreasing, the batch size is the number of tokens corresponding to one batch, i≥1, Ni≥1, 0≤j≤Ni-1;

[0060] Step S4, calculating the processing time corresponding to each of the processing methods according to the processing performance data, and determining the processing method with the smallest processing time as the target processing method, wherein the processing time includes the token execution time and bubble time of each step;

[0061] Step S5: Process the token of the request to be processed according to the target processing method.

[0062] The disclosed embodiment provides a plurality of processing methods for the request to be processed by obtaining model processing performance data based on a preheating request, pre-calculates the processing time of each processing method based on the model processing performance data, and finally uses the processing method with the shortest processing time to process the request to be processed. This can effectively improve the processing efficiency of the large language model, shorten the response time to the request to be processed, shorten the user waiting time, and improve the user experience.

[0063] The disclosed embodiments can be used to improve the response rate of a large language model and optimize the user experience.

[0064] When a large model processes sequence generation tasks (such as machine translation, language models, etc.), it usually uses prefill to generate some tokens in advance, which is equivalent to preprocessing and understanding user requests so that the subsequent decoding stage can rely on these prefilled tokens for further generation to output responses for users. The prefill plus decoding process is collectively called reasoning. The time from the user inputting the request to getting the response (the first character is output in the decoding stage) is the time of the model's prefill stage.

[0065] In order to improve the response rate of large language models, it is necessary to shorten the prefill time. One way to shorten the prefill time is to improve the processing efficiency of the prefill stage.

[0066] Large language models usually divide the requested tokens into multiple batches for processing. It is generally believed that the more tokens are processed in each batch, the higher the processing efficiency of the model will be (the lower the average token execution time for each token). For example, on the experimental model, 64 tokens are divided into 2 batches for processing, and each batch takes about 26-27ms, while processing 64 tokens in one batch only takes 29-31ms, that is, if 64 tokens are processed separately, it takes 52-54ms, and the processing efficiency is increased by 70% together.

[0067] However, experiments have shown that the more tokens a batch processes, the better.

[0068] First, generally speaking, in long text processing scenarios, the number of tokens processed in the pre-filling stage is much larger than the number of tokens processed in the decoding stage. If the number of tokens processed in a batch in the pre-filling stage is too large, the processing time of the pre-filling stage will be much longer than that of the decoding stage, and the decoding stage will need to wait for the pre-filling stage to generate enough tokens before it can start working. This situation will cause different stages to wait for each other, resulting in "bubbles", that is, the situation where computing resources are not fully utilized during the waiting stage. In order to avoid this situation, it is necessary to reasonably set the number of tokens in each batch in the pre-filling stage to minimize the waiting time and improve the utilization of computing resources. Secondly, the larger the batch, the longer the token execution time of a single batch, the longer the time interval between batches, and the longer the waiting time for users. Among them, the token execution time refers to the necessary time required for the model to process the token, and the processing time of a step includes the token execution time and the bubble time.

[0069] Now let's introduce the concept of batch size, which is the number of tokens processed in a batch. Based on the above analysis, the larger the batch size, the shorter the token execution time for each token, but the corresponding bubble time is also longer, and the processing time of a single batch is longer; the smaller the batch size, the longer the token execution time for each token, but the corresponding bubble time is smaller, and the processing time of a single batch is shorter. Therefore, to reduce the overall processing time, we can neither simply set the batch size to the maximum nor to the minimum.

[0070] The relationship between bubble time and processing time, as well as the relationship between batch size and processing time are analyzed below.

[0071] Firstly, the relationship between bubble time and processing time is analyzed.

[0072] Figure 2 Schematic diagram of bubble time in one embodiment of the present disclosure.

[0073] refer to Figure 2, assume that the model divides the hardware into four parts (rank, including rank0-rank3), each part processes a stage (stage) in the computing task, after the first part (rank0) processes the calculation of batch A in the first stage, it sends the calculation result to the second part (rank1) for processing, until the fourth part (rank3) completes the calculation of batch A in the fourth stage and completes the calculation processing of batch A; at the same time, after the first part (rank0) sends the result of batch A to the second part (rank1), it receives the calculation task of the first stage of the new batch, namely batch B, and processes it, thus forming a pipeline in which each part processes one stage of computing respectively. This processing method is called pipeline parallelism.

[0074] In order to simplify the analysis process, the communication time of each part is set to be synchronized, that is, after each part completes the calculation of the current batch, each part sends the processing results to the next part at the same time and receives a new batch at the same time. The time for each part to process a batch is called a step, and the start and end times of each stage are set to be the same.

[0075] like Figure 2 As shown in the figure, the token execution time of batch A in each part rank0, rank1, rank2, and rank3 is A0, A1, A2, A3, and so on. It can be seen that due to the existence of pipeline parallelism, the processing time of each step is limited by the maximum processing time of each part (stage) in the step, that is, the time overhead of the i-th step (step) is the maximum value of the processing time from the i-(pp-1)th step to the i-th step, and the time of negative steps is set to 0 in advance. Where pp is the number of processing stages (stage number) of the pipeline parallel model, that is, the number of ranks (parts) mentioned above, in Figure 2 Medium pp=4.

[0076] For example, in Figure 2 In the example, the time of each stage is the maximum time of the last four stages that have occurred. Therefore, the time from a batch entering the first part to being output from the fourth part, that is, the time it takes for the model to process a batch in one stage (for example, the time from batch A entering rank0 to being output from rank3), is:

[0077]

[0078] where t i is the token execution time of the ith phase (e.g. Figure 2 A1 in the max i-pp+1≤k≤i t kis the longest token execution time among the most recent PP phases that have occurred, that is, the actual processing time of the i-th phase (e.g. Figure 2 T1 in ), n+1 is the total number of stages required to process the token corresponding to a request (in Figure 2 In the example, request 21 is divided into 7 batches for processing, and rank 0 is input 7 times to form batches A0 to G0, then n = 6), from i = 0 to i = n + pp-1 is the number of stages required for the model to complete the processing of the batch and then complete the tasks that the model has received from stage 0 to stage n. For example, Figure 2 In the T6 stage, all tokens of request 21 are received and processed in the rank0 stage. However, three stages, T7 to T9, are required to process all the batches received in the T4 to T6 stages. i =0, that is, before processing the current batch, the token execution time of the batch in each stage is 0, and after completing the processing of n+1 stages (steps) of the current batch, the token execution time of the batch in the subsequent closing stage is 0.

[0079] It can be seen that TTFT includes two terms. The first term It is the sum of the time necessary for the model to process the request in n+1 stages, that is, the sum of the token execution times, for example Figure 2 Assume n+1=7, then the first term of TTFT in formula (1) is A0+B0+C0+D0+E0+F0+G0, and the second term is the bubble time, which is the sum of the waiting time of the request in each stage, for example Figure 2 The second term of formula (1) is K2+K3+K4+K6, i.e. (B1-C0)+(B2-D0)+(B3-E0)+(F1-G0), which is caused by the imbalance of the water flow and can be eliminated. The following proves a proposition: the bubble time is greater than or equal to pp-1 maximum step times.

[0080] For the convenience of expression, common characters such as i, j, k, n, etc. are used in the following proof process to represent variables. The i, j, k, n in the following proof process do not have the same meaning as the i, j, k, n in other parts of the present disclosure.

[0081] Assume that there are n+1 steps in total, and the jth step takes the longest time, that is, max 0≤k≤n t k =t j ,have:

[0082]

[0083] Where T BBubble time.

[0084] Since (max i-pp+1≤k≤i t k )≥t i , so for the first summation we have:

[0085]

[0086] because So for the second summation we have:

[0087]

[0088] Since (max i-pp+1≤k≤i t k )≥t i-pp+1 , so for the third term, we have:

[0089]

[0090] The above items can be sorted out to obtain the inequality:

[0091]

[0092] Assume that when i<0 or i>n, t i＝0 , the last term can be turned into The second term can be obtained by changing the subscript So the second term minus the last term cancels out, and we get the final inequality:

[0093] T B ≥(pp-1)*t j =(pp-1)*max 0≤k≤n t k (8)

[0094] Therefore, the bubble time is greater than or equal to pp-1 maximum step times, that is, the bubble time is greater than pp-1 maximum token execution times.

[0095] According to this conclusion, the bubble time is minimum when the equality of the above inequality (8) is achieved.

[0096] because When the equality sign of inequality (8) holds, Corollary: For pp>1, we have:

[0097]

[0098] because Similarly, for pp>1, we have:

[0099]

[0100] Adjusting the subscripts gives:

[0101]

[0102] And max 0≤k≤n t k =t j , so we have:

[0103] t1≤t2≤…≤t j-1 ≤t j ≥t j+1 ≥t j+2 ≥…≥t n (12)

[0104] Therefore, in order to minimize the bubble time, that is, the equality of inequality (8) holds, it is necessary to make the token execution time of each step monotonically increase first and then monotonically decrease. Of course, it also includes only monotonically increasing or only monotonically decreasing, among which only monotonically increasing can be expressed by j=n, and only monotonically decreasing can be expressed by j=1.

[0105] In summary, the larger the batch size, the higher the operating efficiency. Increasing the batch size will reduce the first term of TTFT (the necessary time to execute prefill, that is, the sum of the token execution times of each step). The bubble time is greater than or equal to the longest token execution time multiplied by the number of pipeline parallel stages minus one (pp-1). Reducing the longest token execution time in each step will reduce the second term of TTFT.

[0106] The relationship between batch size and processing time is analyzed below.

[0107] In the above derivation process, it is assumed that when i>n, =0, that is, after completing the processing of the current batch, each part needs to complete the processing of the batches received from stage 0 to stage n. For long texts, the super-long input makes only one request in the prefill state most of the time. When this request completes the prefill processing, the request enters the decode state. The decode time is not included in TTFT, so each settlement of TTFT satisfies the above assumptions. However, in the scenario of non-long texts, frequent short texts make the prefill processing and decode processing of each request mixed together. As long as there are requests coming, the entire reasoning system is in a state of continuous operation. Only when the system ends or no requests come, the above assumptions are met. There are very few application scenarios.

[0108] In the experiment, the token execution time of the model for each token under a constant number of tokens was counted, and it was found that the token execution time of each token increases linearly with the increase in the number of tokens processed in the prefill stage. For example, the batch size of each batch is 2048 tokens, but as the number of processed batches increases, the stage time of the model processing the same batch size increases from 0.37 seconds to 1.85 seconds, and the average processing time per token increases by 0.0157 seconds. This feature of long text means that if the batch size of each batch is set to the same, the token execution time of each step will gradually increase until it reaches a peak in the first one or two steps of the decoding stage, and the bubble time is pp-1 times the longest token execution time. This method will result in a large bubble time.

[0109] In addition, this characteristic of long texts also makes it difficult to estimate the token execution time corresponding to each batch size in the processing performance data. The token execution time test results of the same batch of tokens vary with the number of tokens that have been tested, and it is difficult to accurately obtain the exact token execution time corresponding to a batch size through direct test results. Therefore, the present disclosure introduces the concept of "time multiplier". Some sampling points for batch sizes are set, that is, multiple preset batch sizes, and the token execution time corresponding to the minimum preset batch size is taken as the preset basic token execution time. After testing and conversion, the multiplier of the token execution time of each preset batch size relative to the adjacent preset batch size for the same token (relative time multiplier) is obtained, and then the multiplier of the token execution time of each preset batch size relative to the preset basic token execution time (absolute time multiplier) is obtained.

[0110] Next, each step of the data processing method 100 is described in detail.

[0111] In step S1, a preheating request is received and processed to obtain a corresponding relationship between the number of tokens and processing performance data.

[0112] It should be noted that step S1 is only executed when the model is online. When the model is formally applied later, method 100 starts from step S2.

[0113] Figure 3 It is a sub-flow chart of step S1 in the embodiment of the present disclosure.

[0114] refer to Figure 3 In an exemplary embodiment, step S1 may include:

[0115] Step S11, processing the first preheating request in multiple batches, wherein the batch size of each batch in the first preheating request is not greater than the batch size of the previous batch;

[0116] Step S12, processing the second preheating request in multiple batches, the batch size of each batch in the second preheating request includes multiple preset batch sizes, and the batch size is the number of tokens corresponding to one batch;

[0117] Step S13, recording the absolute execution time of each token in each batch in processing the second preheating request, and a first corresponding relationship between the absolute execution time of each token and the batch size of the batch to which the token belongs;

[0118] Step S14, determining the absolute time multiplier corresponding to each preset batch size according to the first corresponding relationship, the absolute time multiplier is a multiple of the average token execution time of a token to the preset basic token execution time under the preset batch size, the preset basic token execution time corresponds to the minimum value of multiple preset batch sizes, and the absolute time multiplier is used to represent the processing efficiency corresponding to different batch sizes in the target model;

[0119] Step S15, determining a token execution time function according to the first corresponding relationship and the absolute time multiplier, wherein the token execution time function is used to represent the relative execution time of the target model for each token with a serial number without the influence of the batch size.

[0120] In the disclosed embodiment, after running the preset model, the machine is first started through one or more warm-up requests, so that the performance status is improved from P8 to P0, so that the machine is in the state of maximum performance. In addition, since the first large block memory allocation takes a certain amount of time, the first communication needs to establish a link (of course, the additional time overhead of establishing a communication connection may also be caused by allocating memory), which also takes a certain amount of time. These time consumptions are one-time, and only the first communication and the first memory allocation will occur. By making the model process warm-up requests and using a certain amount of warm-up time to process these things, the model will not have to process these first times when processing formal requests, so that the processing time of formal requests will not increase by several seconds or even more than ten seconds.

[0121] It should be noted that when warming up, you should first run a large batch (large batch size). If you run a small batch first, then when you run a large batch next time, the originally allocated memory cannot accommodate the large batch, which will cause memory reallocation and require additional time. When running a small batch after a large batch, there will be no additional memory allocation time. That is, when running a warm-up request, run at least one of the largest specific batch sizes first, such as running a batch of 8192 tokens at the beginning, and then run some small batches. After the processing time stabilizes (that is, there is no additional memory allocation time), start sampling and testing.

[0122] Therefore, in step S11, the first preheating request is divided into multiple batches for processing, and the number of tokens in each batch in the first preheating request is set not to be greater than the number of tokens in the previous batch, thereby eliminating the above-mentioned "first time" time so that subsequent tests can obtain more accurate results.

[0123] Next, the token execution time corresponding to each batch size is tested using the second preheating request. In step S12, the second preheating request can be processed in multiple batches, and the batch size of each batch in the second preheating request is set to be no less than the batch size of the batch with the previous sequence number, and the batch size of each batch in the second preheating request includes multiple preset batch sizes.

[0124] By processing the preheating request, the token execution time corresponding to multiple preset batch sizes can be sampled for subsequent calculation and processing. In order to improve efficiency, the preset batch size can be smaller, such as 32, 64, 128, 256, 512, 768, 1024, 1536, 2048, 2560, 2816, 3072, 3584, 4096, 5120, 6144, 8192. After processing the preheating request, the data points of the efficiency curve (i.e., the processing efficiency corresponding to different batch sizes) can be obtained.

[0125] Even if the same batch size is run in the same operating scenario, the actual running time will fluctuate, so multiple data collection is required. You can set it in advance for a fixed batch size, and run a set of data in multiple steps. The number of steps is set in advance, at least 2, generally set to 3-6, for example, fixed to 4.

[0126] In step S13, the absolute execution time of each token in each batch in processing the second warm-up request and the first corresponding relationship between the absolute execution time of each token and the batch size of the batch to which the token belongs are recorded.

[0127] Since the calculation time of each token of a long text increases linearly with the number of tokens processed by the model, the embodiment of the present disclosure calculates the time at the connection of two sets of data through linear fitting, and then divides them to obtain the relative time multiplier of the two sets of data.

[0128] In an exemplary embodiment, when determining the absolute time multiplier corresponding to each preset batch size according to the first corresponding relationship in step S14, the first execution time of the first token corresponding to the end of the kth batch processing can be determined according to the absolute execution time of the token of the kth batch, and the second execution time of the first token corresponding to the start of the k+1th batch processing can be determined according to the absolute execution time of the token of the k+1th batch, and the relative time multiplier corresponding to the batch size of the k+1th batch is determined according to the ratio of the second execution time to the first execution time. The batch sizes of the kth batch and the k+1th batch are different, 1≤k≤M-1; the absolute time multiplier corresponding to the batch size of the first batch is assumed to be 1, and the absolute time multiplier corresponding to each batch size is determined according to the relative time multiplier corresponding to each batch size, so as to obtain the absolute time multiplier corresponding to each preset batch size.

[0129] Figure 4 It is a schematic diagram of calculating the relative time multiplier in one embodiment of the present disclosure.

[0130] refer to Figure 4 , assuming that we want to calculate the relative time ratio of batch size 512 (a batch includes 512 tokens) and batch size 768, we run a group of 512 first according to the number of steps of 4, that is, run 4 batches with a batch size of 512 tokens (hereinafter referred to as batch size 512), and then run 4 batches with a batch size of 768 tokens. Since it is assumed that the time to run each token increases linearly, the average time to run each token is equal to the time to run the middle point. Figure 4 The horizontal axis of each data point in is the token number of the midpoint of the batch interval. After obtaining the measured data, the least squares method can be used to fit a straight line showing the change of the token execution time of a batch as the number of tokens run through changes, and the value of the two batches at the intersection point, that is, 3048, can be calculated. Figure 4 The line a in the figure calculates (rather than measures) the execution time (first execution time) for the 3048th token (first token) when processing at a batch size of 512, and Figure 4 Line b in Calculates the execution time (second execution time) for the 3048th token (first token) when processing according to a batch size of 768.

[0131] exist Figure 4In the example, based on the formula for fitting straight line a from sampling points K1, K2, K3, and K4, it is calculated that the point K5 with a horizontal coordinate of 3048 corresponds to a vertical coordinate of 280ms (meaning the average token execution time for tokens with serial numbers from 3048-256 to 3048+256), and based on the formula for fitting straight line b from sampling points G1, G2, G3, and G4, it is calculated that the point G5 with a horizontal coordinate of 3048 corresponds to a vertical coordinate of 360ms (meaning the average token execution time for tokens with serial numbers from 3048-384 to 3048+384). That is, for the same token with serial number 3048, when the batch size is 512, the first execution time of the model is 280÷512=0.547ms, and when the batch size is 768, the second execution time of the model is 360÷768=0.469ms, thus the ratio of the processing efficiency of batch size 768 to batch size 512, that is, the relative time multiplier is 0.469÷0.547=0.857. It can be understood that for the same long text request, the time taken to complete the request with batch size 768 is about 0.857 times the time taken to complete the request with batch size 512.

[0132] Define the average token execution time of the tokens running the minimum preset batch size (batch size 32 in the above example) as the preset basic token execution time, define its absolute time multiplier as 1, and then calculate the relative time multiplier of batch size 64 to batch size 32 by running two sets of data of batch size 32 and batch size 64 continuously, and then multiply the relative time multiplier of batch size 64 to batch size 32 by the absolute time multiplier of batch size 32 to get the absolute time multiplier of 64, and so on, to get the absolute time multiplier corresponding to each preset batch size. For example, assuming that the absolute time multiplier of batch size 512 is calculated to be 0.5, then the absolute time multiplier of batch size 768 can be calculated

[0133] 0.5×0.857=0.429.

[0134] A set of data may be run for each preset batch size in step S12, recorded in step S13, and then in step S14 the relative time multiples of adjacent preset batch sizes are calculated, and the absolute time multiples of each preset batch size are calculated one by one.

[0135] The number of preset batch sizes is limited. In actual processing, batch sizes outside the preset batch size may be encountered. For example, the number of prefill tokens of the pending request is 34567. One processing method determined in step S3 is to first run 16 batches of batch size 2048, and then run a batch of batch size 1799. Batch size 1799 is between the preset batch size 1536 and the preset batch size 2048. At this time, the absolute time multiplier corresponding to the batch can be obtained by interpolation.

[0136] Figure 5 Schematic diagram of absolute time multipliers corresponding to various preset batch sizes in an exemplary embodiment of the present disclosure.

[0137] refer to Figure 5 For the above example, a line graph of "batch size - absolute time ratio" is generated based on the absolute time ratio corresponding to each preset batch size, where the horizontal axis is the batch size and the vertical axis is the absolute time ratio corresponding to the batch size. By observing the data points, it is found that the absolute time ratio generally shows a downward trend with the increase of batch size, but it also jitters back and forth. For example, the time ratio of batch size 2560 is lower than that of batch size 2048 and batch size 2816. If high-order polynomial interpolation is used for data that jitters back and forth, it may cause oscillation (Oscillation) of the fitted data, that is, overfitting. Therefore, the embodiment of the present disclosure adopts linear interpolation.

[0138] Taking batch size 1799 as an example, to get the absolute time multiplier of batch size 1799, we can first use the binary search method to find two adjacent data points, namely batch size 1536 (1536, 0.3345) and batch size 2048 (2048, 0.2085), and calculate the horizontal axis ratio (1799-1536) ÷

[0139] (2048-1536)=0.514, and then the ordinate of the interpolation point with the abscissa of 1799 is calculated as: 0.3345×(1-0.514)+0.2085×0.514=0.2698. Thus, the absolute time factor of the batch size 1799 is 0.2698 based on linear interpolation.

[0140] The batch size corresponding to the actual processing method does not always fall between the two preset batch sizes. There are two other cases.

[0141] The first case is when the batch size is smaller than the minimum preset batch size, for example, the batch size is less than 32. From the machine's perspective, if the batch size is too small, the program startup time will dominate, and the program startup time is constant, that is, the increase in batch size will not increase the time too much. Figure 5 As shown, the absolute time multiplier for a batch size of 32 is 1, and the absolute time multiplier for a batch size of 64 is 0.556. It can be said that running a batch of 32 tokens requires 32 units of time, but running a batch of 64 tokens requires 64×0.556=35.6 units of time, which is only 11% more time. When the batch size is less than 32, the extra time is even more negligible. Therefore, it is believed that when the batch size is small enough, the corresponding token execution time is constant (this constant refers to the same prefill interval. The corresponding time for different prefill intervals must be different, and the amount of calculation is different. The time to run the first 32 tokens may only take 0.15s, but the time to run the 100000th to 100032nd tokens takes 2s). Figure 5 In the example, the token execution time corresponding to batches with a batch size less than 32 is 32 units of time, and the absolute time multiplier is 32 divided by the batch size. For example, for a batch size of 15, the absolute time multiplier is 32÷15=2.13.

[0142] The second case is when the batch size is larger than the maximum preset batch size, e.g. Figure 5 5120 of 5120. Observation Figure 5 The batch size-absolute time ratio line chart shown in the figure shows that it is basically flat after 2048. From the perspective of the machine, if the amount of computing is large enough, the machine will run at full power, the operating efficiency will reach saturation, and the absolute time ratio will tend to be stable. Therefore, for the case where the batch size is larger than the maximum preset batch size, its absolute time ratio is estimated to be the absolute time ratio of the maximum preset batch size. For example, the absolute time ratio of a batch size of 6000 is 0.199199.

[0143] Next, in step S15, a token execution time function is determined according to the first corresponding relationship and the absolute time multiplier, where the token execution time function is used to represent the relative execution time of the target model for each token with a serial number without the influence of the batch size.

[0144] Considering that the execution time of a token is affected by the batch size of the batch it is in, in order to accurately compare the impact of the processing order on the token execution time, the batch size impact needs to be removed. In an exemplary embodiment, the relative execution time corresponding to each token can be determined based on the ratio of the absolute execution time of each token to the absolute time multiplier corresponding to the batch it is in, and then the token execution time function is formed by fitting regression based on the relative execution time of each token.

[0145] In step S15, the sampled token execution time is divided by the absolute time multiplier corresponding to the batch size to form a "token number-absolute execution time" correspondence, and then a function of the absolute execution time that changes with the token number is fitted, namely the token execution time function, where the "absolute execution time" is the token execution time without the effect of the batch size. Assume that the absolute time multiplier function is l, and the absolute execution time corresponding to the sth token is f(s), and the token execution time function is:

[0146] f(s)=slope*s+b (13)

[0147] Where slope is the slope of the token execution time function, and b is the intercept of the token execution time function.

[0148] Next, the processing time corresponding to a step can be determined based on the token execution time function, and then the time of each step in the i-th processing method can be calculated among the multiple processing methods given in step S3. The token execution time of each step in the i-th processing method can be summed up and added to the bubble time to obtain the processing time corresponding to the i-th processing method.

[0149] The settings for different processing methods will be described in detail in subsequent embodiments. Here we first introduce how to calculate the token execution time of a step based on the token execution time function.

[0150] Figure 6 It is a sub-flowchart of step S4 in the exemplary embodiment of the present disclosure.

[0151] refer to Figure 6 In an exemplary embodiment, step S4 may include:

[0152] Step S41, determining the absolute time multiplier corresponding to the batch size aij according to the absolute time multiplier corresponding to each preset batch size, wherein the batch size aij is the batch size of the jth step in the i-th processing method;

[0153] Step S42, determining the average relative execution time corresponding to each token in the j-th step according to the batch size of the first j steps in the i-th processing mode and the token execution time function;

[0154] Step S43, determining the token execution time of the j-th step according to the product of the average relative execution time corresponding to each token, the batch size aij of the j-th step, and the absolute time multiplier corresponding to the batch size aij;

[0155] Step S44, determining the processing time corresponding to the ith processing method according to the sum of the token execution time corresponding to each step in the ith processing method.

[0156] Let the sum of the number of tokens previously run in the i-th processing method be c q for:

[0157]

[0158] Where q is the number of steps that have been run in the i-th processing method, and a ij is the number of tokens (batch size) corresponding to the jth step in the i-th processing method, 0≤j≤Ni-1, Ni is the number of steps in the i-th processing method, then the token execution time t of the jth step in the i-th processing method ij for:

[0159]

[0160] in, is the average relative execution time of each token in the jth step, multiplied by the batch size a of the jth step ij is the relative execution time in the jth step, multiplied by the absolute time multiplier l(a) corresponding to the batch size ij ) obtains the token execution time of the jth step in the ith processing method under the two influencing factors of batch size and token processing order.

[0161] In step S44, the processing time corresponding to the i-th processing mode is the sum of the total processing time from step 0 to step Ni-1 plus the bubble time. Since the bubble time is greater than or equal to pp-1 maximum token execution times, TTFT satisfies:

[0162]

[0163] If the bubble time is to be minimized, the equality sign needs to be established. According to formula (13) and its corresponding proof process, it is necessary to ensure that the token execution time of each step first increases monotonically and then decreases monotonically according to the step number. The processing methods provided in step S2 of the disclosed embodiment can meet this condition. Therefore, the processing time corresponding to the i-th processing method calculated in the disclosed embodiment, that is, step S44, is calculated using the right side of the inequality of formula (16).

[0164] The following describes in detail the processing method provided by the embodiment of the present disclosure.

[0165] In step S2, the model responds to the request arrival message and obtains the number of tokens of the request to be processed (the number of prompt tokens), that is, the number of tokens that need to be prefilled, and then determines multiple processing methods for the number of tokens in step S3. Next, in step S4, the processing time corresponding to each processing method is calculated according to the corresponding relationship between the number of tokens obtained in step S1 and the processing performance data according to the aforementioned calculation method, and the processing method with the smallest total processing time is selected, and then the processing method is output to the reasoning engine in step S5. As long as the reasoning engine is scheduled according to the processing method, TTFT can be made as small as possible, so that the user can get a response as quickly as possible after entering the request.

[0166] In an exemplary embodiment of the present disclosure, in the i-th processing method, the batch sizes of the first Ni-1 steps are equal, the batch size of the Ni-th step is less than or equal to the batch size of the first Ni-1 steps, and the sum of the batch sizes of the Ni steps is equal to the number of tokens of requests to be processed.

[0167] Before using the model, the throughput (e.g., bits per second) under various batch sizes can be tested for a specific model, a specific test scenario, and a specific parallel mode. The batch size corresponding to the highest throughput is taken as the final result, and the batch size is fixed as the configuration under this test scenario.

[0168] For batch size x whose absolute time multiplier is less than threshold z, we can set:

[0169]

[0170] Where m is the number of tokens of pending requests, Ni is the number of steps in the i-th processing method, and m%x is the remainder of m divided by x. For example, m=34567, and x=2048, then Ni=17. The batch size of each of the 16 batches from step 0 to step 15 is 2048, and the batch size of the last step, step 16, is 1799.

[0171] The threshold z is calculated as follows:

[0172] Let x0 be the batch size with the smallest absolute time multiplier (4096 in the above example), and its corresponding time multiplier be l(x0). Then set the threshold z to:

[0173]

[0174] In formula (18), is the number of steps corresponding to the batch size x0, that is, Ni mentioned above, which is n+1 in formula (2). Formula (18) represents the upper bound of the ratio of the actual total time to the execution time of the batch size x0. The value 2 is because the average execution time of each token in the terminal stage (i.e., stage n+1 to stage n+pp-1) is twice the average execution time of each token in all stages (i.e., stage 0 to stage n+pp-1).

[0175] If the absolute time multiplier of the batch size x is greater than the threshold z, it means that processing pending requests according to the batch size x, even if the bubble time is not counted and only the token execution time is counted, will be longer than the total time (including the token execution time and the bubble time) of running according to the batch size. Therefore, the batch size x should be limited by the threshold z, and then the token execution time of the i-th processing method should be limited to less than a certain range. This threshold z is also applicable to the processing methods given according to other ideas.

[0176] The threshold z is not set arbitrarily. The setting principle of the threshold z is introduced below.

[0177] If the total time (execution time + bubble time) to process the request according to batch size x_0 is greater than the execution time to process the request according to batch size x, then the total time corresponding to batch size x is obviously greater. It can be expressed as a formula:

[0178]

[0179] in T represents the number of steps required to complete the prefill of m tokens according to the batch size x, that is, m divided by x and rounded up. B is the bubble time. The execution time part of this formula applies formula (15), and the execution time of each step is:

[0180]

[0181] The total execution time is the sum of the execution time of each step, and the function f is linear. The sum of the arithmetic sequence is the midpoint multiplied by the number of terms, so the total execution time for a batch size of x is

[0182]

[0183] According to formula (16), in the solution with constant batch size, the bubble time is pp-1 times the time of the largest step, so the bubble time corresponding to the batch size x0 is:

[0184]

[0185] Since the slope is greater than 0, we have:

[0186]

[0187] Since the intercept b is greater than 0, we have:

[0188]

[0189] That is:

[0190]

[0191] Substituting into the left side of formula (19), we get an upper bound of the total prefill time corresponding to the batch size x0:

[0192]

[0193] If the execution time corresponding to batch size x is larger than the upper bound of the total time corresponding to batch size x0, then batch size x will definitely not have a very small real total time. The formula at this time is:

[0194]

[0195] In the case of long text, m is very large, with:

[0196]

[0197] Cancel these two terms on both sides at the same time, and then cancel the same terms on both sides again Simplified:

[0198]

[0199] The left part is defined as the threshold z (see formula (18)), thereby filtering out all batch sizes x whose absolute time ratio is greater than z.

[0200] In some embodiments, a "batch size-absolute time ratio" corresponding curve can be obtained according to the absolute time ratio corresponding to each preset batch size, and then the value range of the batch size x with an absolute time ratio less than z is determined, multiple x values ​​are given within the value range, and a processing method is formed for each x value, and finally the processing time is compared with the processing methods determined based on other ideas. By generating different processing methods based on different batch sizes x, the best processing method can be automatically found for different devices, number of cards, parallel processing methods, models, and even for each long text request.

[0201] It should be noted that the various processing methods of the embodiments of the present disclosure may be designed based on multiple ideas, and each idea corresponds to one or more processing methods (for example, the idea in the above example gives multiple processing methods based on multiple x values). Finally, the processing time of the multiple processing methods given by the multiple ideas is calculated separately, and the processing method with the shortest processing time is selected.

[0202] The various processing methods given by the first idea (all satisfying equation (17)) meet the equality condition of equation (16) that minimizes the bubble time. Since the batch size of the first Ni-1 steps is x, the token execution time increases linearly, satisfying the monotonic increase condition of the token execution time; if the number of tokens in the last step is sufficient, then the token execution time of the last step is greater than the token execution time of the second-to-last step, and the token execution time still meets the monotonic increase condition; if the number of tokens in the last step is insufficient, then the time of the last step is less than the time of the second-to-last step, and the token execution timeline of each step increases monotonically to the second-to-last step and then decreases monotonically, which also satisfies the above equality condition.

[0203] Although the second idea is just a small change to the first idea, the improvement effect is very significant.

[0204] In other exemplary embodiments of the present disclosure, in the i-th processing method, the batch sizes from the 2nd step to the Ni-th step are equal, the batch size of the 1st step is less than or equal to the batch size from the 2nd step to the Ni-th step, and the sum of the batch sizes of the Ni steps is equal to the number of tokens of requests to be processed.

[0205] For batch size x whose absolute time ratio is less than threshold z, let:

[0206]

[0207] Where m is the number of tokens of pending requests, Ni is the number of steps in the i-th processing method, and m%x is the remainder of m divided by x. For example, m=34567, and x=2048, then Ni=17. The batch size of the first step, i.e., step 0, is 1799. The batch size of each step in the 16 batches from step 1 to step 16 is 2048.

[0208] The second idea is to move m%x in the first idea from the last step to the first step. Looking back at formula (15), we have:

[0209]

[0210] Given that l(m%x)>l(x), we also know (>> is much greater than), according to the Cauchy inequality, the sum of a large number multiplied by a small number is less than a large number multiplied by a large number plus a small number multiplied by a small number. Therefore, let the steps with very little time consumption at the beginning run a small batch with a higher absolute time multiple, and let the steps with very large time consumption at the end run a large batch with a small absolute time multiple.

[0211] This processing method also satisfies the equality condition of equation (16) that minimizes the bubble time: since the first step only runs m%x tokens, m%x is less than or equal to x, the token execution time of the first step is less than the token execution time of the second step, the batch size of the subsequent steps is x, the token execution time increases linearly, and the token execution time of all steps increases monotonically, satisfying the above equality condition.

[0212] However, the processing method given by the second idea is not necessarily better than the processing method given by the first idea, because the second idea increases the bubble time. Under the first idea, the token interval of the step with the longest token execution time may be [m-(m%x)-x, m-(m%x)], while under the second idea, the token interval of the step with the longest token execution time is [mx, m]. The bubble time is pp-1 times the longest time, so the bubble time of the second idea is longer.

[0213] Multiple x can be given based on the threshold z, and multiple processing methods corresponding to different x values ​​can be obtained based on the second idea, i.e., formula (28). Finally, the processing time is compared with multiple processing methods given based on other ideas.

[0214] The third idea is a further improvement of the above two ideas. In the first two ideas, most steps have a preset batch size, leaving only one step with an uncertain batch size. The absolute time multiplier of the batch size of this step may be very small or very large. If it is evenly divided, so that the batch size of each step is similar, with a maximum difference of 1, the overall time may be significantly reduced when the absolute time multiplier of the batch size is very small.

[0215] In an exemplary embodiment, in the i-th processing method, the maximum difference between the batch sizes of each step is 1, the batch size of the j+1-th step is greater than or equal to the batch size of the j-th step, and the sum of the batch sizes of Ni steps is equal to the number of tokens of requests to be processed.

[0216] For example, when m=32799, according to the first idea, Ni=17 can be set, the batch size of steps 0 to 15 is 2048, and the batch size of step 16 is 31. However, the processing efficiency of batch size 31 is very, very low. Under the third idea, Ni=16 can be set, the batch size of steps 0 to 14 is 2050, and the batch size of step 15 is 2049. Compared with the first idea, the total number of steps is reduced by 1, but the latter not only has a high processing efficiency of the batch size of each step, but also reduces the startup time by one time, and the total processing time is shorter. Compared with the first and second ideas, the batch size of each step in the third idea is more balanced, and there will be no steps with very low processing efficiency.

[0217] Expressed in formula, for a given number of steps Ni, the batch size of each step in the i-th processing method satisfies:

[0218]

[0219] in, It means m divided by Ni is rounded down. Indicates rounding up. That is, m tokens are divided into Ni steps as evenly as possible. When it cannot be divided evenly, the remainder is evenly distributed to the following steps first, so that the difference in the number of tokens in each step is not greater than 1. First, since the processing efficiency is low when the batch size is small, the small batch is placed in the front and the large batch is placed in the back; second, since the token execution time of the token increases linearly, if the batch size only increases and does not decrease, it can ensure that the token execution time of each step is monotonically increasing, satisfying the equality condition of formula (16), and can minimize the bubble time.

[0220] In the third approach, different batch size values ​​can be obtained based on different Ni numbers, so that the batch size of the first n steps is x, and the batch size of the next Ni-n steps is x+1. Different Ni corresponds to different n and x. Find one or more x with the smallest absolute time multiplier and less than the aforementioned threshold z among multiple batch sizes x, form multiple processing methods, and compare the processing time with the processing methods given by other ideas at the end.

[0221] The fourth idea is more complicated. First, the batch size is increased to improve processing efficiency. Second, in order to limit the bubble time, the token execution time of each step is limited to not exceed the token execution time of the step with the longest original token execution time. When the batch size x is fixed, the step with the longest token execution time is generally one of the two last steps. Taking all factors into consideration, the batch size can be accurately calculated based on the token execution time of one of the two last steps, so that the token execution times of adjacent steps are equal (rather than the batch size of the third idea being equal).

[0222] In an exemplary embodiment, in the ith processing method, the batch size of the Nith step is the smallest, the token execution time of the jth step does not exceed the token execution time of the j+1th step, the batch size of the jth step is the sum of half the batch size of the j+1th step and the calculation batch size, the processing time corresponding to the calculation batch size is half the processing time of the j+1th step, and the sum of the batch sizes of the Ni steps is equal to the number of tokens of requests to be processed.

[0223] In order not to increase the bubble time, the batch size of the last step is fixed first. In an exemplary embodiment, the batch size of the last step can be set to the batch size of the stage with the longest token execution time calculated in the first three ideas, so that the bubble time of each processing method under the fourth idea does not exceed the processing methods of the first three ideas.

[0224] Next, starting from the last step, we can infer the batch size of the previous step from the batch size of the next step. Under the assumption that the time multiplier is the same, if the batch size of the next step is a ij , the token number interval is [e, e+a ij ), assuming that the batch size of the previous step is x, and the token number interval of the previous step is [ex, e). Combined with the token execution time function f obtained in the above embodiment, considering that the processing time of two adjacent steps is similar, it can be set:

[0225]

[0226] set up:

[0227]

[0228] The above formula is transformed into:

[0229]

[0230] Further simplified to:

[0231]

[0232] There exists a discriminant:

[0233]

[0234] When Δ>0, the solution of the two-variable linear equation is:

[0235]

[0236] Remove a non-conforming solution:

[0237]

[0238] In order to minimize the bubble time, the token execution time of the previous step cannot exceed the token execution time of the next step, so it can be rounded down.

[0239] When Δ<0, the number of remaining tokens is not enough to make the time reach the token execution time of the previous step, so just take the remaining tokens.

[0240] So the batch size of the j-1th step is:

[0241]

[0242] However, token execution time fluctuates. The time obtained by executing twice on the same machine with the same configuration and the same other conditions is different. Therefore, even if the token execution time estimated by the above formula is slightly lower than that of the next step, it cannot be guaranteed that this condition will be met during actual execution. Once this condition is not met, pp-1 times the time difference will be immediately added to the bubble time.

[0243] Therefore, in an exemplary embodiment, a linear combination of the batch size estimated above and the batch size of the next step can be taken. Fixed linear coefficient λ∈[0,1], for example λ=0.5, that is, half of a ij So that the time calculated in the previous step is equal to half the time of the next step, and then add half of a to the previous step. ij , that is, keep the token execution time of half of the previous step equal to the token execution time of half of the next step, and the number of tokens in the remaining token execution time is equal to the number of tokens (batch size) of half of the next step. Since the token execution time of each token increases linearly with the number of tokens processed, the time of the previous step will be less than the time of the next step. Selecting a suitable linear coefficient λ can make the time difference between adjacent steps cover the time fluctuation caused by the execution environment, and satisfy the equality condition of formula (16).

[0244] In addition, if the batch size of a certain step exceeds the maximum batch size processed when processing the first preheating request, it will trigger memory reallocation, causing a sharp increase in token execution time. Therefore, it is necessary to limit the batch size of each stage to be smaller than the maximum batch size corresponding to the first preheating request.

[0245] The final solution can be

[0246]

[0247] According to actual measurements, when λ=0.2, a processing method that is better than the previous three ideas can be obtained. Of course, the processing time calculated for different machine conditions, configurations, and the number of tokens requested is different. The value of λ can be adjusted to give multiple processing methods, and the final processing time can be compared with the multiple processing methods given by other ideas.

[0248] The above four ideas are only examples. In actual applications, multiple processing methods can be given based on the constraint condition that "the token execution time of each step first increases monotonically and then decreases monotonically", and the present disclosure does not make any special restrictions on this.

[0249] The method for calculating the processing time of different processing modes has been given in the above embodiment and will not be repeated here. Finally, the processing mode with the smallest processing time is set as the target processing mode, and the token to be processed is processed according to the target processing mode in step S5.

[0250] To summarize, the disclosed embodiment shortens the model processing time by using preheating requests, obtains the correspondence between the number of tokens and the processing performance data, provides multiple processing methods that satisfy the minimum bubble time condition for the requests to be processed, and finally calculates the processing time of each processing method including the token execution time and the bubble time. The requests to be processed are processed according to the processing method with the shortest processing time, which can effectively shorten the time for the model to process long text requests and improve the model efficiency.

[0251] Corresponding to the above method embodiments, the present disclosure also provides a data processing device that can be used to execute the above method embodiments.

[0252] Figure 7 is a block diagram of a data processing device in an exemplary embodiment of the present disclosure.

[0253] refer to Figure 7 , the data processing device 700 may include:

[0254] A preheating module 71, configured to receive and process a preheating request to obtain a corresponding relationship between the number of tokens and processing performance data;

[0255] A request pre-processing module 72, configured to respond to a request arrival message and determine the number of tokens of the request to be processed;

[0256] The processing mode generating module 73 is configured to determine a plurality of processing modes according to the number of tokens of the request to be processed and the processing performance data, wherein the i-th processing mode is to divide the tokens of the request to be processed into Ni steps for processing, the batch size corresponding to the j-th step in the i-th processing mode is aij, and the token execution time from the 1st step to the Ni-th step in the i-th processing mode satisfies the requirement of first monotonically increasing and then monotonically decreasing, and the batch size is the number of tokens corresponding to one batch, i≥1, Ni≥1, 0≤j≤Ni-1;

[0257] A processing mode evaluation module 74 is configured to calculate the processing time corresponding to each processing mode according to the processing performance data, and determine the processing mode with the smallest processing time as the target processing mode, wherein the processing time includes the token execution time and the bubble time of each step;

[0258] The request processing module 75 is configured to process the token of the request to be processed according to the target processing method.

[0259] In an exemplary embodiment of the present disclosure, receiving and processing a preheating request to obtain a corresponding relationship between the number of tokens and processing performance data includes:

[0260] Processing the first preheating request in multiple batches, wherein the batch size of each batch in the first preheating request is not greater than the batch size of the previous batch;

[0261] Processing the second preheating request in multiple batches, wherein the batch size of each batch in the second preheating request includes multiple preset batch sizes;

[0262] Recording the absolute execution time of each token in each batch in processing the second warm-up request, and a first corresponding relationship between the absolute execution time of each token and the batch size of the batch to which the token belongs;

[0263] Determine the absolute time multiplier corresponding to each preset batch size according to the first corresponding relationship, the absolute time multiplier is a multiple of the average token execution time of a token to the preset basic token execution time under the preset batch size, the preset basic token execution time corresponds to the minimum value of multiple preset batch sizes, and the absolute time multiplier is used to represent the processing efficiency corresponding to different batch sizes in the target model;

[0264] A token execution time function is determined according to the first corresponding relationship and the absolute time multiplier, and the token execution time function is used to represent the relative execution time of the target model for each token with a serial number without the influence of the batch size.

[0265] In an exemplary embodiment of the present disclosure, determining the absolute time multiplier corresponding to each preset batch size according to the first corresponding relationship includes:

[0266] Determine the first execution time of the first token corresponding to the end of the kth batch processing according to the absolute execution time of the tokens of the kth batch, determine the second execution time of the first token corresponding to the start of the k+1th batch processing according to the absolute execution time of the tokens of the k+1th batch, and determine the relative time multiplier corresponding to the batch size of the k+1th batch according to the ratio of the second execution time to the first execution time, the batch size of the kth batch is different from that of the k+1th batch, 1≤k≤M-1, M is the number of preset batch sizes;

[0267] Assume that the absolute time multiplier corresponding to the batch size of the first batch is 1, determine the absolute time multiplier corresponding to each batch size according to the relative time multiplier corresponding to each batch size, and obtain the absolute time multiplier corresponding to each preset batch size.

[0268] In an exemplary embodiment of the present disclosure, determining a token execution time function according to the first corresponding relationship and the absolute time multiplier includes:

[0269] Determine the relative execution time corresponding to each token according to the ratio of the absolute execution time of each token to the absolute time multiplier corresponding to the batch to which it belongs;

[0270] The token execution time function is formed by fitting regression according to the relative execution time of each token.

[0271] In an exemplary embodiment of the present disclosure, calculating the processing time corresponding to each of the processing modes according to the processing performance data includes:

[0272] Determine the absolute time multiplier corresponding to the batch size aij according to the absolute time multiplier corresponding to each preset batch size, where the batch size aij is the batch size of the jth step in the i-th processing method;

[0273] Determine the average relative execution time corresponding to each token in the j-th step according to the batch size of the first j steps in the i-th processing mode and the token execution time function;

[0274] Determine the token execution time of the j-th step according to the product of the average relative execution time corresponding to each token, the batch size aij of the j-th step, and the absolute time multiplier corresponding to the batch size aij;

[0275] The processing time corresponding to the ith processing method is determined according to the sum of the token execution times corresponding to each step in the ith processing method.

[0276] In an exemplary embodiment of the present disclosure, determining multiple processing methods according to the number of tokens of the request to be processed and the processing performance data includes:

[0277] In the i-th processing method, the batch sizes of the first Ni-1 steps are equal, the batch size of the Ni-th step is less than or equal to the batch size of the first Ni-1 steps, and the sum of the batch sizes of the Ni steps is equal to the number of tokens of the pending requests.

[0278] In an exemplary embodiment of the present disclosure, determining multiple processing methods according to the number of tokens of the request to be processed and the processing performance data includes:

[0279] In the i-th processing method, the batch sizes from the 2nd step to the Ni-th step are equal, the batch size of the 1st step is less than or equal to the batch size from the 2nd step to the Ni-th step, and the sum of the batch sizes of the Ni steps is equal to the number of tokens of the pending requests.

[0280] Since the functions of the device 700 have been described in detail in the corresponding method embodiments, the present disclosure will not elaborate on them here.

[0281] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0282] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0283] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0284] Refer to the following Figure 8 An electronic device 800 according to this embodiment of the present invention will be described. Figure 8 The electronic device 800 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0285] like Figure 8As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0286] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the method shown in the embodiment of the present disclosure.

[0287] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0288] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0289] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0290] The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0291] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0292] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0293] The program product for implementing the above method according to an embodiment of the present invention can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0294] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0295] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0296] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0297] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0298] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0299] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and concept of the present disclosure are indicated by the claims.

Claims

1. A data processing method, characterized in that: Executed by the target model, including: Receive and process the warm-up request to obtain the corresponding relationship between the number of tokens and processing performance data; In response to the request arrival message, determine the number of tokens for the pending request; Determine multiple processing methods according to the number of tokens of the pending request and the processing performance data, the i-th processing method is to divide the tokens of the pending request into Ni steps for processing, the batch size corresponding to the j-th step in the i-th processing method is aij, the token execution time from the 1st step to the Ni-th step in the i-th processing method satisfies the requirement of first monotonically increasing and then monotonically decreasing, the batch size is the number of tokens corresponding to one batch, i≥1, Ni≥1, 0≤j≤Ni-1; Calculate the processing time corresponding to each processing method according to the processing performance data, and determine the processing method with the smallest processing time as the target processing method, wherein the processing time includes the token execution time and bubble time of each step; The token of the request to be processed is processed according to the target processing mode.

2. The data processing method according to claim 1, characterized in that: Receive and process the warm-up request to obtain the corresponding relationship between the number of tokens and the processing performance data, including: Processing the first preheating request in multiple batches, wherein the batch size of each batch in the first preheating request is not greater than the batch size of the previous batch; Processing the second preheating request in multiple batches, wherein the batch size of each batch in the second preheating request includes multiple preset batch sizes; Recording the absolute execution time of each token in each batch in processing the second warm-up request, and a first corresponding relationship between the absolute execution time of each token and the batch size of the batch to which the token belongs; Determine the absolute time multiplier corresponding to each preset batch size according to the first corresponding relationship, the absolute time multiplier is a multiple of the average token execution time of a token to the preset basic token execution time under the preset batch size, the preset basic token execution time corresponds to the minimum value of multiple preset batch sizes, and the absolute time multiplier is used to represent the processing efficiency corresponding to different batch sizes in the target model; A token execution time function is determined according to the first corresponding relationship and the absolute time multiplier, and the token execution time function is used to represent the relative execution time of the target model for each token with a serial number without the influence of the batch size.

3. The data processing method according to claim 2, characterized in that: Determining the absolute time multiplier corresponding to each preset batch size according to the first corresponding relationship includes: Determine the first execution time of the first token corresponding to the end of the kth batch processing according to the absolute execution time of the tokens of the kth batch, determine the second execution time of the first token corresponding to the start of the k+1th batch processing according to the absolute execution time of the tokens of the k+1th batch, and determine the relative time multiplier corresponding to the batch size of the k+1th batch according to the ratio of the second execution time to the first execution time, the batch size of the kth batch is different from that of the k+1th batch, 1≤k≤M-1, M is the number of preset batch sizes; Assume that the absolute time multiplier corresponding to the batch size of the first batch is 1, determine the absolute time multiplier corresponding to each batch size according to the relative time multiplier corresponding to each batch size, and obtain the absolute time multiplier corresponding to each preset batch size.

4. The data processing method according to claim 2, characterized in that: Determining a token execution time function according to the first corresponding relationship and the absolute time multiplier includes: Determine the relative execution time corresponding to each token according to the ratio of the absolute execution time of each token to the absolute time multiplier corresponding to the batch to which it belongs; The token execution time function is formed by fitting regression according to the relative execution time of each token.

5. The data processing method according to claim 2, characterized in that: Calculating the processing time corresponding to each processing method according to the processing performance data includes: Determine the absolute time multiplier corresponding to the batch size aij according to the absolute time multiplier corresponding to each preset batch size, where the batch size aij is the batch size of the jth step in the i-th processing method; Determine the average relative execution time corresponding to each token in the j-th step according to the batch size of the first j steps in the i-th processing mode and the token execution time function; Determine the token execution time of the j-th step according to the product of the average relative execution time corresponding to each token, the batch size aij of the j-th step, and the absolute time multiplier corresponding to the batch size aij; The processing time corresponding to the ith processing method is determined according to the sum of the token execution times corresponding to each step in the ith processing method.

6. The data processing method according to claim 1, characterized in that: Determining multiple processing methods according to the number of tokens of the request to be processed and the processing performance data includes: In the i-th processing method, the batch sizes of the first Ni-1 steps are equal, the batch size of the Ni-th step is less than or equal to the batch size of the first Ni-1 steps, and the sum of the batch sizes of the Ni steps is equal to the number of tokens of the pending requests.

7. The data processing method according to claim 1, characterized in that: Determining multiple processing methods according to the number of tokens of the request to be processed and the processing performance data includes: In the i-th processing method, the batch sizes from the 2nd step to the Ni-th step are equal, the batch size of the 1st step is less than or equal to the batch size from the 2nd step to the Ni-th step, and the sum of the batch sizes of the Ni steps is equal to the number of tokens of the pending requests.

8. The data processing method according to claim 1, characterized in that: Determining multiple processing methods according to the number of tokens of the request to be processed and the processing performance data includes: In the i-th processing method, the maximum difference between the batch sizes of each step is 1, the batch size of the j+1-th step is greater than or equal to the batch size of the j-th step, and the sum of the batch sizes of Ni steps is equal to the number of tokens of the pending requests.

9. The data processing method according to claim 1, characterized in that: Determining multiple processing methods according to the number of tokens of the request to be processed and the processing performance data includes: In the i-th processing method, the batch size of the Ni-th step is the smallest, the token execution time of the j-th step does not exceed the token execution time of the j+1-th step, the batch size of the j-th step is the sum of half the batch size of the j+1-th step and the calculation batch size, the processing time corresponding to the calculation batch size is half the processing time of the j+1-th step, and the sum of the batch sizes of the Ni steps is equal to the number of tokens of the pending requests.

10. A data processing device, characterized in that: include: a preheating module, configured to receive and process a preheating request to obtain a corresponding relationship between the number of tokens and processing performance data; A request pre-processing module, configured to respond to a request arrival message and determine the number of tokens for a pending request; A processing method generation module is configured to determine multiple processing methods according to the number of tokens of the pending request and the processing performance data, wherein the i-th processing method is to divide the tokens of the pending request into Ni steps for processing, the batch size corresponding to the j-th step in the i-th processing method is aij, and the token execution time from the 1st step to the Ni-th step in the i-th processing method satisfies a monotonically increasing and then a monotonically decreasing trend, and the batch size is the number of tokens corresponding to a batch, i≥1, Ni≥1, 0≤j≤Ni-1; a processing method evaluation module, configured to calculate the processing time corresponding to each processing method according to the processing performance data, and determine the processing method with the smallest processing time as the target processing method, wherein the processing time includes the token execution time and the bubble time of each step; The request processing module is configured to process the token of the request to be processed according to the target processing method.

Images