A large model inference optimization method and device, a storage medium and an electronic device

Abstract

The specification provides a large model inference optimization method, device, storage medium and electronic equipment. In this method, the obtained batch data set can be divided into multiple micro-batch data sets, and each micro-batch data set contains the input data of part of the tasks to be executed. Then, the data in each micro-batch data set is submitted to the inference device together with the corresponding operator description information, so that the inference device can concurrently perform second type operator processing on another micro-batch data set while performing first type operator processing on one micro-batch data set. They cover each other, thereby reducing the waiting time of calculation or communication, thereby improving the efficiency of large model in executing inference tasks.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method, apparatus, storage medium and electronic device for large model inference optimization. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, the field of natural language processing has achieved remarkable results. Among them, the Transformer architecture based on the self-attention mechanism has become a hot topic in research and application due to its excellent long sequence modeling capabilities and outstanding performance on various tasks. Since its inception, the Transformer architecture has undergone continuous evolution and innovation, gradually giving rise to three major variants. Among them, closed-source models represented by the GPT series and open-source models represented by Llama have gradually shown a convergence trend in architectural design, both tending to adopt the Decoder-Only architecture. The Decoder-Only architecture, with its concise and efficient structure, provides good support for language generation tasks and has become one of the important development directions of current language model architectures.

[0003] However, due to the large scale of the large model, each inference requires the execution of the entire model inference process, which results in a high time delay when performing inference tasks through the large model, thus limiting the widespread use of the model in application scenarios with high real-time requirements.

[0004] Therefore, how to improve the efficiency of large models in performing inference tasks is an urgent problem to be solved. Summary of the Invention

[0005] In view of the above, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a large model inference optimization method is proposed, comprising:

[0007] Obtain a batch processing dataset, which contains input data corresponding to multiple tasks to be executed;

[0008] The batch processing dataset is divided into at least two micro-batch processing datasets;

[0009] Each micro-batch dataset contains input data and operator description information corresponding to each task to be executed, which are submitted to the inference device. This allows the inference device to execute a first type of operator to process the input data corresponding to the task to be executed in one micro-batch dataset, while concurrently executing a second type of operator to process the input data corresponding to the task to be executed in another micro-batch dataset. The first type of operator and the second type of operator require different hardware resources to execute.

[0010] According to a second aspect of one or more embodiments of this specification, a large model inference optimization apparatus is proposed, comprising:

[0011] The acquisition module is used to acquire a batch processing dataset, which contains input data corresponding to multiple tasks to be executed.

[0012] The segmentation module is used to segment the batch processing dataset into at least two micro-batch processing datasets;

[0013] The inference module is used to submit the input data and operator description information corresponding to each task to be executed in each micro-batch dataset to the inference device, so that the inference device can execute a first type of operator to process the input data corresponding to the task to be executed in one micro-batch dataset, and concurrently execute a second type of operator to process the input data corresponding to the task to be executed in another micro-batch dataset. The first type of operator and the second type of operator require different hardware resources when they are executed.

[0014] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the large model inference optimization method described above by executing the executable instructions.

[0015] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the large model inference optimization method described above.

[0016] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the large model inference optimization method described above.

[0017] As can be seen from the above embodiments, this specification obtains a batch dataset containing input data corresponding to multiple tasks to be executed, and divides the obtained batch dataset into at least two micro-batch datasets. Then, the input data and operator description information corresponding to each task to be executed contained in each micro-batch dataset are submitted to the inference device. This allows the inference device to execute a first type of operator to process the input data corresponding to the tasks to be executed in one micro-batch dataset, while concurrently executing a second type of operator to process the input data corresponding to the tasks to be executed in the other micro-batch dataset. The first type of operator and the second type of operator require different hardware resources to execute.

[0018] In this method, the batch dataset can be divided into at least two micro-batch datasets to execute concurrently the operators that require communication resources and the operators that require computational resources contained in the different micro-batch datasets. By masking each other, the utilization of hardware resources can be improved and the inference latency can be reduced. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a Decoder-Only model provided in an exemplary embodiment.

[0020] Figure 2 This is a schematic diagram of a tensor parallel process provided in an exemplary embodiment.

[0021] Figure 3 This is a schematic diagram of the execution process of a Transformer layer provided in an exemplary embodiment.

[0022] Figure 4 This is a flowchart illustrating a large model inference optimization method provided in an exemplary embodiment.

[0023] Figure 5 This is a schematic diagram illustrating the concurrent execution of multiple micro-batch dataset computation and communication processes provided in an exemplary embodiment.

[0024] Figure 6 This is a schematic diagram showing the breakdown of the computation process of a self-attention operator provided in an exemplary embodiment.

[0025] Figure 7A This is a schematic diagram of an execution queue provided in an exemplary embodiment.

[0026] Figure 7B This is a schematic diagram of an execution queue provided in another exemplary embodiment.

[0027] Figure 8 This is a schematic diagram of an event object addition method provided in an exemplary embodiment.

[0028] Figure 9This is a schematic diagram of a large model inference process provided in an exemplary embodiment.

[0029] Figure 10 This is a schematic structural diagram of a device provided in an exemplary embodiment.

[0030] Figure 11 This is a block diagram of a large model inference optimization apparatus provided in an exemplary embodiment. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0032] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0033] Currently, with the rapid development of artificial intelligence technology, large models using the Decoder-Only architecture have achieved remarkable results in fields such as natural language processing and are widely used in various tasks such as language generation, text summarization, and machine translation. However, these large models typically have billions or even tens of billions of parameters, which puts enormous pressure on the storage and computing resources of large models and poses a severe challenge to practical inference deployment and application.

[0034] The inference process of the Decoder-Only model can be divided into two typical processes: prefill and decoding, as detailed below. Figure 1 As shown.

[0035] Figure 1 This is a schematic diagram of a Decoder-Only model provided in an exemplary embodiment.

[0036] Combination Figure 1As can be seen, during the Prefill process, the Decoder-Only model first receives the complete input sequence. This input sequence can be natural language text, code snippets, or other data that needs to be processed, depending on the application scenario of the model. Then, the model encodes each token in the input sequence (i.e., the smallest semantic unit after the model has segmented the input text, which can be a word, subword (such as "un" + "happy"), character, or even punctuation mark) through an embedding layer and a self-attention layer (which can be a self-attention layer or a multi-head attention layer, etc.) and calculates its association weight with other tokens in the input sequence. This allows the model to understand the relationships and contextual information between the various tokens in the input sequence.

[0037] Following the self-attention computation, a feedforward network (FFN) is used to further extract and transform features from each token. After these computations, the model can generate the first output token based on the contextual information of the input sequence. This token is generated based on a comprehensive understanding of the entire input sequence and marks the beginning of the output sequence.

[0038] As mentioned above, FFN typically includes two layers of linear transformations and activation functions to increase the nonlinear expressive power of the model.

[0039] In order to avoid problems such as gradient vanishing and training instability during the training of large models, model residual connections and normalization layers (Add&Normalize layers) can be set between the multi-head self-attention mechanism layer and the feedforward neural network layer, as well as between the feedforward neural network and the output layer, to improve the convergence speed and generalization performance of the model, thereby improving the overall model performance.

[0040] Furthermore, during the Decode process, the model uses the token generated in the previous iteration as input each time. These tokens are sequentially fed into the model to generate the next new token. The input token for the first iteration is the token generated during the Prefill process described above.

[0041] To improve computational efficiency, the model utilizes the Key-Value (KV) history stored in the KV cache during previous calculations during the decoding process. This KV history contains the Key and Value information from the previously calculated self-attention mechanism, allowing the model to generate new tokens without recalculating the attention vector matrix of all historical tokens, thus saving computational resources and time.

[0042] To facilitate understanding, the following uses the input sequence "Once upon a time, there was a scholar" as an example to explain the above process in detail. When the above input sequence is input into the large model, during the Prefill process of the large model, it can calculate based on all the tokens contained in the input sequence (i.e., each unit in "Once upon a time, there was a scholar") to generate the first token of the output sequence, such as "he".

[0043] Then, in the subsequent decoding process, the large model can run the model reasoning process again based on the token generated in the previous round of calculation and the KV history stored in the previous calculation process to obtain a new token, such as "bitter", until the reasoning is completed and the output sequence "Once upon a time, there was a scholar who studied hard and hoped that one day he could pass the imperial examination".

[0044] In this process, to reduce inference latency of large models and improve system throughput, researchers and engineers have tried various acceleration methods. Among them, Tensor Parallelism (TP) technology can partition the model's weight matrix across multiple inference cards according to different dimensions. Each inference card only needs to store and compute a portion of the weights, effectively reducing the memory pressure on a single card and improving computational efficiency, thus accelerating the model inference process. This alleviates the resource and latency issues of large-scale model inference to some extent. Specifically, for example... Figure 2 As shown.

[0045] Figure 2 This is a schematic diagram of a tensor parallel process provided in an exemplary embodiment.

[0046] Combination Figure 2 As can be seen from process a, when accelerating the inference process of a large model through TP technology, the weight matrix of the feedforward neural network layer FFN (or multilayer perceptron MLP) in each Transformer layer of the large model can be divided into two parts according to different dimensions and calculated on two inference cards respectively. After the calculation is completed, the communication process g is executed through the AllReduce operator to integrate the calculation results of the two inference cards and achieve data synchronization. Figure 2The process b is to divide the weight matrix of the self-attention layer into two parts according to different dimensions and calculate it on two inference cards. It is similar to the process a above, and will not be described in detail here.

[0047] Combination Figure 1 and Figure 2 It can be seen that after the large model completes the computation of the self-attention layer and the feedforward neural network layer, there are still computation processes for the residual connections and the normalization layer. The normalization process requires calculating the complete hidden representation (hidden_state) of a token. This necessitates summarizing the fragmented computation results from multiple inference cards before performing the normalization calculation. The most common summarization process is through... Figure 2 The Allreduce operator in the process executes the communication process g to synchronize the fragmented computation results on each inference card to other inference cards for integration, thereby achieving data synchronization.

[0048] In other words, during the inference task of a large model, after each self-attention layer and feedforward neural network layer computation, an Allreduce communication must follow, so that 2*layers Allreduce communications are required in one inference process.

[0049] The Allreduce operator mentioned above is used to transfer and aggregate data between multiple inference cards. The latency can be calculated using the following formula:

[0050]

[0051] In the above formula, start latency is the startup latency, link latency is the link latency, N is the number of communication cards, M is the amount of communication data, and BW is the inter-card bandwidth. When the startup latency and link latency are small, the Allreduce latency is mainly constrained by the link latency when the data volume is small. However, when the data volume is large, the Allreduce performance is constrained by the inter-card bandwidth. This means that on machines without high-speed inter-card interconnection, such as L20 and L40s, the Allreduce operation will become a serious performance bottleneck, significantly affecting the inference efficiency of the model, specifically as follows: Figure 3 As shown.

[0052] Figure 3 This is a schematic diagram of the execution process of a Transformer layer provided in an exemplary embodiment.

[0053] Combination Figure 3It can be seen that during the execution of the Transformer layer in the large model, the Allreduce communication process accounts for more than 63% of the execution time, which significantly affects the inference efficiency of the model.

[0054] Based on this, this specification provides a large model inference optimization method. The technical solutions provided by the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0055] Figure 4 This is a flowchart illustrating a large model inference optimization method provided in an exemplary embodiment, including:

[0056] S400: Obtain a batch processing dataset, which contains input data corresponding to multiple tasks to be executed.

[0057] S402: Divide the batch dataset into at least two micro-batch datasets.

[0058] Combination Figure 3 As can be seen, in a large model, the inference process of a Transformer layer mainly includes: self-attention operator, Allreduce operator 1, feedforward neural network operator, and Allreduce operator 2. There is a strict sequential dependency between these operators. It can be understood that the next operator can only start executing after the previous operator has been completed.

[0059] Here, the self-attention operator can encompass the entire computation process of the self-attention layer, the residual connections after the self-attention layer, and the normalization layer, as described above. Similarly, the feedforward neural network operator can encompass the entire computation process of the feedforward neural network layer, the residual connections after the feedforward neural network layer, and the normalization layer, as described above. In this specification, the entire computation process of the self-attention layer, the residual connections after the self-attention layer, and the normalization layer, as well as the entire computation process of the feedforward neural network layer, and the residual connections after the feedforward neural network layer, can be considered as a single "operator." This allows for adaptation to different inference engines and avoids optimization coupling specific to certain architectures.

[0060] In the above, large models can adopt a Decoder-Only structure. The self-attention operator and feedforward neural network operator in the large model mainly require computational resources during execution, and have no demand for network bandwidth resources. Allreduce operator 1 and Allreduce operator 2 mainly require network bandwidth resources during execution, and have less demand for computational resources.

[0061] Based on this, the business platform can concurrently execute the self-attention operator and Allreduce operator 1, and the feedforward neural network operator and Allreduce operator 2, on the inference device during large model inference, to fully utilize different hardware resources and effectively improve hardware resource utilization. Since the hardware resource requirements of the self-attention operator and feedforward neural network operator differ from those of Allreduce operator 1 and Allreduce operator 2, concurrent execution will not result in intense hardware resource contention, thus effectively improving the efficiency of large model inference tasks.

[0062] In the above context, inference devices can refer to hardware devices used for inference tasks in artificial intelligence and deep learning. Examples include inference cards, training cards, and graphics processing units (GPUs).

[0063] Furthermore, for a single task to be executed, the inference device executes the operators in the following order: self-attention operator, Allreduce operator 1, feedforward neural network operator, and Allreduce operator 2, exhibiting a strict dependency relationship. However, in practical applications, the input data corresponding to multiple tasks to be executed can be grouped into a batch processing dataset for batch processing. The operators used to process the input data corresponding to different tasks within the batch processing dataset do not have the aforementioned dependency relationship.

[0064] Therefore, when the business platform executes operators that process the input data corresponding to the tasks to be executed through the inference device, it can simultaneously acquire input data corresponding to multiple tasks to be executed, forming a batch dataset. This batch dataset can then be divided into at least two micro-batch datasets. This allows the inference device to execute a self-attention operator or feedforward neural network operator that processes the input data corresponding to a specific task in one micro-batch dataset, while simultaneously executing an Allreduce operator 1 or Allreduce operator 2 that processes the input data corresponding to a specific task in the other micro-batch dataset. This allows the computation and communication processes to run in parallel, thus masking the time required for computation and communication, thereby improving the overall execution efficiency of the large model. Specifically, as follows... Figure 5 As shown.

[0065] Figure 5 This is a schematic diagram illustrating the concurrent execution of multiple micro-batch dataset computation and communication processes provided in an exemplary embodiment.

[0066] Combination Figure 5It can be seen that while the business platform executes the inference process for task n to be executed in one micro-batch dataset through the inference device, it concurrently executes the inference process for task m to be executed in another micro-batch dataset.

[0067] Specifically, when executing the self-attention operator An of task n, the Allreduce operator 2 of the previous task m-1, which is located before task m, is executed concurrently. Figure 4 Ar(m-1)2 in the context of executing Allreduce operator 1 of the task to be executed n, i.e. Figure 5 In the context of Arn1, the self-attention operator Am is executed concurrently for task m. While executing the feedforward neural network operator FFNn for task n, the Allreduce operator 1 for task m is executed concurrently. Figure 5 Arm1 in the context. When executing the Allreduce operator 2 of the task to be executed n, i.e. Figure 5 In Arn2, the feedforward neural network operator FFNm is executed concurrently for the task to be executed m, and so on.

[0068] The overall latency can be calculated using the following formula:

[0069]

[0070] In the above formula, This is the overall latency. It can be seen from this that the overall latency of the above process is the sum of the maximum values ​​of the latency of concurrently executed An and Ar(m-1)2, the maximum values ​​of the latency of Arn1 and Am, the maximum values ​​of the latency of FFNn and Arm1, and the maximum values ​​of the latency of Arn2 and FFNm.

[0071] In the original computation process, all operators of the tasks n and m to be executed are combined into a batch of data by the scheduler and executed sequentially. The overall latency can be calculated using the following formula:

[0072]

[0073] As can be seen from the above formula, the overall delay is the sum of the delays of An, Ar(m-1)2, Arn1, Am, FFNn, Arm1, Arn2, and FFNm.

[0074] This shows that when the first type of operator is executed by the inference device to process the input data corresponding to the task to be executed in a micro-batch dataset, the second type of operator is executed concurrently to process the input data corresponding to the task to be executed in another micro-batch dataset. This allows the execution time of the longer-timed operator in the first and second types of operators to mask the execution time of the other operator, thereby improving the efficiency of large models in performing inference tasks.

[0075] The above describes a method for the business platform to split a batch processing dataset into at least two micro-batch processing datasets. This can be achieved by splitting the batch processing dataset into at least two micro-batch processing datasets according to a preset splitting granularity. The splitting granularity here includes: the granularity of the task to be executed and the granularity of the input token. The following sections will provide detailed explanations of the methods for splitting the batch processing dataset according to these two splitting granularities.

[0076] If it is necessary to split the batch processing dataset according to the granularity of the tasks to be executed, the input data corresponding to each task can be used as a task unit. Then, based on the number of task units required in each micro-batch processing dataset, the batch processing dataset can be divided into at least two micro-batch processing datasets. This can be understood as treating the input data corresponding to each task as a task unit, and then allocating each task unit to a micro-batch processing dataset according to actual needs.

[0077] In addition, if it is necessary to split the batch processing dataset according to the granularity of input tokens, the input data corresponding to at least a portion of the tasks to be executed can be split based on the number of each input token contained in the input data corresponding to each task to be executed. The resulting sets of at least two input tokens are then used as task units (each set of input tokens contains at least one input token). Therefore, based on the required number of task units in each micro-batch processing dataset, the batch processing dataset can be split into at least two micro-batch processing datasets. This can be understood as using at least a portion of the input tokens contained in the input data corresponding to each task to be executed as task units, and then, according to actual needs, either the entire input data or a portion of the input data corresponding to each task to be executed can be allocated to each micro-batch processing dataset.

[0078] It should be noted that the task units contained in each of the micro-batch datasets mentioned above can be different. Of course, to maximize the efficiency of large models performing inference tasks, the number of task units contained in each of the micro-batch datasets can also be the same.

[0079] At this point, the business platform can select the target splitting granularity based on the number of tasks to be executed and the expected number of micro-batch datasets, provided that the number of input tokens in each task to be executed is the same. Based on the target splitting granularity, the batch dataset can be split into at least two micro-batch datasets.

[0080] Specifically, when the number of tasks to be executed is an integer multiple of the expected number of micro-batch datasets, the input data corresponding to each task to be executed can be evenly divided into each micro-batch dataset.

[0081] At this point, the business platform can select the granularity of the tasks to be executed as the target splitting granularity, provided that the number of tasks to be executed is an integer multiple of the expected number of micro-batch datasets. Based on the target splitting granularity, the input data corresponding to the tasks to be executed is taken as the task unit. Then, based on the number of task units required to be included in each micro-batch dataset, the batch dataset can be split into at least two micro-batch datasets.

[0082] The number of task units required in each micro-batch dataset can be determined based on the number of tasks to be executed and the desired number of micro-batch datasets. For example, if there are 10 tasks to be executed and 2 micro-batch datasets, then the number of task units required in each micro-batch dataset can be 5.

[0083] Furthermore, when the number of tasks to be executed is not an integer multiple of the expected number of micro-batch datasets, the business platform can select the input token granularity as the target segmentation granularity. Then, based on the target segmentation granularity and the number of each input token in the input data corresponding to each task to be executed, the platform can segment at least a portion of the input data corresponding to each task to be executed. The segmented input token set and the input data corresponding to the unsegmented tasks to be executed are both used as task units. Finally, based on the number of task units required to be included in each micro-batch dataset, the batch dataset is segmented into at least two micro-batch datasets.

[0084] The number of task units required in each of the micro-batch datasets mentioned above can be determined based on the sum of the input token set obtained from the splitting and the number of unsplit tasks to be executed, as well as the expected number of micro-batch datasets.

[0085] At this point, the business platform can further split at least a portion of the tasks to be executed according to the granularity of the input token.

[0086] For example, if there are 5 tasks to be executed and 2 micro-batch datasets, the input data of one task to be executed can be split into two sets of input tokens. Then, all the input tokens contained in the input data of each other task to be executed can be treated as a task unit, and each set of input tokens corresponding to the task to be executed after being split according to the granularity of input tokens can be treated as a task unit.

[0087] For example, if there are 3 tasks to be executed and 2 micro-batch datasets, then for each task, the input data of that task can be split into two sets of input tokens. Each set of input tokens can then be considered a task unit.

[0088] At this point, the business platform can determine the number of task units required in each micro-batch dataset based on each input token set, the number of unsegmented tasks to be executed, and the expected number of micro-batch datasets. Then, based on the number of task units required in each micro-batch dataset, the batch dataset can be divided into at least two micro-batch datasets.

[0089] In real-world applications, the number of input tokens in each task to be executed can vary. In this case, some tasks may require a much larger number of input tokens than others. This can lead to an imbalance in workload across different micro-batch datasets after tasks are distributed to different batch datasets based on their number, thus hindering further improvement in the efficiency of large models performing inference tasks.

[0090] Therefore, the business platform can also select the target splitting granularity based on the number of input tokens in each task to be executed and the expected number of micro-batch datasets, and split the batch dataset into at least two micro-batch datasets according to the target splitting granularity, when the number of input tokens in each task to be executed is different.

[0091] Specifically, the business platform can, when the difference between the number of input tokens in the input data corresponding to each task to be executed meets a preset difference condition, use the input token granularity as the target segmentation granularity, and determine at least a portion of the tasks to be executed as tasks to be segmented. Based on the target segmentation granularity and the number of input tokens in the input data corresponding to the tasks to be segmented, the platform segments the input data corresponding to the tasks to be segmented, and uses the segmented input token set and the input data of other tasks to be executed besides the tasks to be segmented as task units. Based on the number of task units required in each micro-batch dataset, the batch processing dataset is segmented into at least two micro-batch datasets.

[0092] The aforementioned difference conditions can be set according to actual needs. For example, if the range between the number of input tokens in the input data corresponding to each task exceeds a preset range threshold, it can be considered that the above difference conditions are met. In this case, the business platform can select the top N tasks with the highest number of input tokens in the input data corresponding to each task as the selected tasks to be split. Alternatively, it can select tasks from the task list where the number of input tokens in the input data exceeds a preset threshold as the selected tasks to be split.

[0093] For example, when the variance between the number of each input token in the input data corresponding to each task to be executed exceeds the preset variance threshold, it can be considered as satisfying the above difference condition. At this time, the business platform can select the top N tasks with the most input tokens in the corresponding input data from the tasks to be executed, based on the number of each input token in the input data corresponding to each task to be executed, as the selected tasks to be split.

[0094] It's important to note that for feedforward neural network operators and Allreduce operators in large models, there are no dependencies between multiple input tokens. In this case, splitting the input data of a single task into at least two sets of input tokens has no impact on the computation of feedforward neural network operators and Allreduce operators. However, for self-attention operators, for each input token, it's necessary to consider the correlation between that input token and other input tokens preceding it to calculate the attention weight corresponding to that input token. Figure 6 As shown.

[0095] Figure 6This is a schematic diagram showing the breakdown of the computation process of a self-attention operator provided in an exemplary embodiment.

[0096] Combination Figure 6 As can be seen, during the prefilling process of the large model and the subsequent decoding, when calculating the self-attention operator, the query vector matrix (Q matrix), key vector matrix (K matrix), and value vector matrix (V matrix) required for the self-attention operator calculation can be decomposed into two parts. Assuming the length of the input token sequence is n, and it is decomposed starting from position m, each vector in the vector matrix has a dimension of d. The first part after decomposition uses Q1[m, d], K1[m, d], and V1[m, d], while the second part uses Q2[nm, d], K2[n, d], and V2[n, d]. These two parts can then be divided into different micro-batch datasets for concurrent computation, resulting in A1[m, d] and A2[nm, d]. Afterward, the Allreduce operator can be used to merge A1[m, d] and A2[nm, d] to obtain the final A[n, d].

[0097] It should be noted that the input data corresponding to the task to be executed may include: input sequence data and attention metadata. The input sequence data includes: input identifier sequence input_ids and position identifier sequence position_ids. Each input identifier in the input identifier sequence is used to represent the vocabulary index sequence corresponding to each input token. Each position identifier in the position identifier sequence is used to represent the position of each input token in the entire input token sequence. The attention metadata is used to represent the position of at least a portion of the attention tensor matrix that the inference device needs to use when executing the attention operator in the entire attention tensor matrix. The attention tensor matrix here is the Q matrix, K matrix, and V matrix required for self-attention calculation as described above.

[0098] For example, if the input text corresponding to the task to be performed is "A Technology Co., Ltd. is a company focused on artificial intelligence", after word segmentation of the input text, the resulting input token sequence is ["A", "science", "technology", "have", "limited", "company", "is", "one", "family", "specialize", "focus", "in", "human", "engineering", "intelligence", "of", "company"], and the corresponding input identifier sequence input_ids is input_ids = [101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 109, 110,

[122] Here, each value in the input identifier sequence is a unique identifier corresponding to each input token in the vocabulary. The corresponding position identifier sequence position_ids is position_ids = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]. Here, each value in the position identifier sequence is the position of each input token in the entire input token sequence.

[0099] The attention metadata mentioned above may include: query_lens: the length of the input sequence used to generate the query vector for the task to be executed; query_start_loc: the starting position offset of the input sequence used to generate the query vector for the task to be executed within the entire token sequence; seq_lens: the length of the entire input token sequence for the task to be executed; and seq_start_loc: the starting position offset of the entire input token sequence for the task to be executed within the sequence formed by concatenating the input token sequences of all tasks to be executed included in the batch dataset.

[0100] As can be seen from the above, the two types of input sequence data are used to represent each input token contained in the input token sequence from different dimensions. By directly segmenting the two types of input sequence data, the input token sequence is segmented, thereby obtaining the segmented task units.

[0101] The aforementioned attention metadata is used to mark the position of the attention tensor required when executing the attention operator within the entire attention tensor. This allows the system to retrieve at least a portion of the attention tensor matrix from the KV cache when executing the large model self-attention operator, based on the attention metadata, as the segmented attention tensor matrix, and to perform the corresponding self-attention calculation.

[0102] Based on this, the segmentation position information corresponding to at least a portion of the tasks to be executed can be determined according to the target segmentation granularity and the number of each input token in the input data corresponding to each task to be executed. Then, the attention metadata corresponding to each input token set obtained by segmentation can be re-determined according to the segmentation position information and the original attention metadata corresponding to at least a portion of the tasks to be executed. For each input token set, the attention metadata corresponding to the input token set is used to mark the position of at least a portion of the attention tensor matrix that needs to be used when performing self-attention calculation on the input token set in the entire attention tensor matrix.

[0103] As can be seen from the above, the business platform can further segment the input sequence data and attention metadata in the input data corresponding to each task to be executed in the batch dataset according to the above method, so as to split the batch dataset into two micro-batch datasets, so that the inference device can process the task units contained in the two micro-batch datasets concurrently.

[0104] It should be noted that during the fill process of the large model and during each Decoder iteration, a new token is generated as the input token for the next iteration. Therefore, at the beginning of each iteration, the business platform can re-segment the batch dataset using the above method and submit the input data and operator description information corresponding to each task to be executed in each micro-batch dataset to the inference device for concurrent execution.

[0105] In this specification, the executing entity used to implement the large model inference optimization method can refer to a designated device set up in the business platform, such as a server, or a terminal device such as a desktop computer or a laptop computer. For ease of description, the following description will only use the control device as the executing entity to illustrate the large model inference optimization method provided in this specification.

[0106] S404: Submit the input data and operator description information corresponding to each task to be executed contained in each micro-batch dataset to the inference device, so that the inference device executes the first type of operator to process the input data corresponding to the task to be executed in one micro-batch dataset, and concurrently executes the second type of operator to process the input data corresponding to the task to be executed in another micro-batch dataset. The first type of operator and the second type of operator require different hardware resources when they are executed.

[0107] In this specification, the control device can submit the input data and operator description information corresponding to each task to be executed contained in each micro-batch dataset to the inference device, so that the inference device can execute the first type of operator to process the input data corresponding to the task to be executed in one micro-batch dataset, and concurrently execute the second type of operator to process the input data corresponding to the task to be executed in another micro-batch dataset.

[0108] The first type of operator and the second type of operator require different hardware resources to execute.

[0109] The operator description information corresponding to the task to be executed mentioned above can be the information required for calling the program code of the operator. Here, the operator can refer to the operator required to process the input data corresponding to the task to be executed.

[0110] It should be noted that when performing inference processes for two micro-batch datasets simultaneously on the same control device, operators within the same execution queue stream will be executed sequentially according to the order in which they were submitted to the execution queue. Operators in different execution queues can be executed concurrently in any relative order. Therefore, the control device can submit each task unit contained in each micro-batch dataset and the operator description information required to process each task unit to preset execution queues. This allows the inference device to alternately obtain the operator description information and task units required to execute different types of operators from each execution queue, thus concurrently executing different types of operators.

[0111] Specifically, the control device can submit each task unit contained in each micro-batch processing dataset and the operator description information required to process each task unit to the preset execution queues in two ways, as follows: Figure 7A , Figure 7B As shown, the following is combined Figure 7A , Figure 7B These two methods will be explained in detail.

[0112] Figure 7A This is a schematic diagram of an execution queue provided in an exemplary embodiment.

[0113] exist Figure 7AIn this process, the control device, for each micro-batch dataset, according to the order of the task units corresponding to the tasks to be executed within that micro-batch dataset, sequentially submits the operators required to process each task unit corresponding to the tasks within that micro-batch dataset to the execution queue corresponding to that micro-batch dataset. This allows the inference device to alternately retrieve the operator description information and task units required to execute different types of operators from each execution queue according to the order in which the operators are in the execution queue, thus concurrently executing different types of operators. Each execution queue here corresponds one-to-one with each micro-batch dataset.

[0114] Figure 7B This is a schematic diagram of an execution queue provided in another exemplary embodiment.

[0115] exist Figure 7B In this process, the control device, for each micro-batch dataset, according to the order of the task units corresponding to the tasks to be executed within that micro-batch dataset, sequentially submits the different types of operators required to process each task unit to different execution queues. This allows the inference device to alternately retrieve the operator description information and task units required to execute different types of operators from each execution queue according to their order in the execution queues, thus concurrently executing different types of operators. Each execution queue here corresponds one-to-one with each type of operator.

[0116] In practical applications, since different task units may have different sizes, and the time required for the inference device to execute operators for task units of different sizes is also different, the inference device may obtain operator description information and task units required to execute different types of operators from each execution queue according to the order of each operator in the execution queue, and execute different types of operators concurrently. This may lead to the simultaneous execution of the same type of operator in different execution queues, resulting in hardware resource contention and thus a decrease in the efficiency of large model inference tasks.

[0117] For example, if the inference device takes too long to execute the first type operator a obtained from execution queue A, the second type operator b obtained from execution queue B, which is being executed concurrently with the first type operator a, may have already been completed. If the inference device then obtains the first type operator c from execution queue B again, it will compete with the currently executing first type operator a for hardware resources.

[0118] Based on this, the control device can also adjust the order of each task unit in each micro-batch processing dataset according to the size of each task unit contained in the micro-batch processing dataset, to obtain an adjusted micro-batch processing dataset, and submit each task unit contained in each adjusted micro-batch processing dataset and the operator description information required to process each task unit to the inference device.

[0119] As can be seen from the above, the control device can adjust the order of each task unit in each micro-batch dataset before submitting each task unit and the operator description information required to process each task unit to the inference device. This ensures that the task units in the same position in each micro-batch dataset are of similar size, thereby avoiding the aforementioned hardware resource contention problem.

[0120] For example, if the size of task unit a in micro-batch dataset A is 500, the size of task unit b is 200, and the size of task unit c is 400, and the size of task unit d in micro-batch dataset B is 400, the size of task unit e is 200, and the size of task unit f is 500, then the control device can adjust the order of the task units in micro-batch dataset A to: task unit b, task unit c, task unit a, based on the size of each task unit in micro-batch dataset A. Similarly, it can adjust the order of the task units in micro-batch dataset B to: task unit e, task unit d, task unit f, based on the size of each task unit in micro-batch dataset B, to avoid the aforementioned hardware resource contention problem.

[0121] In addition, the control device can also add event objects required for the execution of operators that process each task unit when submitting each task unit contained in each micro-batch dataset and the operator description information required to process each task unit to the preset execution queues, so that the inference device can alternately obtain the operator description information and task units required to execute different types of operators from each execution queue, so as to execute different types of operators concurrently.

[0122] The event objects described above are used to record the execution status of operators and adjust the execution timing of operators in different execution queues. For example, the `event.record` event object is used to record an event at a specified position in the current execution queue. This event will be marked as operator completion when the execution queue reaches that position. The `event.wait` event is used to make subsequent operators wait for the specified event to complete before continuing execution. For easier understanding, the following uses... Figure 8The process of fine-grained control over each execution queue through event objects is explained in detail above.

[0123] Figure 8 This is a schematic diagram of an event object addition method provided in an exemplary embodiment.

[0124] exist Figure 8 Each solid arrow represents an `event.record` event object, and each dashed arrow represents an `event.wait` event object. (Combined...) Figure 8 As can be seen, the control device can add an event.record event object after each operator to record the completion of each operator after its execution, and can add an event.wait event object before each operator to control the operator to wait for the completion of other operators in the execution queue that are executed concurrently with the operator above it before the operator can be executed.

[0125] For example: Figure 8 As shown, when the An operator in one execution queue completes its execution, it triggers an event.record object to mark the completion of the An operator's execution. At this time, due to the event.wait restriction added before the Arn1 operator in another execution queue, the Arn1 operator in the other execution queue only begins execution after the An operator has completed its execution. Furthermore, since operators in the same execution queue are executed strictly in the order they appear in the queue, when the An operator completes its execution, the inference device can continue to retrieve the Am operator from this execution queue and execute it concurrently with the Arn1 operator, which has only started execution due to the event.wait restriction, and so on.

[0126] To facilitate understanding, the following provides a detailed explanation of the large model's inference process during the execution of each task in the batch dataset using the large model inference optimization method described above. Specifically, as follows... Figure 9 As shown.

[0127] Figure 9 This is a schematic diagram of a large model inference process provided in an exemplary embodiment.

[0128] Combination Figure 9 It can be seen that after the control device obtains the batch dataset, it can divide the batch dataset into two micro-batch datasets. Then, by submitting the task units contained in each micro-batch dataset to the corresponding execution queue in sequence, the inference device can alternately obtain different types of operators from different execution queues and execute them concurrently, thereby improving the efficiency of large model inference tasks.

[0129] It should be noted that the aforementioned model inference optimization method splits the batch dataset input into the large model into two microprocessor datasets and submits them to two execution queues, so that the tasks to be executed in these two microprocessor datasets can be executed concurrently by a single inference device. The tensor parallel method, on the other hand, splits the parameters of the large model (i.e., the weight parameters of the self-attention layer and the weight parameters of the feedforward neural network layer) into multiple groups, and assigns each group of parameters to a different inference device for computation. In other words, the aforementioned model inference optimization method and the tensor parallel method decompose the computational operations during the execution of the inference task of the large model from two different dimensions (i.e., splitting the input data of the large model and the parameters of the large model separately).

[0130] Therefore, the control device can also split the parameters of the large model into different parameter sets and send each parameter set to a different inference device. Then, for each inference device, the input data and operator description information corresponding to each task to be executed contained in each micro-batch dataset can be submitted to the inference device. This allows the inference device to execute the first type of operator to process the input data corresponding to the task to be executed in one micro-batch dataset, while simultaneously executing the second type of operator to process the input data corresponding to the task to be executed in another micro-batch dataset. This yields the sub-execution result corresponding to each task to be executed returned by the inference device. Then, based on the sub-execution result corresponding to each task to be executed returned by each inference device, the execution result corresponding to each task to be executed can be obtained.

[0131] It should be noted that the tasks to be performed mentioned above can be determined according to actual needs, such as image processing, text generation, etc., and this specification does not impose any restrictions here. Specifically, if the task to be performed is an image processing task, then the input data corresponding to the task can be the image data to be processed. If the task to be performed is a text generation task, then the input data corresponding to the task can be the text data of the problem to be processed.

[0132] As can be seen from the above, the control device can divide the batch dataset into at least two micro-batch datasets to enable concurrent communication and computation between the micro-batch datasets. This mutual masking improves hardware resource utilization and reduces inference latency. Furthermore, based on model characteristics, it supports balanced partitioning of micro-batch datasets at the task-to-execute granularity and input token granularity. A specially designed balanced partitioning algorithm ensures load balancing among the micro-batch datasets, further improving resource utilization efficiency. More importantly, this method allows control only of the communication and computation processes of the large model without modifying the original large model structure. This maintains good compatibility with various large models based on the transformer architecture, improving inference efficiency and reducing latency while possessing broad applicability to multiple model architectures.

[0133] Figure 10 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 10 At the hardware level, the device includes a processor 1002, an internal bus 1004, a network interface 1006, memory 1008, and non-volatile memory 1010, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 1002 reads the corresponding computer program from the non-volatile memory 1010 into memory 1008 and then runs it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0134] Please refer to Figure 11 Large model inference optimization devices can be applied to, for example... Figure 11 The device shown is used to implement the technical solution described in this specification. The large model inference optimization device may include:

[0135] The acquisition module 1101 is used to acquire a batch processing dataset, which contains input data corresponding to multiple tasks to be executed;

[0136] The segmentation module 1102 is used to segment the batch processing dataset into at least two micro-batch processing datasets;

[0137] The inference module 1103 is used to submit the input data and operator description information corresponding to each task to be executed contained in each micro-batch dataset to the inference device, so that when the inference device executes a first type of operator to process the input data corresponding to the task to be executed in one micro-batch dataset, it concurrently executes a second type of operator to process the input data corresponding to the task to be executed in another micro-batch dataset. The first type of operator and the second type of operator require different hardware resources to execute.

[0138] Optionally, the segmentation module 1102 is specifically used to segment the batch processing dataset into at least two micro-batch processing datasets according to a preset segmentation granularity; the segmentation granularity includes: the granularity of the task to be executed and the granularity of the input token.

[0139] Optionally, each micro-batch dataset contains the same number of task units; each task unit contains at least a portion of the input data corresponding to the task to be executed.

[0140] Optionally, the segmentation module 1102 is specifically configured to, when the number of each input token in the input data corresponding to each task to be executed is the same, select a target segmentation granularity based on the number of tasks to be executed and the expected number of micro-batch datasets, and segment the batch dataset into at least two micro-batch datasets based on the target segmentation granularity; when the number of each input token in the input data corresponding to each task to be executed is different, select a target segmentation granularity based on the number of each input token in the input data corresponding to each task to be executed and the expected number of micro-batch datasets, and segment the batch dataset into at least two micro-batch datasets based on the target segmentation granularity.

[0141] Optionally, the segmentation module 1102 is specifically used to: select the granularity of the tasks to be executed as the target segmentation granularity when the number of tasks to be executed is an integer multiple of the expected number of micro-batch datasets; take the input data corresponding to the tasks to be executed as task units according to the target segmentation granularity; and segment the batch dataset into at least two micro-batch datasets according to the number of task units required to be included in each micro-batch dataset.

[0142] Optionally, the segmentation module 1102 is specifically configured to: select the input token granularity as the target segmentation granularity when the number of tasks to be executed is not an integer multiple of the expected number of micro-batch datasets; segment the input data corresponding to at least a portion of the tasks to be executed according to the target segmentation granularity and the number of each input token in the input data corresponding to each task to be executed, and use the segmented input token set as a task unit; and segment the batch dataset into at least two micro-batch datasets according to the number of task units required to be included in each micro-batch dataset.

[0143] Optionally, the segmentation module 1102 is specifically configured to, when the difference between the number of each input token in the input data corresponding to each task to be executed meets a preset difference condition, use the input token granularity as the target segmentation granularity, and determine at least a portion of the tasks to be executed as tasks to be segmented from each task to be executed; segment the input data corresponding to the task to be segmented according to the target segmentation granularity and the number of each input token in the input data corresponding to the task to be segmented, and use the segmented input token set and the input data of other tasks to be executed besides the task to be segmented as task units; and segment the batch processing dataset into at least two micro-batch processing datasets according to the number of task units required to be included in each micro-batch processing dataset.

[0144] Optionally, the input data includes: input sequence data and attention metadata;

[0145] The segmentation module 1102 is specifically used to segment at least a portion of the input sequence data corresponding to each task to be executed, based on the target segmentation granularity and the number of each input token in the input data corresponding to each task to be executed, and to use the segmented set of input tokens as a task unit; and

[0146] Based on the target segmentation granularity and the number of each input token in the input data corresponding to each task to be executed, determine the segmentation position information corresponding to at least a portion of the tasks to be executed.

[0147] Based on the segmentation location information and the original attention metadata corresponding to at least part of the tasks to be executed, the attention metadata corresponding to each input token set obtained by the segmentation is re-determined.

[0148] Optionally, the inference module 1103 is specifically used to submit each task unit contained in each micro-batch dataset and the operator description information required to process each task unit to the inference device.

[0149] Optionally, the inference module 1103 is specifically used to adjust the order of each task unit in the micro-batch processing dataset according to the size of each task unit contained in the micro-batch processing dataset, so as to obtain an adjusted micro-batch processing dataset.

[0150] Each adjusted micro-batch dataset contains a description of each task unit and the operator descriptions required to process each task unit, which is then submitted to the inference device.

[0151] Optionally, the inference module 1103 is specifically used to submit each task unit contained in each micro-batch dataset and the operator description information required to process each task unit to preset execution queues, so that the inference device alternately obtains the operator description information and task units required to execute different types of operators from each execution queue, so as to concurrently execute different types of operators.

[0152] Optionally, the inference module 1103 is specifically used to submit each task unit contained in each micro-batch processing dataset and the operator description information required to process each task unit to preset execution queues, and add event objects required when the operator to process each task unit is executed, so that the inference device alternately obtains the operator description information and task units required to execute different types of operators from each execution queue, so as to execute different types of operators concurrently; the event objects are used to record the execution status of operators and adjust the execution timing of operators in different execution queues.

[0153] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0154] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0155] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

Claims

1. A large-scale model inference optimization method, comprising: Obtain a batch processing dataset, which contains input data corresponding to multiple tasks to be executed; Identify at least a portion of the tasks to be executed as tasks to be split; Based on the number of each input token in the input data corresponding to the task to be segmented, the input data corresponding to the task to be segmented is segmented, and the segmented input token set and the input data of other tasks to be executed besides the task to be segmented are used as task units. The batch processing dataset is divided into at least two micro-batch processing datasets based on the number of task units required to be included in each micro-batch processing dataset; Each micro-batch dataset contains input data and operator description information corresponding to each task to be executed, which are submitted to the inference device. This allows the inference device to execute a first type of operator to process the input data corresponding to the task to be executed in one micro-batch dataset, while concurrently executing a second type of operator to process the input data corresponding to the task to be executed in another micro-batch dataset. The first type of operator and the second type of operator require different hardware resources to execute.

2. The method as described in claim 1, wherein the batch processing dataset is divided into at least two micro-batch processing datasets, specifically including: According to the preset granularity of the segmentation, the batch processing dataset is divided into at least two micro-batch processing datasets; The segmentation granularity includes: the granularity of the task to be executed and the granularity of the input token.

3. The method as described in claim 2, wherein each micro-batch dataset contains the same number of task units; the task unit contains at least a portion of the input data corresponding to the task to be executed.

4. The method as described in claim 3, wherein the batch processing dataset is divided into at least two micro-batch processing datasets according to a preset segmentation granularity, specifically including: If the number of each input token in the input data corresponding to each task to be executed is the same, select a target splitting granularity according to the number of tasks to be executed and the expected number of micro-batch datasets, and split the batch dataset into at least two micro-batch datasets according to the target splitting granularity. When the number of each input token in the input data corresponding to each task to be executed is different, a target splitting granularity is selected based on the number of each input token in the input data corresponding to each task to be executed and the expected number of micro-batch datasets, and the batch dataset is split into at least two micro-batch datasets based on the target splitting granularity.

5. The method as described in claim 4, wherein a target segmentation granularity is selected based on the number of tasks to be executed and the expected number of micro-batch processing datasets, and the batch processing dataset is segmented into at least two micro-batch processing datasets based on the target segmentation granularity, specifically including: When the number of tasks to be executed is an integer multiple of the expected number of micro-batch datasets, the granularity of the tasks to be executed is selected as the target splitting granularity. Based on the target segmentation granularity, the input data corresponding to the task to be executed is taken as a task unit; The batch dataset is divided into at least two micro-batch datasets based on the number of task units required to be included in each micro-batch dataset.

6. The method as described in claim 4, wherein a target segmentation granularity is selected based on the number of tasks to be executed and the expected number of micro-batch datasets, and the batch dataset is segmented into at least two micro-batch datasets based on the target segmentation granularity, specifically including: If the number of tasks to be executed is not an integer multiple of the expected number of micro-batch datasets, the input token granularity is selected as the target splitting granularity. Based on the target segmentation granularity and the number of each input token in the input data corresponding to each task to be executed, the input data corresponding to at least a portion of the tasks to be executed is segmented, and the segmented set of input tokens is used as a task unit. The batch dataset is divided into at least two micro-batch datasets based on the number of task units required to be included in each micro-batch dataset.

7. The method of claim 6, wherein the input data includes: Input sequence data and attention metadata; Based on the target segmentation granularity and the number of each input token in the input data corresponding to each task to be executed, at least a portion of the input data corresponding to each task to be executed is segmented, and the segmented set of input tokens is used as a task unit, specifically including: Based on the target segmentation granularity and the number of each input token in the input data corresponding to each task to be executed, the input sequence data corresponding to at least a portion of the tasks to be executed is segmented, and the resulting set of input tokens is used as a task unit; and Based on the target segmentation granularity and the number of each input token in the input data corresponding to each task to be executed, determine the segmentation position information corresponding to at least a portion of the tasks to be executed. Based on the segmentation location information and the original attention metadata corresponding to at least part of the tasks to be executed, the attention metadata corresponding to each input token set obtained by the segmentation is re-determined.

8. The method as described in any one of claims 5-6, wherein the input data and operator description information corresponding to each task to be executed contained in each micro-batch dataset are submitted to the inference device, specifically comprising: Each micro-batch dataset contains a task unit and the operator description information required to process each task unit is submitted to the inference device.

9. The method of claim 8, wherein each task unit contained in each micro-batch processing dataset and the operator description information required to process each task unit are submitted to the inference device, specifically comprising: For each micro-batch dataset, the order of each task unit in the micro-batch dataset is adjusted according to the size of each task unit in the micro-batch dataset to obtain the adjusted micro-batch dataset. Each adjusted micro-batch dataset contains a description of each task unit and the operator descriptions required to process each task unit, which is then submitted to the inference device.

10. The method of claim 8, wherein submitting each task unit contained in each micro-batch dataset and the operator description information required to process each task unit to the inference device, specifically includes: Each task unit contained in each micro-batch dataset and the operator description information required to process each task unit are submitted to preset execution queues, so that the inference device alternately obtains the operator description information and task units required to execute different types of operators from each execution queue, so as to execute different types of operators concurrently.

11. The method of claim 8, wherein each task unit contained in each micro-batch processing dataset and the operator description information required to process each task unit are submitted to preset execution queues, so that the inference device alternately obtains the operator description information and task units required to execute different types of operators from each execution queue, so as to concurrently execute different types of operators, specifically including: Each task unit contained in each micro-batch processing dataset and the operator description information required to process each task unit are submitted to preset execution queues. Event objects required for the execution of the operator to process each task unit are added, so that the inference device alternately obtains the operator description information and task units required to execute different types of operators from each execution queue, so as to execute different types of operators concurrently. The event objects are used to record the execution status of the operators and adjust the execution timing of the operators in different execution queues.

12. A large model inference optimization device, comprising: The acquisition module is used to acquire a batch processing dataset, which contains input data corresponding to multiple tasks to be executed. The segmentation module is used to identify at least a portion of the tasks to be executed as tasks to be segmented from the tasks to be executed. Based on the number of each input token in the input data corresponding to the task to be segmented, the input data corresponding to the task to be segmented is segmented, and the segmented input token set and the input data of other tasks to be executed besides the task to be segmented are used as task units. The batch processing dataset is divided into at least two micro-batch processing datasets based on the number of task units required to be included in each micro-batch processing dataset; The inference module is used to submit the input data and operator description information corresponding to each task to be executed in each micro-batch dataset to the inference device, so that the inference device can execute a first type of operator to process the input data corresponding to the task to be executed in one micro-batch dataset, and concurrently execute a second type of operator to process the input data corresponding to the task to be executed in another micro-batch dataset. The first type of operator and the second type of operator require different hardware resources when they are executed.

13. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-11 by executing the executable instructions.

14. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-11.

Images