Task scheduling method and device for large language model, equipment and storage medium

Abstract

The invention discloses a task scheduling method and device for a large language model, equipment and a storage medium, and the method comprises the steps: carrying out the tensor segmentation of a to-be-segmented tensor in the large language model, so as to obtain a CC tensor, a CG tensor and a GG tensor; and scheduling hardware resources corresponding to the CC tensor, the CG tensor and the GG tensor to cooperatively work so as to control the large language model to execute tasks. According to the method, the tensor is divided into the CC tensor, the CG tensor and the GG tensor which use different hardware resources, and then all the hardware resources work cooperatively to make full use of available computing, memory and communication resources, so that the reasoning efficiency of a small computing system can be improved under the condition that the reasoning accuracy of a large language model is ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and particularly to a task scheduling method, device, equipment and storage medium for large language models. Background Art

[0002] Generative large language models (LLMs) are well-known for their excellent capabilities in many artificial intelligence applications. Due to their large model sizes, these models require a great deal of computing and memory, so they are mainly deployed in data centers equipped with high-end GPUs (such as Nvidia Tesla H100) to provide low-latency and high-throughput services. Recently, there has been an emerging trend to run LLMs on more accessible local platforms, such as edge devices and personal computers (PCs) with medium GPUs (such as Nvidia RTX 3090). This shift is due to the need to improve data privacy, model customization, and reduce inference costs. Deploying LLMs on medium GPUs is a challenge because it requires making the model compatible with these medium GPU systems. In addition, it is necessary to optimize its inference latency to ensure that it can effectively handle real-time query processing.

[0003] To solve the above problems, currently, compression methods such as quantization, distillation, and pruning are generally used to reduce the model size so that the compressed model can be fully loaded into the GPU memory. However, even highly compressed models may exceed the memory capacity of medium GPUs, especially in sparse mixture-of-experts models. For example, loading the Mixtral-8x22B MoE model with 4-bit precision requires approximately 110GB of video memory to store its parameters, exceeding the memory capacity of many medium GPUs (such as Nvidia RTX2080 / 3090 / 4090), whose memory does not exceed 24GB.

[0004] Therefore, the existing technology still needs to be improved. Summary of the Invention

[0005] The technical problem to be solved by the present application is to provide a task scheduling method, device, equipment and storage medium for large language models in view of the deficiencies of the prior art.

[0006] To solve the above technical problems, in the first aspect of the present application, a task scheduling method for a large language model is provided. The task scheduling method for the large language model is applied to a small computing system with a single graphics card. The task scheduling method for the large language model specifically includes:

[0007] Perform tensor slicing on the tensor to be sliced in the large language model to obtain a CC tensor, a CG tensor, and a GG tensor, where the CC tensor is stored and executed on the CPU side, the CG tensor is stored on the CPU side and executed on the GPU side, and the GG tensor is stored and executed on the GPU side;

[0008] Schedule the respective hardware resources of the CC tensor, the CG tensor, and the GG tensor to work together to control the large language model to execute tasks.

[0009] The task scheduling method for the large language model, where the performing tensor slicing on the tensor to be sliced in the large language model to obtain a CC tensor, a CG tensor, and a GG tensor specifically includes:

[0010] Obtain the hardware configuration information of the running device for running the large language model;

[0011] Based on the hardware configuration information, determine the tensor slicing rate of the large language model, and perform tensor slicing on the tensor to be sliced in the large language model according to the tensor slicing rate to obtain a CC tensor, a CG tensor, and a GG tensor.

[0012] The task scheduling method for the large language model, where the determining the tensor slicing rate of the large language model based on the hardware configuration information specifically includes:

[0013] Select a fixed GG slicing rate based on the hardware configuration information, and determine the CG slicing rate based on the fixed GG slicing rate;

[0014] Within the preset GG slicing rate search space, based on the time cost, search for the optimal GG slicing rate based on the CG slicing rate, and determine the CC slicing rate based on the optimal GG slicing rate and the CG slicing rate to obtain the tensor slicing rate of the large language model.

[0015] The task scheduling method for the large language model, where after performing tensor slicing on the tensor in the large language model to obtain a CC tensor, a CG tensor, and a GG tensor, the method further includes:

[0016] Determine a CG' tensor based on the CC tensor, where the CG' tensor represents the same tensor data as the CC tensor in terms of storage, and the CG' tensor will be executed on the GPU side;

[0017] Use the CC tensor, the CG tensor, and the GG tensor as the tensor slicing in the generation stage of the large language model for the tensor to be sliced;

[0018] Use the CC tensor, the CG' tensor, the CG tensor, and the GG tensor as the tensor slicing in the prompting stage of the large language model for the tensor to be sliced.

[0019] The task scheduling method of the large language model, wherein scheduling the hardware resources corresponding to the CC tensor, the CG tensor, and the GG tensor to work together to control the large language model to execute tasks specifically includes:

[0020] When the large language model is in the prompting stage, scheduling the hardware resources corresponding to the CC tensor, the CG' tensor, the CG tensor, and the GG tensor to work together to control the prompting stage of the large language model to execute tasks;

[0021] When the large language model is in the generation stage, scheduling the hardware resources corresponding to the CC tensor, the CG tensor, and the GG tensor to work together to control the generation stage of the large language model to execute tasks.

[0022] The task scheduling method of the large language model, wherein scheduling the hardware resources corresponding to the CC tensor, the CG tensor, and the GG tensor to work together specifically includes:

[0023] Set four asynchronous execution streams, where the four asynchronous execution streams include a CPU computing stream, a GPU kernel startup stream, a CPU-GPU communication stream, and a GPU computing stream;

[0024] Schedule the CPU computing stream to execute the CC tensor, schedule the GPU kernel startup stream and the GPU computing stream to execute the GG tensor, and schedule the GPU kernel startup stream, the CPU-GPU communication stream, and the GPU computing stream to execute the CG tensor, so as to realize the coordinated work of the hardware resources corresponding to the CC tensor, the CG tensor, and the GG tensor.

[0025] The task scheduling method of the large language model, wherein scheduling the hardware resources corresponding to the CC tensor, the CC' tensor, the CG tensor, and the GG tensor to work together specifically includes:

[0026] Schedule the GPU kernel startup stream and the GPU computing stream to execute the GG tensor, and schedule the GPU kernel startup stream, the CPU-GPU communication stream, and the GPU computing stream to execute the CG tensor and the CG' tensor, so as to realize the coordinated work of the hardware resources corresponding to the CC tensor, the CG tensor, and the GG tensor.

[0027] The second aspect of the present application provides the task scheduling device of the large language model, wherein the task scheduling device of the large language model specifically includes:

[0028] A memory manager, configured to perform tensor splitting on the tensors to be split in the large language model to obtain a CC tensor, a CG tensor, and a GG tensor, wherein the CC tensor is stored and executed on the CPU side, the CG tensor is stored on the CPU side and executed on the GPU side, and the GG tensor is stored and executed on the GPU side;

[0029] A task scheduler for scheduling the hardware resources corresponding to the CC tensor, the CG tensor, and the GG tensor to work together to control a large language model to execute tasks.

[0030] The third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the task scheduling method of the large language model as described above.

[0031] The fourth aspect of this application provides a terminal device, which includes: a processor and a memory;

[0032] The memory stores a computer-readable program executable by the processor;

[0033] When the processor executes the computer-readable program, it implements the steps in the task scheduling method of the large language model as described above.

[0034] Beneficial effects: Compared with the prior art, this application provides a task scheduling method, device, equipment, and storage medium for a large language model. The method includes tensor slicing the tensors to be sliced in the large language model to obtain the CC tensor, the CG tensor, and the GG tensor; scheduling the hardware resources corresponding to the CC tensor, the CG tensor, and the GG tensor to work together to control the large language model to execute tasks. By slicing the tensors into the CC tensor, the CG tensor, and the GG tensor that utilize different hardware resources, and then having the various hardware resources work together, this application can make full use of the available computing, memory, and communication resources, thereby improving the inference efficiency of a small computing system while ensuring the inference accuracy of the large language model. Description of the Drawings

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of this application, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0036] Figure 1 It is a flowchart of the task scheduling method for the large language model provided by the embodiment of this application.

[0037] Figure 2 It is a flow example diagram of the scheduling process for a large language model including two connection layers.

[0038] Figure 3 It is an example diagram of the processing process of the weight tensor.

[0039] Figure 4 Schematic diagram of three situations in the process of determining the CG segmentation rate.

[0040] Figure 5 This is a block diagram of the principles of a task scheduling device for a large language model provided in an embodiment of the present application.

[0041] Figure 6 This is a block diagram of the principles of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The present application provides a method, apparatus, device, and storage medium for scheduling tasks in a large language model. To clarify the purpose, technical solution, and effects of this application, the present application is further described below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to explain this application and are not intended to limit this application.

[0043] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0044] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0045] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0046] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0047] This embodiment provides a task scheduling method for large language models, which is applied to a small computing system with a single graphics card. A small computing system with a single graphics card usually has limited GPU memory and CPU computing resources, resulting in limited available resources for such computing technology systems. For example, loading the Mixtral-8x22B MoE model with 4-bit precision requires approximately 110GB of video memory to store its parameters, exceeding the memory capacity of many small computing systems (such as Nvidia RTX 2080 / 3090 / 4090), whose GPU memory does not exceed 24GB, thus causing many small computing systems to be unable to carry large language models. This application provides a task scheduling method for large language models for such small computing systems. By splitting the tensors of the large language model into CC tensors, CG tensors, and GG tensors; scheduling the corresponding hardware resources of the CC tensors, CG tensors, and GG tensors to work together to control the large language model to execute tasks, this can make full use of the available computing, memory, and communication resources, and can improve the inference efficiency of small computing systems while ensuring the inference accuracy of large language models.

[0048] To implement the task scheduling method for large language models, as Figure 1 shown, this application provides a task scheduling system for large language models. The task scheduling of the large language model includes a Memory Manager, a Profiler & Solver, and a Task Scheduler. The Memory Manager is responsible for splitting tensors and supervising their storage allocation to handle the memory addresses required to transfer tensors from the CPU to the GPU; the Profiler & Solver is used to determine the optimal splitting ratios of the CC tensors, CG tensors, and GG tensors; the Task Scheduler is used to schedule the corresponding hardware resources of the CC tensors, CG tensors, and GG tensors to work together to ensure maximum parallelism, so as to control the large language model to execute tasks.

[0049] As Figure 2 shown, the task scheduling method for large language models provided by the embodiments of this application specifically includes:

[0050] S10. Perform tensor splitting on the tensors to be split in the large language model to obtain CC tensors, CG tensors, and GG tensors.

[0051] Specifically, the tensor to be split can be most of the parameters of the large language model, or a specified tensor in the large language model, etc. In the embodiments of the present application, the tensor to be split can be most of the parameters of the large language model, which can be determined according to a preset parameter quantity threshold. That is, before splitting the tensor to be split in the large language model, the tensor to be split with a parameter quantity greater than the preset parameter quantity threshold in the large language model can be read first. Among them, the preset parameter quantity threshold can be determined based on all the parameters of the large language model (such as half or one-third of all the parameters); it can also be a preset default value (such as setting a default value according to the model structure of the large language model), etc.

[0052] Illustrative example: Suppose the large language model structure mainly consists of multiple Transformer layers. Each Transformer layer consists of a self-attention layer and an MLP block (dense LLM) or a MoE block (sparse MoE LLM). The self-attention layer constructs the input sequence representation by identifying the relationships between tokens. The MLP or MoE block uses fully connected layers with activation functions to enhance the input sequence representation, and the MLP or MoE block occupies most of the parameters of the large language model. Then, the weight tensor in the MLP or MoE block can be used as the tensor to be split.

[0053] After obtaining the tensor to be split, as Figure 1 shown, the memory manager splits the tensor to be split into CC tensor, CG tensor, and GG tensor. The CC tensor is stored and executed on the CPU side. The CG tensor is stored on the CPU side and executed on the GPU side. The GG tensor is stored and executed on the GPU side. Specifically, the CC tensor is executed on the CPU (requiring CPU resources), and its calculation result is then transmitted to the GPU memory as the input of the next layer (requiring PCIe resources). The CG tensor is stored in the CPU memory to save GPU memory, but they need to be transmitted to the GPU memory through PCIe resources for execution (requiring both PCIe resources and GPU computing resources). The GG tensor is executed on the GPU. In the present application, the CG tensor utilizes PCIe and GPU resources to reduce the burden on the CPU and maximize the parallel use of resources, thereby improving the inference efficiency of the large language model.

[0054] Furthermore, due to the differences in the workloads of the prompting stage and the generation stage of the large language model, the splitting ratios of the CC tensor, CG tensor, and GG tensor in the prompting stage and the generation stage are different, which in turn leads to the need to frequently merge the CC tensor, CG tensor, and GG tensor from the CPU and GPU memories and reorganize them into continuous and executable tensors, affecting the inference efficiency of the large language model. To this end, the tensor to be split can be first split into the CC tensor, CG tensor, and GG tensor based on the generation stage, and then the CG' tensor can be set based on the CC tensor obtained by the division in the generation stage, and the difference between the prompting stage and the generation stage can be compensated by additionally controlling the computation amounts of the CC tensor and the CG' tensor.

[0055] Exemplarily, in one implementation, after splitting the tensors in the large language model to obtain the CC tensor, CG tensor, and GG tensor, the method further includes:

[0056] Determining a CG' tensor based on the CC tensor, where the CG' tensor represents the same tensor data as the CC tensor in terms of storage, and the CG' tensor will be executed on the GPU side;

[0057] Regarding the CC tensor, the CG tensor, and the GG tensor as the tensor splitting of the tensor to be split in the generation stage of the large language model;

[0058] Regarding the CC tensor, CG' tensor, the CG tensor, and the GG tensor as the tensor splitting of the tensor to be split in the prompting stage of the large language model.

[0059] Specifically, since the duration of the generation stage is longer than that of the prompting stage, the CC tensor, CG tensor, and GG tensor with fixed division in this embodiment of the present application are used as the tensor splitting in the generation stage to ensure that the generation stage can work with the optimal tensor splitting to ensure the inference efficiency of the large language model. However, since CPU calculation takes more time than other operations, and the prompting stage requires fewer hardware resources than the generation stage, the calculation part executed on the CPU can be divided to the GPU to improve the model performance of the large language model in the prompting stage. To this end, after obtaining the CC tensor, CG tensor, and GG tensor, the number of tokens allocated to the GPU segment in the prompting stage can be set based on the CC tensor, thereby reducing CPU calculation and increasing the overlap degree between different resources.

[0060] To this end, a CG' tensor can be created. The CG' tensor shares the CPU memory address with the CC tensor but operates on the GPU, and then the n originally processed by the CC tensor GA number of tokens are allocated to the CG' tensor operation to reduce CPU calculations, and the CPU calculation time can be reduced and the parallel utilization rate of different resources can be improved by adjusting the number of tokens allocated to the CG' tensor operation. For example, all the tokens processed by the CC tensor can be allocated to the CG' tensor, or some of the tokens processed by the CC tensor can be allocated to the CG' tensor, etc. Among them, the number of tokens allocated can also be determined by the method of determining the tensor splitting rate in the generation stage described below, or can be adjusted according to a preset allocation amount. In addition, it is worth noting that the embodiments of the present application are described by taking the example of allocating all the tokens processed by the CC tensor to the CG' tensor.

[0061] For example: Figure 2 As shown, assume that the large language model has two fully connected layers (Linear Layer), and the tensor to be split is the weight tensor in the MLP layer of the two fully connected layers. To clearly show the tensor splitting process, the activation calculation is ignored here. Formally, let X→[X1 X2] represent the input tensor of the MLP layer stored on the GPU. The input tensor contains T tokens. W1 and W2 represent the weight tensors of FC1 and FC2 in the MLP layer respectively. W1 and W2 will be split into CC tensor, CG tensor and GG tensor, that is and As Figure 3 shown, the MLP layer calculation can be split into:

[0062]

[0063] Among them, A(·) represents the activation function. X2 will be calculated with the CG' tensor, so the corresponding part is replaced by CG'. represents the output data of the fully connected layer at the back, represents the output data of the fully connected layer at the front.

[0064] Furthermore, the tensor splitting rates of the CC tensor, CG tensor and GG tensor can be preset, or can be determined according to the model structure of the large language model and the hardware configuration information of the running device for running the large language model, etc. In a specific implementation manner, the tensor splitting of the tensor to be split in the large language model to obtain the CC tensor, CG tensor and GG tensor specifically includes:

[0065] Obtain the hardware configuration information of the running device for running the large language model;

[0066] Based on the hardware configuration information, determine the tensor splitting rate of the large language model, and perform tensor splitting on the tensor to be split in the large language model according to the tensor splitting rate to obtain the CC tensor, CG tensor and GG tensor.

[0067] Specifically, the hardware configuration information can be the GPU memory of the running device. That is, the optimal segmentation rate corresponding to the GG tensor can be determined according to the GPU memory. This is because the GG tensor is stored in the GPU, so the memory required by the GG tensor is less than or equal to the GPU memory. Therefore, the optimal allocation rate of the GG tensor can be determined based on the GPU memory first, and then the optimal allocation rates of the CC tensor and the CG tensor can be determined based on the optimal allocation rate of the GG tensor. For example, the tensor that can be stored in the GPU memory can be used as the optimal allocation rate corresponding to the GG tensor, and then the optimal segmentation rate of the CC tensor can be determined based on the computing power of the CPU. Finally, the optimal segmentation rate of the CG tensor can be determined based on the optimal allocation rate corresponding to the GG tensor and the optimal segmentation rate of the CC tensor.

[0068] In the embodiments of the present application, the determination process of the optimal segmentation rates of the CC tensor, the CG tensor, and the GG tensor can be divided into two stages. In the first stage, the fixed GG tensor segmentation rate is determined based on the hardware configuration information, and the optimal allocation rates of the CC tensor and the CG tensor are determined based on the GG tensor segmentation rate. In the second stage, the search space of the GG tensor segmentation rate is restricted, and then the optimal allocation rates of the CC tensor and the CG tensor are updated through the greedy algorithm. Based on this, the determination of the tensor segmentation rate of the large language model based on the hardware configuration information specifically includes:

[0069] H10. Select a fixed GG sharding rate based on the hardware configuration information, and determine the CG sharding rate based on the fixed GG sharding rate;

[0070] H20. Search for the optimal GG sharding rate based on the time cost within the preset GG sharding rate search space, and determine the CC sharding rate based on the optimal GG sharding rate and the CG sharding rate to obtain the tensor segmentation rate of the large language model.

[0071] Specifically, in step H10, the fixed GG sharding rate is determined based on the hardware configuration information, and the CG sharding rate is determined based on the time cost of the network layer corresponding to the tensor to be segmented. Among them, the CG sharding rate can be expressed as:

[0072] t * fin = minY

[0073] r * CG = X[argminY]

[0074] Where t * finDenote the minimum time cost of the network layer corresponding to the tensor to be partitioned. Let \(X\) represent the CG partitioning rate array that can minimize the time cost of the network layer corresponding to the tensor to be partitioned, \(Y\) represent the time cost array corresponding to the CG partitioning rate array, and \(r\) * CG represents the CG partitioning rate.

[0075] The CG partitioning rate array can be determined according to the relationship between the data transfer time from CPU to GPU, the completion time of GPU kernel startup, and the GPU GEMM calculation time. Among them, the relationship between the data transfer time from CPU to GPU, the completion time of GPU kernel startup, and the GPU GEMM calculation time is divided into three relationships \(Q1\), \(Q2\), and \(Q3\). \(Q1\), \(Q2\), and \(Q3\) are respectively:

[0076] \(Q1: t\) L \(< t\) C2G

[0077] \(Q2: t\) G \(< t\) C2G

[0078] \(Q3: t\) L \(< t\) G

[0079] Among them, \(t\) G represents the GPU GEMM calculation time, \(t\) C2G represents the data transfer time from CPU to GPU, \(t\) L represents the completion time of GPU kernel startup.

[0080] Furthermore, the process of determining the CG partitioning rate array based on the three relationships \(Q1\), \(Q2\), and \(Q3\) may include:

[0081] First, during the model inference process, the tensors corresponding to the network layer include the input tensor and the weight tensor (regarded as the tensor to be partitioned). The number of bytes \(n\) of the tensors corresponding to the network layer GEMM \(= T\cdot M\cdot H\), where \(T\) represents the number of tokens, \(M\) represents the model dimension, and \(H\) represents the hidden dimension. The input tensor and the weight tensor (regarded as the tensor to be partitioned) can be \([T, M]\) and \([M, H]\) respectively. The number of bytes of the GG tensor executed on the GUP, the number of bytes of the CC tensor executed on the CUP, the number of bytes of the CG tensor executed on the GUP, and the number of bytes transferred from the CPU to the GPU can be respectively expressed as:

[0082] \(n\) G,GG \(= r\) GG \(\cdot n\) GEMM ,

[0083] \(n\) C,CC \(= r\) CC \(\cdot n\)GEMM ,

[0084] n G,CG = r CG ·n GEMM ,

[0085] n C2G,CG = r CG ·n W ,

[0086] where n G,GG represents the number of bytes of the GG tensor executed by the GUP, n C,CC represents the number of bytes of the CC tensor executed by the CUP, n G,CG represents the number of bytes of the CG tensor executed by the GUP, n C2G,CG represents the number of bytes transferred from the CPU to the GPU, n W represents the total number of bytes of the tensor to be segmented, r CC represents the tensor segmentation rate of the CC tensor, r CG represents the tensor segmentation rate of the CG tensor, r GG represents the tensor segmentation rate of the GG tensor.

[0087] Furthermore, since t G , t C , t C2G and t Launch can be expressed as:

[0088] t G = α G + n G ·β G

[0089] t C = α C + n C ·β C

[0090] t C2G = α C2G + n C2G ·β C2G

[0091] t Launch = constant

[0092] where n * represents the volume or number of bytes of the communication message, α * represents the startup time, β * represents the time required for each byte transfer or each unit of workload processing.

[0093] Thus, t G , t C , t C2Gand the completion time t of GPU kernel startup L can be expressed as:

[0094] t G = α G ·[sgn(r CG ) + sgn(r GG )] + (r CG + r GG )n GEMM ·β G

[0095] t C = α C ·sgn(r CC ) + r CC ·n GEMM ·β C

[0096] t C2G = α C2G + r CG ·n W ·β C2G

[0097] t L = [2·sgn(r CG ) + sgn(r GG [[ID=)]]·t Launch

[0098] where sgn(·) represents the sign function, t L represents the completion time of GPU kernel startup, r CC represents the tensor splitting rate of the CC tensor, r CG represents the tensor splitting rate of the CG tensor, r GG represents the tensor splitting rate of the GG tensor.

[0099] Correspondingly, the GPU computing time in the i-th GEMM calculation The CPU computing time in the i-th GEMM calculation The completion timestamp of the communication from CPU to GPU The completion timestamp τ of GPU kernel startup i L can be expressed as:

[0100]

[0101] where n l represents the number of GEMMs.

[0102] Therefore, the time cost of the network layer corresponding to the tensor to be split is And the time cost of the network layer corresponding to the tensor to be sliced can be minimized by changing the tensor slicing rate. represents the time cost of the GPU. represents the time cost of the CPU. Correspondingly, as Figure 4 shown, the three solution cases of r CG can be respectively expressed as:

[0103] Case 1 (Q1 is true and Q2 is true): The communication time between the CPU and the GPU is greater than the calculation time of GPU GEMM and the GPU kernel launch time. Therefore, the communication between the CPU and the GPU dominates the overall time cost, that is and Thus, it can be obtained that:

[0104]

[0105] Case 2 (Q1 is true and Q2 is false) or (Q1 is false and Q3 is true): The GPU calculation dominates the overall time cost, that is Thus, it can be obtained that:

[0106]

[0107] Case 3 (Q1 is false and Q3 is false): The GPU kernel launch time dominates the overall time cost, that is and Thus, it can be obtained that:

[0108]

[0109] Since there is only one unknown variable in the above three cases and the highest power is 1, the optimal r CG is located at the boundary points of each case, including t L = t C2G , t G = t C2G , t L = t G . r CG = 0 and r CG = 1 - r GG . Therefore, the time complexity is O(1). Represent the boundary points as the CG slicing rate array X, and then represent the time cost corresponding to the CG slicing rate array X as the array Y.

[0110] Furthermore, in step H20, after determining the tensor slicing rate of the CG tensor, it is feasible to optimize together with r CG based on the time cost to obtain the best inference speed under limited GPU memory. Specifically, set a set v i={i / n G , 1 ≤ i ≤ n G} to represent r GG , let the importance of the i-th value in the set at the j-th layer in the t-th iteration be:

[0111]

[0112] where represents r in the previous iteration of the j-th layer GG , v i ·n m represents the GPU memory consumption when r GG = v i . Then, in each iteration, calculate the importance of each value and each layer, and by adopting a greedy strategy, select the network layer with the highest importance and update the of this network layer. Then continue the iteration until the GPU memory is exhausted to obtain the tensor slicing rate of the GG tensor. Finally, determine the tensor slicing rate of the CC tensor based on the tensor slicing rate of the GG tensor and the tensor slicing rate of the CG tensor.

[0113] S20. Schedule the hardware resources corresponding to the CC tensor, CG tensor, and GG tensor to work together to control the large language model to execute tasks.

[0114] Specifically, when performing inference through the large language model, the hardware resources corresponding to the CC tensor, CG tensor, and GG tensor can be scheduled to arrange CPU computing tasks, GPU computing tasks, and communication tasks between the CPU and GPU to control the large language model to execute tasks. And when the large language model is in the prompt stage, schedule the hardware resources corresponding to the CC tensor, CG' tensor, the CG tensor, and the GG tensor to work together to control the prompt stage of the large language model to execute tasks. For example, Figure 2 shows an example of the task scheduler when processing the MLP block. In the prompt stage, the MLP block contains two linear layers, organizing the CPU computing tasks, GPU computing tasks, and CPU-GPU communication tasks into a task set to achieve efficient scheduling. Figure 3 then shows Figure 2 the execution result of the task set in. It should be noted that X2 is allocated n G tokens from X and T tokens, which are only used for CPU computing in the prompt stage to ensure that it does not affect the GPU computing.

[0115] Furthermore, to enable better cooperation between the CPU and GPU, four asynchronous execution streams can be set up in the pipeline, namely the CPU computing stream Stream-A, the GPU kernel launch stream Stream-B, the CPU-GPU communication stream Stream-C, and the GPU computing stream Stream-D. Stream-A is used for CPU computing, Stream-B is used for GPU kernel launch, Stream-C is used for communication between the CPU and GPU, and Stream-D is used for GPU computing. Independent operations in different execution streams can be carried out simultaneously.

[0116] Based on this, when scheduling the hardware resources corresponding to the CC tensor, CG tensor, and GG tensor to work together, the CPU computing stream can be scheduled to execute the CC tensor, the GPU kernel launch stream and the GPU computing stream can be scheduled to execute the GG tensor, and the GPU kernel launch stream, the CPU-GPU communication stream, and the GPU computing stream can be scheduled to execute the CG tensor, so as to realize the cooperation of the hardware resources corresponding to the CC tensor, CG tensor, and GG tensor. When scheduling the hardware resources corresponding to the CG' tensor, the CG tensor, and the GG tensor to work together, the GPU kernel launch stream and the GPU computing stream can be scheduled to execute the GG tensor, and the GPU kernel launch stream, the CPU-GPU communication stream, and the GPU computing stream can be scheduled to execute the CG tensor and the CG' tensor, so as to realize the cooperation of the hardware resources corresponding to the CC tensor, CG tensor, and GG tensor.

[0117] In summary, this embodiment provides a task scheduling method, device, equipment, and storage medium for a large language model. The method includes tensor slicing of the tensors to be sliced in the large language model to obtain the CC tensor, CG tensor, and GG tensor; scheduling the hardware resources corresponding to the CC tensor, CG tensor, and GG tensor to work together to control the large language model to execute tasks. By slicing the tensors into CC tensors, CG tensors, and GG tensors that utilize different hardware resources, and then enabling the cooperation of each hardware resource, the available computing, memory, and communication resources can be fully utilized, thereby improving the inference efficiency of a small computing system while ensuring the inference accuracy of the large language model.

[0118] Based on the above task scheduling method for a large language model, this embodiment provides a task scheduling device for a large language model, as Figure 5 shown. The task scheduling device for the large language model specifically includes:

[0119] A memory manager 100 is used to perform tensor slicing on the tensors to be sliced in a large language model to obtain CC tensors, CG tensors, and GG tensors. Among them, the CC tensors are stored and executed on the CPU side, the CG tensors are stored on the CPU side and executed on the GPU side, and the GG tensors are stored and executed on the GPU side;

[0120] A task scheduler 200 is used to schedule the respective hardware resources of the CC tensors, CG tensors, and GG tensors to work together to control the large language model to execute tasks.

[0121] Based on the above task scheduling method of the large language model, in this embodiment, a computer-readable storage medium is provided. The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the task scheduling method of the large language model as described in the above embodiment.

[0122] Based on the above task scheduling method of the large language model, the present application further provides a terminal device, as Figure 6 shown, which includes at least one processor 20; a display screen 21; and a memory 22, and may further include a communication interface 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22, and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is set to display a user guidance interface preset in the initial setting mode. The communication interface 23 can transmit information. The processor 20 can call the logical instructions in the memory 22 to execute the method in the above embodiment.

[0123] In addition, when the logical instructions in the above-mentioned memory 22 are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium.

[0124] The memory 22, as a computer-readable storage medium, can be set to store software programs and computer-executable programs, such as the program instructions or modules corresponding to the method in the embodiment of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, that is, to implement the method in the above embodiment.

[0125] The memory 22 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs, may also be transient storage media.

[0126] In addition, the specific processes of loading and executing multiple instructions in the above-mentioned storage medium and the terminal device have been described in detail in the above method, and will not be repeated here one by one.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A task scheduling method for a large language model, characterized in that The task scheduling method of the large language model is applied to a small computing system with a single graphics card. The task scheduling method of the large language model specifically includes: Perform tensor slicing on the tensors to be sliced in the large language model to obtain CC tensors, CG tensors, and GG tensors. Among them, the CC tensors are stored and executed on the CPU side, the CG tensors are stored on the CPU side and executed on the GPU side, and the GG tensors are stored and executed on the GPU side; Schedule the hardware resources corresponding to the CC tensors, CG tensors, and GG tensors to work together to control the large language model to execute tasks.

2. The task scheduling method for the large language model according to claim 1, wherein, The specific steps of performing tensor slicing on the tensors to be sliced in the large language model to obtain CC tensors, CG tensors, and GG tensors include: Obtain the hardware configuration information of the running device for running the large language model; Based on the hardware configuration information, determine the tensor slicing rate of the large language model, and perform tensor slicing on the tensors to be sliced in the large language model according to the tensor slicing rate to obtain CC tensors, CG tensors, and GG tensors.

3. The task scheduling method for the large language model according to claim 2, wherein, The specific steps of determining the tensor slicing rate of the large language model based on the hardware configuration information include: Select a fixed GG slicing rate based on the hardware configuration information, and determine the CG slicing rate based on the fixed GG slicing rate; Search for the optimal GG slicing rate based on the CG slicing rate within the preset GG slicing rate search space according to the time cost, and determine the CC slicing rate based on the optimal GG slicing rate and the CG slicing rate to obtain the tensor slicing rate of the large language model.

4. The task scheduling method for the large language model according to claim 1, wherein, After performing tensor slicing on the tensors in the large language model to obtain CC tensors, CG tensors, and GG tensors, the method further includes: Determine the CG' tensors based on the CC tensors. Among them, the CG' tensors represent the same tensor data as the CC tensors in terms of storage, and the CG' tensors will be executed on the GPU side; Use the CC tensors, the CG tensors, and the GG tensors as the tensor slicing of the tensors to be sliced in the generation stage of the large language model; Use the CC tensors, CG' tensors, the CG tensors, and the GG tensors as the tensor slicing of the tensors to be sliced in the prompting stage of the large language model.

5. The task scheduling method for the large language model according to claim 4, wherein The specific steps of scheduling the hardware resources corresponding to the CC tensors, CG tensors, and GG tensors to work together to control the large language model to execute tasks include: When the large language model is in the prompting stage, schedule the hardware resources corresponding to the CC tensors, CG' tensors, the CG tensors, and the GG tensors to work together to control the prompting stage of the large language model to execute tasks; When the large language model is in the generation stage, schedule the hardware resources corresponding to the CC tensors, CG tensors, and GG tensors to work together to control the generation stage of the large language model to execute tasks.

6. The task scheduling method for the large language model according to claim 1 or 4, characterized in that The specific steps of scheduling the hardware resources corresponding to the CC tensors, CG tensors, and GG tensors to work together include: Set four asynchronous execution streams, where the four asynchronous execution streams include a CPU computing stream, a GPU kernel startup stream, a CPU-GPU communication stream, and a GPU computing stream; Schedule the CPU computing stream to execute the CC tensor, schedule the GPU kernel launch stream and the GPU computing stream to execute the GG tensor, and schedule the GPU kernel launch stream, the CPU-GPU communication stream, and the GPU computing stream to execute the CG tensor, so as to enable the hardware resources corresponding to the CC tensor, the CG tensor, and the GG tensor to work together collaboratively.

7. The task scheduling method for the large language model according to claim 6, wherein The collaborative work of the hardware resources corresponding to the scheduled CC tensor, CC' tensor, CG tensor, and GG tensor specifically includes: Schedule the GPU kernel launch stream and the GPU computing stream to execute the GG tensor, and schedule the GPU kernel launch stream, the CPU-GPU communication stream, and the GPU computing stream to execute the CG tensor and CG' tensor, so as to enable the hardware resources corresponding to the CC tensor, the CG tensor, and the GG tensor to work together collaboratively.

8. A task scheduling device for a large language model, characterized in that, The task scheduling device of the large language model specifically includes: A memory manager, which is used to perform tensor slicing on the tensors to be sliced in the large language model to obtain the CC tensor, the CG tensor, and the GG tensor. Among them, the CC tensor is stored and executed on the CPU side, the CG tensor is stored on the CPU side and executed on the GPU side, and the GG tensor is stored and executed on the GPU side; A task scheduler, which is used to schedule the hardware resources corresponding to the CC tensor, the CG tensor, and the GG tensor to work together collaboratively to control the large language model to execute tasks.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the task scheduling method of the large language model as described in any one of claims 1-7.

10. A terminal device, characterized in that, It includes: A processor and a memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, it implements the steps in the task scheduling method of the large language model as described in any one of claims 1-7.

Images