Large language model reasoning scheduling method and device and electronic equipment

Abstract

The invention provides a scheduling method and device for big language model reasoning and electronic equipment. The method comprises the steps that multiple micro-batches of a big language model executing reasoning task are determined; dividing the parameter weight of each layer in the large language model into a plurality of weight pages according to the micro-batch number; and executing each micro-batch of the reasoning task, pre-fetching a preset number of weight pages required by execution of subsequent micro-batches while executing each micro-batch, and storing the pre-fetched weight pages in a GPU memory. When each micro-batch is executed, a subsequent micro-batch weight page is pre-fetched and stored in a GPU memory, and a micro-batch processing process is performed based on the pre-fetched weight page, so that the parameter weight of the whole large language model is prevented from being integrally loaded to the GPU at one time, the calculation waiting time caused by weight loading is shortened, the memory occupation is reduced, and the calculation efficiency is improved. Unnecessary data transmission and storage are avoided, and the task processing efficiency based on the large language model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a large language model inference scheduling method and device and electronic equipment. BACKGROUND

[0002] With the continuous expansion of the scale of large language models, the inference process puts forward very high requirements on the graphics processing unit (GPU) memory and bandwidth resources.

[0003] The existing model is usually divided into multiple stages and deployed on different GPUs in turn, and multiple different GPUs each undertake corresponding computing functions. Before implementing the computing process based on multiple different GPUs, the model weights of the large language model need to be loaded to each GPU in one time, which consumes a long time and hinders the use of other data channels, resulting in low efficiency of task processing based on the large language model. SUMMARY

[0004] The present application provides a large language model inference scheduling method, device and electronic equipment to improve the efficiency of task processing based on the large language model.

[0005] The present application provides a large language model inference scheduling method, comprising the following steps: determining a plurality of micro-batches for the large language model to perform an inference task; dividing the parameter weights of each layer in the large language model into a plurality of weight pages according to the number of micro-batches; executing each micro-batch of the inference task, and at the same time of each micro-batch execution, pre-fetching the weight pages required for a preset number of subsequent micro-batches, and storing the pre-fetched weight pages in the GPU memory.

[0006] According to the large language model inference scheduling method provided by the present application, the execution of each micro-batch of the inference task comprises: based on the micro-batches corresponding to the core processing stage of calling the central processing unit (CPU) to execute the attention mechanism, and based on the micro-batches corresponding to other stages outside the core processing stage calling the GPU to execute; wherein the core processing stage includes the O projection stage in the attention mechanism and the feed-forward neural network (FFN) stage.

[0007] According to the large language model inference scheduling method provided by the present application, the micro-batch processing process corresponding to the attention mechanism calculation in the inference task comprises: based on the micro-batches corresponding to the pre-processing stage of calling the GPU to execute the attention mechanism, determining the query key value (QKV) projection result of the GPU for the pre-processing stage; The pre-processing stage includes a LayerNorm stage in the attention mechanism and a QKV projection stage. Based on the data channel between the CPU and the GPU, the QKV projection result is transmitted to the CPU, and the CPU is called to execute the core processing stage of the attention mechanism to obtain an execution result of the attention mechanism calculation. Based on the data channel between the CPU and the GPU, the execution result is transmitted to the GPU.

[0008] According to the large language model inference scheduling method provided by the application, after the execution result is transmitted to the GPU, the method further comprises: Based on calling the GPU, based on the execution result, a subsequent micro-batch processing process of the attention mechanism calculation in the inference task is executed.

[0009] According to the large language model inference scheduling method provided by the application, the method further comprises: Based on the data channel between the GPU and the CPU, the weight pages required for a preset number of subsequent micro-batch executions are asynchronously scheduled to the GPU memory while each micro-batch execution is performed.

[0010] According to the large language model inference scheduling method provided by the application, the method further comprises: Based on the event or semaphore synchronization control mechanism, the weight pages required for a preset number of subsequent micro-batch executions are asynchronously scheduled to the GPU memory.

[0011] According to the large language model inference scheduling method provided by the application, the method further comprises: After determining that the weight pages of the current micro-batch are stored in the GPU memory, the processing process of the current micro-batch is triggered.

[0012] The application also provides a large language model inference scheduling device, comprising the following modules: An execution determination module is configured to determine a plurality of micro-batches of a large language model executing an inference task. A division module is configured to divide the parameter weights of each layer in the large language model into a plurality of weight pages according to the number of micro-batches. The execution module is configured to execute each micro-batch of the inference task, and pre-fetch weight pages required for execution of a preset number of subsequent micro-batches while each micro-batch is being executed, and store the pre-fetched weight pages in a GPU memory.

[0013] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the scheduling method for large language model inference according to any one of the above when executing the program.

[0014] The application provides a scheduling method and device for large language model inference and an electronic device, which decomposes a large-scale inference task into small-scale sub-tasks by determining a plurality of micro-batches of the large language model for executing the inference task, thereby reducing the data volume and computational complexity of single processing. The weight pages of subsequent micro-batches are pre-fetched and stored in the GPU memory while each micro-batch is being executed, and the micro-batch processing process is performed based on the pre-fetched weight pages, thereby avoiding loading the entire parameter weight of the large language model to the GPU at one time, reducing the computational waiting time caused by weight loading, reducing the memory occupation, avoiding unnecessary data transmission and storage, and improving the task processing efficiency based on the large language model. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0016] Figure 1 is a flowchart of the scheduling method for large language model inference provided by the application.

[0017] Figure 2 is a flowchart of the asynchronous scheduling implementation provided by the application.

[0018] Figure 3 is a flowchart of the pipeline arrangement provided by the application.

[0019] Figure 4 is a structural diagram of the scheduling device for large language model inference provided by the application.

[0020] Figure 5 is a structural diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0021] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0022] With the continuous expansion of the scale of large language models, the inference process puts forward very high requirements on GPU memory and bandwidth resources. In the pipeline parallel training or inference framework in the related method, the model is usually divided into multiple stages and deployed on different GPUs in turn, and only the intermediate activation results are passed between stages.

[0023] However, in offline deployment or resource-limited environments, the number of GPUs is limited, and part of the calculation needs to be migrated to the central processing unit (CPU), resulting in multiple types of data transmission: including intermediate results, key value (KV) cache, model weights, etc., and the data flow direction is no longer single.

[0024] The related method has the following problems in data scheduling: different types of data transmission (such as KV cache and weight) are not decoupled and managed, causing bandwidth conflict and blocking; the model weight is loaded to the GPU in one time, which takes a long time and hinders the use of other data channels; the calculation task and data transmission scheduling are not finely coordinated, causing CPU / GPU idle waiting, and the system utilization is low.

[0025] In view of the defects in the related method, the present application provides a scheduling method for language model inference, Figure 1 The present application provides a flowchart of the scheduling method for large language model inference, as shown in Figure 1 The method comprises the following steps: Step 110, determining a plurality of micro-batches for the large language model to execute an inference task; Step 120, dividing the parameter weights of each layer in the large language model into a plurality of weight pages according to the number of micro-batches; Step 130, executing each micro-batch of the inference task, and at the same time of each micro-batch execution, pre-fetching the weight pages required by a preset number of subsequent micro-batches from the CPU, and storing the pre-fetched weight pages in the GPU memory.

[0026] The execution subject of the large language model inference scheduling method provided by the present application can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), or a personal computer (PC), etc., which are not specifically limited by the present application.

[0027] The technical solutions of the present application will be described in detail below with the computer executing the large language model inference scheduling method provided by the present application as an example.

[0028] In step 110, a plurality of micro-batches for the large language model to perform an inference task are determined.

[0029] It can be understood that the large language model generally includes an input layer, an embedding layer, a plurality of hidden layers (including attention mechanism related layers, feedforward neural network layers, etc.), and an output layer. In the process of task execution based on the large language model, the execution is performed from front to back by layer, so each layer in the large language model can be split to determine a plurality of micro-batches in the execution process of each layer.

[0030] It should be noted that in the task processing process of the large language model, the micro-batch (Micro-Batch) is a smaller unit further divided from a larger inference task. In the task inference process of the large language model, the inference task corresponding to each layer of the model is divided into a plurality of small subsets (micro-batches), and each subset can be treated as an independent processing unit and processed by the model sequentially or in parallel.

[0031] By dividing the larger inference task into a plurality of micro-batches, a plurality of micro-batches can be processed simultaneously based on parallel computing, which can accelerate the inference process of the large language model and improve the task processing efficiency.

[0032] In step 120, the parameter weights of each layer in the large language model are divided into a plurality of weight pages according to the number of micro-batches, and each weight page corresponds to the parameter weights required for a micro-batch execution.

[0033] Weight pages are a way of organizing and managing large language model parameters in memory or video memory, usually closely related to model loading, storage, block calculation and hardware optimization. Weight pages are logical or physical block units of model weights. Due to the large size of large language model parameters, weight pages split the weights into multiple small blocks (pages) to achieve on-demand loading, block calculation and efficient memory management.

[0034] After determining the plurality of micro-batches of the large language model in step 110, the parameter weights of the large language model are divided accordingly so that each micro-batch can use the corresponding required parameters when executing.

[0035] The parameter weights contained in each weight page are independent of each other, that is, a micro-batch only needs to use the parameters in its corresponding weight page when executing, without relying on the parameters in other weight pages. This can ensure that there is no parameter conflict when processing multiple micro-batches in parallel.

[0036] Each weight page contains all the parameter weights required for the execution of the micro-batch to ensure the correct execution of the corresponding micro-batch. If a weight page is missing the necessary parameters, it may cause the inference result to be incorrect or the inference task to be unable to complete.

[0037] Specifically, for each layer of the large language model, the parameter weights thereof are divided according to the number of micro-batches. For example, for the Kth layer of the large language model having N micro-batches, the parameter weights of the Kth layer are divided into N weight pages according to the need of the N micro-batches for execution. Each weight page can correspond to the parameter weights required for the execution of a micro-batch, so that the weight page and the micro-batch correspond one-to-one.

[0038] In step 130, each micro-batch of the inference task is executed, and while each micro-batch is executing, a predetermined number of subsequent micro-batches are pre-fetched for execution of the weight pages required, and the pre-fetched weight pages are stored in the GPU memory.

[0039] During the execution of the inference task, pre-fetching weight pages can reduce the time for the GPU to wait for parameter loading, further improving inference efficiency.

[0040] When the current micro-batch starts executing, the pre-fetching operation is started, and a predetermined number of subsequent micro-batches required for execution of the weight pages can be pre-fetched based on the data transmission channel between the CPU and the GPU after the CPU obtains the weight pages. The pre-fetching operation can be performed in the background and does not affect the inference calculation of the current micro-batch, so as to realize the parallel performance of calculation and data loading. For the first micro-batch of the large language model executing the inference task, the weight pages required for execution of the first predetermined number of micro-batches can be pre-loaded to the GPU before execution.

[0041] The pre-fetched weight page can be transmitted from the CPU memory to the GPU memory through a data transmission channel such as a Peripheral Component Interconnect Express (PCIe) bus or the like. During data transmission, the efficiency and bandwidth utilization of data transmission can be optimized, for example, by using data compression, batch transmission and other technologies to reduce data transmission time.

[0042] The preset number needs to be comprehensively considered according to factors such as hardware performance, characteristics of inference tasks, and system load. If the preset number is too large, it may cause a bandwidth bottleneck between the CPU and the GPU, increasing data transmission time; if the preset number is too small, the advantages of pre-fetching cannot be fully utilized, and the effect of reducing GPU waiting time is not obvious. Specifically, the preset number can be set to two.

[0043] Before executing the current micro-batch, it is necessary to ensure that the weight page corresponding to the current micro-batch is already in the GPU memory. The weight page in the GPU memory can be dynamically managed during the subsequent execution process of the micro-batch, and the weight page that is no longer used is released in time to make room for new weight pages.

[0044] The GPU performs inference calculation of the large language model according to the loaded input data and weight page. During the calculation process, each computing core of the GPU processes data in parallel to speed up the inference process. After the calculation is completed, the inference result is read from the GPU memory and subjected to subsequent processing or output.

[0045] The scheduling method for large language model inference provided by the present application divides large-scale inference tasks into small-scale sub-tasks by determining multiple micro-batches of the large language model for performing inference tasks, thereby reducing the amount of data processed at a time and the computational complexity. The weight pages of subsequent micro-batches are pre-fetched and stored in the GPU memory when each micro-batch is executed, and the micro-batch processing process is based on the pre-fetched weight pages, thereby avoiding the loading of the entire parameter weight of the large language model to the GPU at a time, reducing the calculation waiting time caused by weight loading, reducing memory occupation, avoiding unnecessary data transmission and storage, and improving the task processing efficiency based on the large language model.

[0046] In one embodiment, micro-batches corresponding to core processing stages of the attention mechanism are executed based on calling a central processing unit (CPU), and micro-batches corresponding to other stages other than the core processing stages are executed based on calling a GPU; wherein the core processing stages include an O projection stage in the attention mechanism and a feed-forward neural network (FFN) stage.

[0047] It should be noted that the O projection stage (Output Projection) is a key link of the attention mechanism, which functions to weight and sum the attention weights and the value matrix (V) to obtain the final attention output.

[0048] A feedforward neural network (FFN) is usually composed of multiple fully connected layers, containing a large number of parameters and complex nonlinear transformations. In the FFN stage, the input data will pass through multiple hidden layers one by one, and each hidden layer needs to perform matrix multiplication and activation function calculation. Due to the large number of parameters in the fully connected layer, frequent memory access is required during calculation, and there is a certain order dependency between different neurons. In addition, the calculation amount of the FFN stage is large, and in large-scale models, the calculation time often accounts for a considerable proportion of the entire inference process.

[0049] The CPU has strong single-thread processing capability and complex control logic, and can flexibly cope with the complex data dependency and calculation logic in the O projection stage and the FFN stage. For example, in the O projection stage, the CPU can dynamically adjust the calculation order according to different attention heads to ensure the accuracy of the calculation. For the nonlinear transformation of different hidden layers in the FFN stage, the CPU can accurately calculate according to the order defined by the model, avoiding logical errors caused by parallel processing.

[0050] By assigning tasks with different characteristics to the CPU and GPU for execution, the problem of resource idling when a single hardware processes complex tasks is avoided. For example, when the GPU is processing other stages suitable for parallel computation, the CPU can simultaneously process the micro-batch tasks of the O projection stage and the FFN stage, realizing the parallel work of the CPU and the GPU and improving the utilization of hardware resources.

[0051] In one embodiment, the processing process of the attention mechanism calculation of the corresponding micro-batch of the inference task includes: determining the Query Key Value (QKV) projection result of the GPU for the pre-processing stage of the attention mechanism based on calling the GPU to execute the corresponding micro-batch of the pre-processing stage; wherein the pre-processing stage includes the LayerNorm stage and the QKV projection stage in the attention mechanism; transmitting the QKV projection result to the CPU based on the data channel between the CPU and the GPU, calling the CPU to execute the core processing stage of the attention mechanism, and transmitting the obtained calculation result of the attention mechanism to the GPU based on the data channel between the CPU and the GPU.

[0052] In the large language model inference task, the corresponding micro-batch is calculated for the attention mechanism. The process is as follows: first, the GPU processes the pre-processing stage, then the results are transmitted to the CPU core for processing, and finally the results are transmitted back to the GPU, thereby realizing the cooperative processing process of GPU and CPU.

[0053] Based on the GPU execution layer normalization LayerNorm stage and QKV projection stage, these two stages involve a large number of parallelizable matrix operations. With numerous computing cores and high-bandwidth memory, GPU can quickly complete the calculation to determine the QKV projection result, fully leveraging its parallel computing advantage and shortening the calculation time.

[0054] Through the data channel between CPU and GPU, the QKV projection result is transmitted to CPU. CPU is good at processing complex logic and sequential calculation, and can accurately handle data dependency in the core processing stage of attention mechanism, such as calculating attention weight and weighted summation, to ensure calculation accuracy.

[0055] Finally, the attention mechanism result calculated by CPU is transmitted back to GPU for subsequent processing. This division of labor and cooperation can combine the calculation advantages of CPU and GPU, reduce hardware idling, optimize data transmission, and thus improve the processing efficiency of the whole inference task.

[0056] In one embodiment, after transmitting the execution result to the GPU, it further includes: based on calling the GPU, based on the execution result, executing the subsequent micro-batch processing process of attention mechanism calculation in the inference task.

[0057] After transmitting the execution result back to the GPU, one or more GPUs can be used to implement the subsequent micro-batch processing process of attention mechanism calculation. GPU has strong parallel computing capability, and for parallel operations such as large-scale matrix operations in subsequent micro-batches, GPU can simultaneously mobilize many computing cores for processing, greatly improving the calculation speed.

[0058] Moreover, data flows within the GPU, reducing data interaction with external devices and reducing communication delay, making the whole attention mechanism calculation process more compact and efficient, thereby speeding up the overall progress of the inference task.

[0059] In one embodiment, while each micro-batch is executed, a preset number of weight pages required for subsequent micro-batch execution are prefetched and stored in the GPU memory, including: while each micro-batch is executed, based on the data channel between the GPU and the CPU, asynchronously scheduling a preset number of weight pages required for subsequent micro-batch execution to the GPU memory.

[0060] When each micro-batch is executed, the weight pages required by the subsequent micro-batches are asynchronously scheduled to the GPU memory through the data channel between the GPU and the CPU.

[0061] Asynchronous scheduling means that data transmission can be carried out in parallel with the current micro-batch calculation, without occupying additional calculation time, and can fully tap the hardware parallel potential. The preset number of scheduling can prepare data in advance to avoid GPU calculation stagnation due to missing weight pages. When the GPU completes the current calculation, the required weight pages for the subsequent micro-batches have been transmitted to the GPU memory, which can be immediately used for processing the subsequent micro-batches, reducing the waiting time, improving hardware utilization, making the entire inference process more compact and efficient, and significantly speeding up the processing speed of large language model inference tasks.

[0062] In one embodiment, the asynchronous scheduling of the weight pages required by the preset number of subsequent micro-batch executions to the GPU memory includes: a synchronization control mechanism based on events or semaphores, which asynchronously schedules the weight pages required by the preset number of subsequent micro-batch executions to the GPU memory.

[0063] The synchronization control mechanism based on events or semaphores to asynchronously schedule the weight pages of the subsequent micro-batches to the GPU memory can effectively optimize the calculation process. The event mechanism can trigger a notification when a specific operation is completed, and when the CPU is ready with the weight page data, an event signal is sent, and the GPU starts asynchronous transmission upon receiving the signal, without the need for synchronization and waiting.

[0064] The semaphore can control resource access, and by setting an appropriate semaphore value, it can ensure that only a specified number of weight pages are transmitted at the same time, avoiding resource conflicts. This mechanism allows data transmission and calculation to proceed in parallel, stores weight pages in the GPU memory in advance, reduces GPU waiting time, improves hardware utilization, and thus speeds up the execution speed of large language model inference tasks.

[0065] In one embodiment, the calculation tasks of multiple micro-batches of the large language model performing an inference task are executed in a pipeline manner.

[0066] When the large language model performs an inference task, multiple micro-batch calculation tasks are processed in a pipeline manner, which can greatly improve efficiency.

[0067] The pipeline divides the inference process into multiple stages, such as data preprocessing, attention calculation, and feedforward neural network calculation, and different micro-batches can be processed in parallel in each stage.

[0068] When the first micro-batch enters the attention calculation stage, the second micro-batch can start data preprocessing, achieving full utilization of resources, not only shortening the overall inference time, but also improving system throughput without increasing hardware costs, meeting large-scale inference demands.

[0069] The application provides an asynchronous scheduling implementation flowchart, like Figure 2 The application provides an asynchronous scheduling implementation flowchart, and specifically comprises the following steps. Step S1, weight paging scheduling mechanism initialization stage: S1.1: The parameter weights of each layer model are divided into multiple weight pages according to the number of micro-batches, and each weight page serves one micro-batch. S1.2: The block weight is transmitted asynchronously through the path from CPU to GPU, so as to realize the overlap of paging loading and calculation process.

[0070] S1.3: The weight pages of the first two micro-batches are prefetched to the GPU memory.

[0071] Step S2, leading stage: The preprocessing is performed on the first two micro-batches in each generated decode step: S2.1: The attention preprocessing stage (LayerNorm stage and QKV projection stage) is performed on the GPU. S2.2: The QKV projection result is transmitted to the CPU. S2.3: The output of the attention mechanism is calculated on the CPU. S2.4: The weight pages of the next micro-batch are prefetched to the GPU memory.

[0072] Step S3, main pipeline scheduling stage: For each layer of the model and all micro-batches thereof, the following steps are performed in sequence: S3.1: The attention output (intermediate hidden state) on the CPU is transmitted back to the GPU. S3.2: The attention post-processing stage is performed on the GPU, including O projection and FFN. S3.3: The preprocessing of the next micro-batch (steps S2.1 to S2.4) is asynchronously scheduled.

[0073] Step S4, asynchronous scheduling and synchronization control stage: S4.1: All tasks are submitted to the scheduling queue in an asynchronous manner. S4.2: The data dependency between tasks is coordinated through mechanisms such as events and semaphores, to ensure that data is prepared before triggering calculation. S4.3: The data and task flow between the GPU and the CPU is ensured to be non-blocking and high-throughput.

[0074] The specific pipeline arrangement flowchart realized based on the above process can be as follows Figure 3 The application provides a pipeline arrangement flowchart. Wherein, represents the i-th micro-batch of the corresponding step. , The weight pages corresponding to the micro-batches 1 and 2 of step 1.3 are transmitted by the CPU into the GPU memory. Figure 3 The whole flow steps from to are specifically shown. GPU2CPU means that the GPU transmits data to the CPU, and CPU2GPU means that the CPU transmits data to the GPU.

[0075] The scheduling device for large language model inference provided by the present application is described below. The scheduling device for large language model inference described below can be correspondingly referred to the scheduling method for large language model inference described above.

[0076] As shown in Figure 4 , the device comprises: An execution determining module 410 is configured to determine a plurality of micro-batches for a large language model to perform an inference task. A dividing module 420 is configured to divide the parameter weights of each layer in the large language model into a plurality of weight pages according to the number of micro-batches. An execution module 430 is configured to execute each micro-batch of the inference task, and pre-fetch the weight pages required for a preset number of subsequent micro-batches while each micro-batch is being executed, and store the pre-fetched weight pages into a GPU memory.

[0077] The scheduling device for large language model inference provided by the present application divides a large-scale inference task into small-scale sub-tasks by determining a plurality of micro-batches for a large language model to perform an inference task, thereby reducing the data volume and computational complexity of single processing. The weight pages of subsequent micro-batches are pre-fetched and stored into the GPU memory while each micro-batch is being executed, and the micro-batches are processed based on the pre-fetched weight pages, thereby avoiding loading the parameter weights of the entire large language model into the GPU at one time, reducing the computational waiting time caused by weight loading, reducing the memory occupation, avoiding unnecessary data transmission and storage, and improving the task processing efficiency based on the large language model.

[0078] In one embodiment, the execution module 430 is specifically configured to: The execution of each micro-batch of the inference task comprises: based on the micro-batches corresponding to the core processing stage of the attention mechanism executed by the CPU, and based on the micro-batches corresponding to other stages except the core processing stage executed by the GPU; The core processing stage comprises the O projection stage in the attention mechanism and the feed-forward neural network (FFN) stage.

[0079] In one embodiment, the execution module 430 is further specifically configured to: The attention mechanism calculation in the inference task corresponds to a micro-batch processing process, including: Based on the micro-batch corresponding to the pre-processing stage of the attention mechanism calling the GPU to execute, the GPU determines the query key value QKV projection result of the pre-processing stage; The pre-processing stage includes the layer normalization LayerNorm stage and the QKV projection stage in the attention mechanism. Based on the data channel between the CPU and the GPU, the QKV projection result is transmitted to the CPU, and the CPU is called to execute the core processing stage of the attention mechanism to obtain the execution result of the attention mechanism calculation. Based on the data channel between the CPU and the GPU, the execution result is transmitted to the GPU.

[0080] In one embodiment, the execution module 430 is further specifically used for: After the execution result is transmitted to the GPU, it further includes: Based on the execution result, the GPU is called to execute the subsequent micro-batch processing process of the attention mechanism calculation in the inference task.

[0081] In one embodiment, the execution module 430 is further specifically used for: The pre-fetching of the weight pages required for the execution of the preset number of subsequent micro-batches and the storage of the pre-fetched weight pages in the GPU memory are performed at the same time as each micro-batch execution, including: At the same time as each micro-batch execution, the weight pages required for the execution of the preset number of subsequent micro-batches are asynchronously scheduled to the GPU memory based on the data channel between the GPU and the CPU.

[0082] In one embodiment, the execution module 430 is further specifically used for: The asynchronous scheduling of the weight pages required for the execution of the preset number of subsequent micro-batches to the GPU memory includes: Based on the event or semaphore synchronization control mechanism, the weight pages required for the execution of the preset number of subsequent micro-batches are asynchronously scheduled to the GPU memory.

[0083] In one embodiment, the execution module 430 is further specifically used for: The execution of the calculation task of the plurality of micro-batches of the large language model executing the inference task adopts a pipeline manner.

[0084] Figure 5 An example of an entity structure diagram of an electronic device is shown in Figure 5As shown, the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can invoke a logical instruction in the memory 530 to execute a scheduling method for large language model inference, which includes determining a plurality of micro-batches for a large language model to perform an inference task; dividing parameter weights of each layer in the large language model into a plurality of weight pages according to the number of micro-batches; executing each micro-batch of the inference task, and pre-fetching weight pages required for execution of a preset number of subsequent micro-batches while each micro-batch is being executed, and storing the pre-fetched weight pages to a GPU memory.

[0085] In addition, the logical instruction in the memory 530 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0086] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the scheduling method for large language model inference provided by the above-mentioned methods, which includes determining a plurality of micro-batches for a large language model to perform an inference task; dividing parameter weights of each layer in the large language model into a plurality of weight pages according to the number of micro-batches; executing each micro-batch of the inference task, and pre-fetching weight pages required for execution of a preset number of subsequent micro-batches while each micro-batch is being executed, and storing the pre-fetched weight pages to a GPU memory.

[0087] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a method for scheduling large language model inference provided by the above methods, the method comprising: determining a plurality of micro-batches in which a large language model performs an inference task; dividing parameter weights of each layer in the large language model into a plurality of weight pages according to the number of micro-batches; performing each micro-batch of the inference task, and pre-fetching weight pages required for performing a preset number of subsequent micro-batches while each micro-batch is being performed, and storing the pre-fetched weight pages in a GPU memory.

[0088] The device embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0089] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0090] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A scheduling method for reasoning in a large language model, characterized in that, include: Determine multiple micro-batches for large language models to perform inference tasks; The parameter weights of each layer in the large language model are divided into multiple weight pages according to the number of micro-batches. Each micro-batch of the inference task is executed, and while each micro-batch is being executed, a preset number of weight pages required for the execution of subsequent micro-batches are prefetched and stored in the GPU memory.

2. The scheduling method for large language model inference according to claim 1, characterized in that, Each micro-batch executing the inference task includes: Micro-batches are based on the core processing stage corresponding to the central processing unit (CPU) executing the attention mechanism, and micro-batches are based on the GPU executing other stages besides the core processing stage. The core processing stage includes the O-projection stage in the attention mechanism and the FFN stage in the feedforward neural network.

3. The scheduling method for large language model inference according to claim 2, characterized in that, The attention mechanism in the inference task calculates the corresponding micro-batch processing procedure, including: Based on the micro-batch corresponding to the preprocessing stage that invokes the attention mechanism of the GPU, determine the QKV projection result of the query key-value pair of the GPU to the preprocessing stage; The preprocessing stage includes the LayerNorm stage and the QKV projection stage in the attention mechanism. Based on the data channel between the CPU and the GPU, the QKV projection result is transmitted to the CPU, and the CPU is invoked to execute the core processing stage of the attention mechanism to obtain the execution result of the attention mechanism calculation; The execution result is transmitted to the GPU via the data channel between the CPU and the GPU.

4. The scheduling method for large language model inference according to claim 3, characterized in that, After transmitting the execution result to the GPU, the process further includes: Based on the GPU call and the execution result, the subsequent micro-batch processing of the attention mechanism computation in the inference task is performed.

5. The scheduling method for large language model inference according to claim 1, characterized in that, The step of prefetching a preset number of weight pages required for the execution of subsequent micro-batches during the execution of each micro-batch, and storing the prefetched weight pages in GPU memory, includes: While each micro-batch is being executed, a preset number of weighted pages required for the execution of subsequent micro-batches are asynchronously scheduled to the GPU memory based on the data channel between the GPU and the CPU.

6. The scheduling method for large language model inference according to claim 5, characterized in that, The asynchronous scheduling of a preset number of weighted pages required for subsequent micro-batch executions to be allocated to the GPU memory includes: Based on an event or semaphore-based synchronization control mechanism, a preset number of weighted pages required for subsequent micro-batch execution are asynchronously scheduled to the GPU memory.

7. The scheduling method for large language model inference according to claim 1, characterized in that, The step of pre-fetching a preset number of weighted pages required for the execution of subsequent micro-batches while executing each micro-batch further includes: After determining that the weight page of the current micro-batch is stored in GPU memory, the processing of the current micro-batch is triggered.

8. The scheduling method for large language model inference according to claim 1, characterized in that, The computational tasks of multiple micro-batch inference tasks performed by the large language model are executed in a pipeline manner.

9. A scheduling device for reasoning in a large language model, characterized in that, include: The execution determination module is used to determine multiple micro-batches for the large language model to perform inference tasks; The partitioning module is used to divide the parameter weights of each layer in the large language model into multiple weight pages according to the number of micro-batches. The execution module is used to execute each micro-batch of the inference task, and while executing each micro-batch, it prefetches a preset number of weight pages required for the execution of subsequent micro-batches, and stores the prefetched weight pages in the GPU memory.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the scheduling method for large language model inference as described in any one of claims 1 to 9.

Images