LoRA weight loading method and device for large language model reasoning

Abstract

The invention provides a LoRA weight loading method and device for large language model reasoning. The method comprises the steps that when a reasoning request of a large language model contains LoRA weight calculation, the reasoning request is added to a preparation queue; in the preparation queue, carrying out preprocessing operation on the N reasoning requests; in the preprocessing operation process, copying the LoRA weight to a continuous cache region of a CPU according to a preset rule; after the preprocessing operation is completed, adding the reasoning request to a waiting queue; and when the reasoning request is scheduled and executed, loading the LoRA weight corresponding to the reasoning request into the GPU of the large language model to perform reasoning calculation. According to the LoRA weight loading method and device for large language model reasoning, efficient loading of the LoRA weight is achieved through an efficient weight definition mode and a batch processing copying mechanism, and the reasoning efficiency of the large language model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer information processing, and in particular to a LoRA weight loading method and device for large language model inference. Background Art

[0002] In recent years, with the rapid development of deep learning technology, it has been widely used and made significant progress in the fields of natural language processing (NLP), computer vision (CV) and reinforcement learning (RL). Among them, large language models (LLMs), as representative models of deep learning, have mastered rich language knowledge and semantic information through pre-training on large-scale corpora, and can show excellent performance in a variety of downstream tasks such as text generation, machine translation, and question-answering systems.

[0003] Although large language models have a wide range of language capabilities through pre-training, fine-tuning is usually required to better adapt to the needs of specific tasks. Fine-tuning methods use a small amount of labeled data to conduct targeted training on the model, thereby improving its performance in specific tasks. However, traditional fine-tuning methods usually require adjusting a large number of parameters of the entire model, which places high demands on computing resources and time.

[0004] In order to solve the above problems, LoRA (Low-Rank Adaptation) came into being and became an efficient fine-tuning method. The core idea of ​​LoRA is to adjust only a small number of low-rank matrices based on the pre-trained model to achieve fast fine-tuning. This method not only reduces the computing resources and time required for fine-tuning, but also significantly improves the performance of the model in specific tasks.

[0005] However, the huge parameter scale of LLM also brings great challenges to reasoning performance. In the reasoning stage, it is necessary to face practical application scenarios with high throughput and low latency. In order to optimize the reasoning efficiency of LLM, the industry has developed a variety of reasoning engines (such as TensorRT-LLM, vLLM and LMDeploy), which improve reasoning performance by adopting KV Cache caching, batch processing, quantization and other technologies. However, in the existing framework, support for LoRA is still in its early stages, and its low dynamic loading efficiency has not been effectively solved.

[0006] In a typical inference system, when a new inference request is received, the corresponding LoRA weights need to be loaded into the GPU for calculation. Since weight loading and scheduling of inference tasks are usually performed sequentially, this mode may affect the throughput of the inference system. Therefore, in order to achieve efficient dynamic loading of LoRA, it is necessary to design a more optimized loading and scheduling process in the inference system to minimize loading time and thus improve inference performance.

[0007] In response to the above-mentioned problems in the prior art, the present application proposes a new LoRA weight loading method and device for large language model inference.

[0008] The above information disclosed in the Background section is only for enhancement of understanding of the background of the present application and therefore it may contain information that does not constitute the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0009] In view of this, the present application provides a LoRA weight loading method and device for large language model reasoning, which realizes efficient loading of LoRA weights through an efficient weight definition method and a batch copy mechanism, thereby improving the reasoning efficiency of large language models.

[0010] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0011] According to one aspect of the present application, a LoRA weight loading method for large language model inference is proposed, the method comprising: when the inference request of the large language model includes LoRA weight calculation, adding the inference request to a preparation queue; in the preparation queue, performing preprocessing operations on N inference requests; during the preprocessing operation, copying the LoRA weight to a continuous cache area of ​​the CPU according to preset rules; after the preprocessing operation is completed, adding the inference request to a waiting queue; when the inference request is scheduled for execution, loading the LoRA weight corresponding to the inference request into the GPU of the large language model for inference calculation.

[0012] In an exemplary embodiment of the present application, when the inference request of the large language model includes LoRA weight calculation, the inference request is added to the preparation queue, and also includes: when the inference request of the large language model does not include LoRA weight calculation, the inference request is added to the waiting queue.

[0013] In an exemplary embodiment of the present application, in a preparation queue, N inference requests are preprocessed, including: dynamically determining the number of requests to be processed N according to the system processing speed and preprocessing time; extracting the first N inference requests in the preparation queue for preprocessing.

[0014] In an exemplary embodiment of the present application, during the preprocessing operation, the LoRA weights are copied to the continuous cache area of ​​the CPU according to preset rules, including: defining the arrangement of tensors in non-continuous memory; defining the continuous storage method of the LoRA weights; preprocessing the LoRA weights; and copying the LoRA weights to the continuous cache area of ​​the CPU according to the above rules.

[0015] In an exemplary embodiment of the present application, a continuous storage method of LoRA weights is defined, including: each LoRA linear layer weight occupies a continuous column; each row is all the linear layer weights corresponding to a LoRA.

[0016] In an exemplary embodiment of the present application, the LoRA weight is preprocessed, including: configuring parameters of the LoRA weight; checking the shape of the Tensor; performing parallel segmentation processing on the LoRA weight; and standardizing the LoRA weight.

[0017] In an exemplary embodiment of the present application, the LoRA weights are copied to the continuous cache area of ​​the CPU according to the above rules, including: creating a continuous cache area in the CPU; and asynchronously copying the LoRA weights to the continuous cache area according to the above rules.

[0018] In an exemplary embodiment of the present application, when an inference request is scheduled for execution, the LoRA weights corresponding to the inference request are loaded into the GPU of a large language model for inference calculation, including: batch scheduling inference requests; when the inference request is scheduled for execution, the LoRA weights corresponding to the inference request are batch synchronously loaded into the GPU of a large language model for inference calculation.

[0019] In an exemplary embodiment of the present application, the LoRA weights corresponding to the inference request are batch-synchronously loaded into the GPU of the large language model for inference calculation, including: synchronously loading the LoRA weights into the weight structure of the large language model; the large language model performs inference calculation based on the loaded weights.

[0020] According to one aspect of the present application, a LoRA weight loading device for large language model inference is proposed, and the device includes: a preparation module, which is used to add the inference request to a preparation queue when the inference request of the large language model includes LoRA weight calculation; a preprocessing module, which is used to perform preprocessing operations on N inference requests in the preparation queue; a copy module, which is used to copy the LoRA weights to the continuous cache area of ​​the CPU according to preset rules during the preprocessing operation; a waiting module, which is used to add the inference request to the waiting queue after the preprocessing operation is completed; a loading module, which is used to load the LoRA weights corresponding to the inference request into the GPU of the large language model for inference calculation when the inference request is scheduled for execution.

[0021] According to one aspect of the present application, an electronic device is proposed, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0022] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0023] According to the LoRA weight loading method and device for large language model inference of the present application, when the LoRA weight calculation is included in the inference request of the large language model, the inference request is added to the preparation queue; in the preparation queue, the N inference requests are preprocessed; during the preprocessing operation, the LoRA weight is copied to the continuous cache area of ​​the CPU according to the preset rules; after the preprocessing operation is completed, the inference request is added to the waiting queue; when the inference request is scheduled for execution, the LoRA weight corresponding to the inference request is loaded into the GPU of the large language model for inference calculation. Through the efficient weight definition method and batch copy mechanism, efficient loading of LoRA weights is achieved, and the inference efficiency of the large language model is improved.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects, features and advantages of the present application will become more apparent by describing in detail the exemplary embodiments thereof with reference to the accompanying drawings. The accompanying drawings described below are only some embodiments of the present application, and it is clear to a person skilled in the art that other accompanying drawings can be obtained from these accompanying drawings without creative effort.

[0026] Figure 1It is a schematic diagram of the reasoning system of a large language model.

[0027] Figure 2 It is a flowchart of a LoRA weight loading method for large language model inference according to an exemplary embodiment.

[0028] Figure 3 is a flowchart of a LoRA weight loading method for large language model inference according to another exemplary embodiment.

[0029] Figure 4 is a schematic diagram of a LoRA weight loading method for large language model inference according to another exemplary embodiment.

[0030] Figure 5 is a schematic diagram of a LoRA weight loading method for large language model inference according to another exemplary embodiment.

[0031] Figure 6 is a schematic diagram of a LoRA weight loading method for large language model inference according to another exemplary embodiment.

[0032] Figure 7 is a schematic diagram of a LoRA weight loading method for large language model inference according to another exemplary embodiment.

[0033] Figure 8 It is a block diagram of a LoRA weight loading device for large language model inference according to an exemplary embodiment.

[0034] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment.

[0035] Fig.10 It is a block diagram of a computer-readable medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0037] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0039] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0040] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0041] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0042] After research, the applicant of this case found that in the original Linear module, its core is a weight tensor, and the calculation process is the matrix multiplication of the input and the weight tensor. The calculation formula is as follows: base_output = input@weight.T Here, @ represents matrix multiplication and T represents matrix transpose.

[0043] The LoRA Linear module adds two new tensors based on the weight tensor: LoRA_A and LoRA_B. The calculation process is to calculate the multiplication of two small matrices based on the original output and add the result to the original output: LoRA_output =input@LoRA_A.T@LoRA_B.T Here, @ represents matrix multiplication, and .T represents matrix transpose.

[0044] output=base_output+LoRA_output In the LLM reasoning system, the main difference between the LoRA Linear module and a single LoRA Linear is the addition of batch processing support, that is, multiple different LoRA requests can be calculated at the same time. A batch dimension is introduced in each LoRA Linear to store the weights of different LoRA types.

[0045] A typical large language model (such as Qwen2.5 7B) is composed of multiple DecoderLayer stacks, each of which contains sub-modules such as self_attn, feed forward, norm, etc. During the LoRA fine-tuning process, the Linear parameters in each DecoderLayer need to be adjusted to adapt to specific tasks and data.

[0046] An LLM model (i.e. base model) contains many Linear modules, while a LoRA model replaces all Linear modules in the base model with LoRA Linear modules. LoRA Linear modules consist of the following weights: base_layer: weight of the original Linear; LoRA_A and LoRA_B: Newly added LoRA weights.

[0047] A LoRA model consists of a set of weights for all LoRA Linears. Without merging weights, the LoRA model structure is as follows: Figure 1 shown.

[0048] When batch processing LoRA requests for inference, first calculate the output of base_layer: base_output = base_layer(input) Then for each LoRA request, calculate its corresponding LoRA output and merge: LoRA_output_i=input@LoRA_A.weight[i].T@LoRA_B.weight[i].T Here, @ represents matrix multiplication, and .T represents matrix transpose.

[0049] This process involves a lot of calculations and requires frequent loading of LoRA weights, which greatly limits the reasoning efficiency of the model.

[0050] A LoRA model contains many LoRA Linear weights. Currently, mainstream frameworks usually use a loop copy method to load LoRA models, which is inefficient. For example, take the Qwen2.57B model inference on an H20 graphics card as an example: It takes about 0.02 seconds to load the weights; The output throughput of the inference engine is about 100 tokens / s, and the forward propagation time is about 0.01 seconds.

[0051] Therefore, loading weights alone may block the inference engine for about 2 steps, resulting in a decrease in inference speed. Especially during streaming output, users will experience a brief pause.

[0052] In order to solve the above problems, this application proposes an efficient LoRA weight loading mechanism. The core optimizations include: Asynchronous processing: non-GPU operations, some operations that do not involve GPU are processed in advance; Batch copy optimization: Merge the CPU to GPU cyclic copy process into one operation to reduce the number of copies.

[0053] Through these optimizations, the time required for LoRA weight loading can be significantly reduced, thereby improving the overall efficiency of LLM reasoning and effectively improving the user experience of streaming output.

[0054] The content of this application is described in detail below with the help of specific embodiments.

[0055] Figure 2 1 is a flow chart of a LoRA weight loading method for large language model inference according to an exemplary embodiment. The LoRA weight loading method 20 for large language model inference at least includes steps S202 to S210.

[0056] like Figure 2 As shown, in S202, when the inference request of the large language model includes LoRA weight calculation, the inference request is added to the preparation queue.

[0057] In one embodiment, it further includes: when the LoRA weight calculation is not included in the inference request of the large language model, adding the inference request to the waiting queue.

[0058] In an actual application scenario, after the LLM inference system receives a new inference request, the system determines whether the request includes LoRA weight calculation: when it includes LoRA weight calculation, the request can be added to the preparation queue for subsequent preprocessing operations; when it does not include LoRA weight calculation, the request can be directly added to the waiting queue without occupying additional resources.

[0059] More specifically, by analyzing the meta-information in the inference request (such as request identifier, task type, whether it includes customization requirements), it can be quickly identified whether it is a LoRA request.

[0060] The processing method in the steps can reduce the occupation of invalid resources in the classification stage, separate the processing of LoRA weight loading from the inference tasks of non-LoRA requests, and improve the overall throughput of the system.

[0061] In S204, in the preparation queue, preprocessing operations are performed on N inference requests. For example, the number of requests to be processed N can be dynamically determined according to the system processing speed and preprocessing time; the first N inference requests in the preparation queue are extracted for preprocessing operations.

[0062] In a specific embodiment, assume that the current system processing speed is X req / s, that is, X requests are processed per second, and the time required for preprocessing each request is Ts. The average waiting time for the first N requests in the preparation queue to be scheduled is: t1 = N / X, The time required for preprocessing N requests is t2 = T.

[0063] To ensure that the preprocessing does not affect the scheduling, it should be ensured that t2 < t1, that is, T < N / X, N > T * X.

[0064] For example, when the processing speed of the LLM inference system is 10 req / s and the preprocessing time is 0.5 s, then N is set to: 10 * 0.5 = 5 In this embodiment, 5 can meet the requirements of the system processing speed.

[0065] In this application, the dynamic adjustment mechanism of N can flexibly change according to the current system load and the complexity of the inference task to ensure that the preprocessing operation does not block the scheduling of requests.

[0066] In S206, during the preprocessing operation, the LoRA weights are copied to the continuous cache area of ​​the CPU according to the preset rules. For example, the arrangement of the tensors of the non-continuous memory is defined; the continuous storage method of the LoRA weights is defined; the LoRA weights are preprocessed; and the LoRA weights are copied to the continuous cache area of ​​the CPU according to the above rules.

[0067] In actual application scenarios, you can check whether the LoRA weight already exists in the CPU cache: If the weight already exists: directly return the location in the cache and skip the loading operation. If the weight does not exist: follow the steps below to load: Define weight storage rules: Determines how non-contiguous memory tensors are arranged.

[0068] Determine the contiguous storage rules for LoRA weights (e.g., each Linear weight occupies a contiguous column, and each row represents a set of weights for a certain LoRA model).

[0069] Perform preprocessing operations: verify and standardize LoRA weights, etc.

[0070] The preprocessed LoRA weights are asynchronously loaded into the continuous cache area of ​​the CPU according to the rules.

[0071] In this application, this process optimizes the storage and loading efficiency of weights and reduces the time consumption of frequent transmission between CPU and GPU during the inference phase. In addition, in this application, the asynchronous loading and caching mechanism of weights ensures efficient resource utilization and avoids repeated loading.

[0072] In S208, after the preprocessing operation is completed, the inference request is added to the waiting queue. For the request that has completed the preprocessing, it can also be moved from the preparation queue to the waiting queue and marked as "schedulable state". The update of the request state simplifies the scheduling logic and improves the overall operation efficiency of the inference system.

[0073] In S210, when the inference request is scheduled for execution, the LoRA weight corresponding to the inference request is loaded into the GPU of the large language model for inference calculation. For example, the inference request can be scheduled in batches; when the inference request is scheduled for execution, the LoRA weight corresponding to the inference request is loaded in batches synchronously into the GPU of the large language model for inference calculation.

[0074] More specifically, the LoRA weights can be synchronously loaded into the weight structure of the large language model; the large language model performs inference calculations based on the loaded weights.

[0075] Schedule inference requests and load the corresponding LoRA weights to the GPU. By scheduling multiple requests in batches, GPU utilization is improved. For each request, its LoRA weights are batch-synchronously loaded into the GPU weight structure. Use the loaded weights for inference calculations and output the final results.

[0076] The batch scheduling mechanism reduces the overhead of weight loading and improves the GPU's inference efficiency. The logic of synchronous weight loading combined with the streaming output strategy ensures the timeliness and stability of the inference results.

[0077] Through the optimization of the above steps, the loading process of LoRA weights is refined into multiple links. The system completes complex operations in advance in the preprocessing stage. Combined with batch processing and caching strategies, the throughput and response speed of the inference system are greatly improved.

[0078] According to the LoRA weight loading method for large language model inference of the present application, when the LoRA weight calculation is included in the inference request of the large language model, the inference request is added to the preparation queue; in the preparation queue, N inference requests are preprocessed; during the preprocessing operation, the LoRA weight is copied to the continuous cache area of ​​the CPU according to preset rules; after the preprocessing operation is completed, the inference request is added to the waiting queue; when the inference request is scheduled for execution, the LoRA weight corresponding to the inference request is loaded into the GPU of the large language model for inference calculation. Through efficient weight definition method and batch copy mechanism, efficient loading of LoRA weights is achieved, and the inference efficiency of the large language model is improved.

[0079] It should be clearly understood that the present application describes how to form and use specific examples, but the principles of the present application are not limited to any details of these examples. On the contrary, based on the teaching of the content disclosed in the present application, these principles can be applied to many other embodiments.

[0080] Figure 3 is a flowchart of a LoRA weight loading method for large language model inference according to another exemplary embodiment. Figure 3 The process 30 shown is for Figure 2 A detailed description of S206 "During the preprocessing operation, copying the LoRA weights to the continuous cache area of ​​the CPU according to preset rules" in the process shown.

[0081] like Figure 3 As shown, in S302, the arrangement of tensors in non-contiguous memory is defined. In the prior art, In S304, a continuous storage method of the LoRA weights is defined. For example, each LoRA linear layer weight occupies a continuous column; for example, each row is all the linear layer weights corresponding to a LoRA.

[0082] The original storage method of LoRA weights is to define the weights of several linears as a tensor, and add a dimension to represent the index value of LoRA. The naive definition method is that all LoRA linears of each corresponding weight are arranged continuously in memory, and the weights of the same LoRA model are distributed in different LoRA linears, that is, when loading a LoRA model, the weights need to be copied to different locations in the memory.

[0083] like Figure 4 As shown, this is a weight distribution diagram of max_LoRAs=4; rank=4; in_features=6; out_features=6, where each weight is composed of several LoRAs, each row corresponds to a LoRA linear, and the number of elements in each row is rank * num_features.

[0084] In this application, non-contiguous tensors are arranged in memory as follows: Figure 5 As shown, the weight of each LoRA linear is a continuous rank*num_features column, and each row is the weight of a LoRA in all LoRA linears.

[0085] This definition method of the present application makes the weights corresponding to a LoRA model continuous in memory, and only needs to be copied once when copying from the CPU to the GPU.

[0086] In S306, the LoRA weight is preprocessed. More specifically, for example, the tensor may be verified; for example, the LoRA weight may be verified; for example, the LoRA weight may be parallel-split; and for example, the LoRA weight may be standardized.

[0087] In practical applications, for example, the shape, data type, and integrity of the Tensor can be checked to ensure that they meet the requirements of the inference task. Parameters of the weight matrix can be fused. For multi-card inference environments, weight data can be processed in blocks according to the multi-card inference format. According to the access characteristics of the target hardware, the storage location of the weights can be optimized for memory alignment to reduce memory access latency.

[0088] In this application, the preprocessing operation is decoupled from the weight loading process and can be executed asynchronously to make full use of system resources.

[0089] In S308, the LoRA weight is copied to the continuous cache area of ​​the CPU according to the above rules. For example, a continuous cache area is created in the CPU; and the LoRA weight is asynchronously copied to the continuous cache area according to the above rules.

[0090] The copying method in the prior art is as follows Figure 6 As shown in the figure, after loading the LoRA model from the file, each LoRA linear weight is an independent small tensor. When the LLM inference engine is about to schedule the use of this LoRA model, it needs to be copied to the corresponding GPU memory in a loop.

[0091] In this application, the improved method is as follows Figure 7 As shown, after loading the LoRA model from the file, you can, for example, first create a continuous CPU buffer, then copy the small LoRA linear weights to the CPU buffer tensor once, and then wait for the LLM inference engine to schedule, and then copy the CPU_buffer to the corresponding position of the GPU_buffer.

[0092] In actual scenarios, the process of copying the loaded LoRA weights to the CPU_buffer is called step 1, and the process of copying the weights from the CPU_buffer to the GPU_buffer is called step 2. In the setting of this application, since the current LLM inference engine has not been scheduled to this LoRA model, step 1 can be executed asynchronously relative to the LLM inference engine.

[0093] Step 2 is performed synchronously when the inference engine is called. Therefore, when calculating the copy overhead when the LLM inference engine is not idle, the time overhead of step 2 only needs to be considered.

[0094] The splitting of this step can reduce the frequency of GPU data transmission and concentrate the weight loading operation into the continuous CPU cache area. After the weights are stored in the CPU continuous cache area, they can be directly used for subsequent GPU batch loading, further improving the inference efficiency.

[0095] Through the above steps, this application realizes a complete optimization process from weight definition to preprocessing and cache loading. This design significantly improves the loading efficiency and system resource utilization of LLM reasoning through non-continuous weight definition and batch processing mechanism.

[0096] In a specific embodiment, the technical effect of the present application was tested on an H20 graphics card. The base models tested were Qwen2.5-7B-Instruct and Qwen2.5-72B-Instruct. Two lora ranks 8 and 16 were tested. The copy time calculation acceleration ratios of the two methods were compared. From the results, it can be seen that the acceleration effect is obvious. model rank baseline copy time (s) optimized copy time (s) Qwen2.5-7B-Instruct 8 0.021332 0.003453 Qwen2.5-7B-Instruct 16 0.021953 0.003485 Qwen2.5-72B-Instruct 8 0.053675 0.008996 Qwen2.5-72B-Instruct 16 0.054622 0.008998

[0097] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided in the present application are performed. The program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0098] In addition, it should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0099] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0100] Figure 8 is a block diagram of a LoRA weight loading device for large language model inference according to an exemplary embodiment. Figure 8 As shown, the LoRA weight loading device 80 for large language model inference includes: a preparation module 802, a preprocessing module 804, a copy module 806, a waiting module 808, and a loading module 810.

[0101] The preparation module 802 is used to add the inference request to the preparation queue when the inference request of the large language model includes LoRA weight calculation; the preparation module 802 is also used to add the inference request to the waiting queue when the inference request of the large language model does not include LoRA weight calculation.

[0102] The preprocessing module 804 is used to perform preprocessing operations on N inference requests in the preparation queue; the preprocessing module 804 is also used to dynamically determine the number of requests to be processed N according to the system processing speed and preprocessing time; and extract the first N inference requests in the preparation queue for preprocessing operations.

[0103] The copy module 806 is used to copy the LoRA weights to the continuous cache area of ​​the CPU according to preset rules during the preprocessing operation; the copy module 806 is also used to define the arrangement of tensors in non-continuous memory; define the continuous storage method of LoRA weights; preprocess the LoRA weights; and copy the LoRA weights to the continuous cache area of ​​the CPU according to the above rules.

[0104] The waiting module 808 is used to add the reasoning request to a waiting queue after the preprocessing operation is completed; The loading module 810 is used to load the LoRA weights corresponding to the inference request into the GPU of the large language model for inference calculation when the inference request is scheduled for execution. The loading module 810 is also used to batch schedule inference requests; when the inference request is scheduled for execution, the LoRA weights corresponding to the inference request are batch synchronously loaded into the GPU of the large language model for inference calculation.

[0105] According to the LoRA weight loading device for large language model inference of the present application, when the LoRA weight calculation is included in the inference request of the large language model, the inference request is added to the preparation queue; in the preparation queue, N inference requests are preprocessed; during the preprocessing operation, the LoRA weight is copied to the continuous cache area of ​​the CPU according to the preset rules; after the preprocessing operation is completed, the inference request is added to the waiting queue; when the inference request is scheduled for execution, the LoRA weight corresponding to the inference request is loaded into the GPU of the large language model for inference calculation. Through the efficient weight definition method and batch copy mechanism, efficient loading of LoRA weights is achieved, and the reasoning efficiency of the large language model is improved.

[0106] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment.

[0107] Refer to the following Fig. 9 hereinafter describes an electronic device 900 according to this embodiment of the present application. Fig. 9 The electronic device 900 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0108] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), a display unit 940, etc.

[0109] The storage unit stores a program code, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 910 can perform the following steps: Figure 2 , Figure 3 Follow the steps shown in .

[0110] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 9201 and / or a cache memory unit 9202 , and may further include a read-only memory unit (ROM) 9203 .

[0111] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0112] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0113] The electronic device 900 may also communicate with one or more external devices 900' (e.g., keyboards, pointing devices, Bluetooth devices, etc.) so that a user can communicate with the device that interacts with the electronic device 900, and / or the electronic device 900 can communicate with any device that communicates with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 950. In addition, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 960. The network adapter 960 may communicate with other modules of the electronic device 900 through a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0114] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Fig.10As shown, the technical solution according to the implementation mode of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation mode of the present application.

[0115] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0116] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0117] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0118] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by a device, the computer-readable medium implements the following functions: when the inference request of the large language model includes LoRA weight calculation, the inference request is added to the preparation queue; in the preparation queue, N inference requests are preprocessed; during the preprocessing operation, the LoRA weight is copied to the continuous cache area of ​​the CPU according to preset rules; after the preprocessing operation is completed, the inference request is added to the waiting queue; when the inference request is scheduled for execution, the LoRA weight corresponding to the inference request is loaded into the GPU of the large language model for inference calculation.

[0119] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0120] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present application.

[0121] The exemplary embodiments of the present application are specifically shown and described above. It should be understood that the present application is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present application is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A LoRA weight loading method for large language model inference, characterized in that: include: When a LoRA weight calculation is included in an inference request of a large language model, the inference request is added to a preparation queue; In the preparation queue, preprocessing operations are performed on N inference requests; During the preprocessing operation, the LoRA weights are copied to the continuous cache area of ​​the CPU according to the preset rules; After the preprocessing operation is completed, adding the inference request to a waiting queue; When an inference request is scheduled for execution, the LoRA weight corresponding to the inference request is loaded into the GPU of the large language model for inference calculation.

2. The method according to claim 1, characterized in that When the inference request of the large language model includes the LoRA weight calculation, the inference request is added to the preparation queue, and further includes: When the inference request of the large language model does not include LoRA weight calculation, the inference request is added to the waiting queue.

3. The method according to claim 1, characterized in that In the preparation queue, preprocessing operations are performed on N inference requests, including: Dynamically determine the number of pending requests N based on system processing speed and preprocessing time; The first N inference requests in the preparation queue are extracted for preprocessing operations.

4. The method according to claim 1, characterized in that During the preprocessing operation, the LoRA weights are copied to the continuous cache area of ​​the CPU according to the preset rules, including: Define the arrangement of tensors in non-contiguous memory; Define the continuous storage method of LoRA weights; Preprocess the LoRA weights; Copy the LoRA weights to the continuous cache area of ​​the CPU according to the above rules.

5. The method according to claim 4, characterized in that Defines the continuous storage method of LoRA weights, including: Each LoRA linear layer weight occupies consecutive columns; Each row is the weights of all linear layers corresponding to a LoRA.

6. The method according to claim 4, characterized in that Preprocess the LoRA weights, including: Configure parameters for LoRA weight; Tensor shape check; Parallel segmentation of LoRA weights; Normalize the LoRA weights.

7. The method according to claim 4, characterized in that Copy the LoRA weights to the continuous cache area of ​​the CPU according to the above rules, including: Create a continuous buffer area in the CPU; The LoRA weights are asynchronously copied to the continuous buffer area according to the above rules.

8. The method according to claim 1, characterized in that When an inference request is scheduled for execution, the LoRA weight corresponding to the inference request is loaded into the GPU of the large language model for inference calculation, including: Batch scheduling of inference requests; When an inference request is scheduled for execution, the LoRA weights corresponding to the inference request are synchronously loaded in batches into the GPU of the large language model for inference calculation.

9. The method according to claim 1, characterized in that The LoRA weights corresponding to the inference request are synchronously loaded in batches into the GPU of the large language model for inference calculation, including: Synchronously load the LoRA weights into the weight structure of the large language model; The large language model performs inference calculations based on the loaded weights.

10. A LoRA weight loading device for large language model inference, characterized in that: include: A preparation module, configured to add an inference request to a preparation queue when a LoRA weight calculation is included in an inference request of a large language model; A preprocessing module is used to perform preprocessing operations on N inference requests in the preparation queue; The copy module is used to copy the LoRA weights to the continuous cache area of ​​the CPU according to preset rules during the preprocessing operation; A waiting module, used for adding the inference request to a waiting queue after the preprocessing operation is completed; The loading module is used to load the LoRA weight corresponding to the inference request into the GPU of the large language model for inference calculation when the inference request is scheduled for execution.

Images