LoRA weight management method and device for large language model

Abstract

The invention provides a LoRA weight management method and device for a large language model. The method comprises the steps that an inference system of a large language model obtains LoRA weight data, wherein the LoRA weight data comprises a low-rank matrix A and a low-rank matrix B; determining a storage position of the LoRA weight data; respectively storing the low-rank matrix A and the low-rank matrix B by taking a page as a unit according to the storage position; recording a mapping relationship between the LoRA weight data and the corresponding storage page; and an inference system of the large language model calls the LoRA weight data according to the mapping relation to perform inference calculation. According to the LoRA weight management method and device for the large language model, through paging storage and mapping management of the LoRA weight, the video memory utilization rate and the calculation efficiency can be improved, flexible expansion of a large-scale model is supported, a multi-hardware environment is adapted, and the reasoning performance is optimized.

Description

Technical Field

[0001] The present application relates to the field of computer information processing, and in particular to a LoRA weight management method and device for a large language model. Background Art

[0002] In recent years, deep learning technology has made significant progress in fields such as natural language processing (NLP), computer vision (CV), and reinforcement learning (RL). Large language models (LLMs) are an important application of deep learning, focusing on processing text data. These models have mastered rich language knowledge and semantic information through pre-training on large-scale corpora, and have performed well in downstream tasks such as text generation, machine translation, and question-answering systems.

[0003] Although large language models already have strong language understanding capabilities during the pre-training phase, fine-tuning is often required to further improve performance in specific tasks. Fine-tuning refers to retraining a small amount of data for a specific task based on the pre-trained model, so that the model is more suitable for the task requirements. However, traditional fine-tuning methods require updating a large number of model parameters, consuming a lot of computing resources and increasing the complexity of storage and deployment.

[0004] To solve these problems, LoRA (Low-Rank Adaptation) was proposed as an efficient fine-tuning technology. The core idea of ​​LoRA is to achieve fast fine-tuning by adding a low-rank matrix with a small number of parameters on the basis of the pre-trained model. This method avoids direct modification of large-scale model parameters, greatly reduces the resource consumption required for fine-tuning, while maintaining a high performance improvement effect.

[0005] However, in the LoRA fine-tuning process, in order to add adjustments to the original calculation logic of the model, it is necessary to introduce additional structures on the basis of the traditional model. These newly added low-rank matrices supplement and optimize the output of the original model by adjusting a small number of parameters, thereby achieving the fine-tuning effect. When processing LoRA requests in batches, this process places high demands on computing resources and time, especially when it is necessary to support parallel reasoning of multiple LoRA requests. The efficiency of the existing technology's computing method has become a key bottleneck limiting the reasoning performance of the LoRA model.

[0006] Therefore, a new LoRA weight management method and device for a large language model is needed.

[0007] The above information disclosed in the Background section is only for enhancement of understanding of the background of the present application and therefore it may contain information that does not constitute the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0008] In view of this, the present application provides a LoRA weight management method and device for a large language model. By managing LoRA weights through paging storage and mapping, it can improve video memory utilization and computing efficiency, support flexible expansion of large-scale models, adapt to multiple hardware environments, and optimize reasoning performance.

[0009] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0010] According to one aspect of the present application, a LoRA weight management method for a large language model is proposed, the method comprising: an inference system of the large language model obtains LoRA weight data, the LoRA weight data comprising: a low-rank matrix A and a low-rank matrix B; determining a storage location of the LoRA weight data; storing the low-rank matrix A and the low-rank matrix B in units of pages according to the storage location; recording a mapping relationship between the LoRA weight data and its corresponding storage page; the inference system of the large language model calls the LoRA weight data according to the mapping relationship to perform inference calculations.

[0011] In an exemplary embodiment of the present application, determining the storage location of the LoRA weight data includes: determining the storage location of the LoRA weight data based on system parameters of a large language model inference system, storage page parameters, and characteristics of the LoRA weight data.

[0012] In an exemplary embodiment of the present application, the storage location of the LoRA weight data is determined according to the system parameters of the large language model inference system, the storage page parameters, and the characteristics of the LoRA weight data, including: determining the storage page size; determining the storage location of the LoRA weight data according to the maximum number of LoRA layers of the large language model inference system, the storage page size, and the dimension of the LoRA weight data.

[0013] In an exemplary embodiment of the present application, determining a storage page size includes: determining the storage page size according to a matrix size of a tensor core in an inference system of a large language model.

[0014] In an exemplary embodiment of the present application, the storage location of the LoRA weight data is determined according to the maximum number of LoRA layers of the large language model inference system, the storage page size, and the dimension of the LoRA weight data, including: the position of the low-rank matrix A is set to: (maximum number of LoRA layers, storage page size, input feature dimension of the low-rank matrix A); the position of the low-rank matrix B is set to: (maximum number of LoRA layers, output feature dimension of the low-rank matrix B, storage page size).

[0015] In an exemplary embodiment of the present application, recording the mapping relationship between LoRA weight data and its corresponding storage page includes: setting a page table to record the mapping relationship between the low-rank matrix A and the low-rank matrix B and their corresponding storage pages.

[0016] In an exemplary embodiment of the present application, the reasoning system of the large language model calls the LoRA weight data according to the mapping relationship to perform reasoning calculations, including: the reasoning system of the large language model calls the LoRA weight data page by page to perform reasoning calculations based on the mapping relationship; and summarizes the calculation results of all pages to generate a final reasoning result.

[0017] In an exemplary embodiment of the present application, the reasoning system of the large language model calls the LoRA weight data page by page based on the mapping relationship to perform reasoning calculations, including: the reasoning system of the large language model calls the low-rank matrix A and the low-rank matrix B in the LoRA weight data page by page; and performs reasoning calculations based on the low-rank matrix A and the low-rank matrix B.

[0018] In an exemplary embodiment of the present application, reasoning calculation is performed based on the low-rank matrix A and the low-rank matrix B, including: matrix multiplication of input data with corresponding pages of the low-rank matrix A and the low-rank matrix B.

[0019] According to one aspect of the present application, a LoRA weight management device for a large language model is proposed, the device comprising: a data module, used for an inference system of a large language model to obtain LoRA weight data, the LoRA weight data comprising: a low-rank matrix A and a low-rank matrix B; a position module, used to determine the storage location of the LoRA weight data; a storage module, used to store the low-rank matrix A and the low-rank matrix B in units of pages according to the storage location; a mapping module, used to record the mapping relationship between the LoRA weight data and its corresponding storage page; an inference module, used for the inference system of the large language model to call the LoRA weight data according to the mapping relationship for inference calculation.

[0020] According to one aspect of the present application, an electronic device is proposed, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0021] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0022] According to the LoRA weight management method and device for a large language model of the present application, LoRA weight data is obtained through the reasoning system of the large language model, and the LoRA weight data includes: a low-rank matrix A and a low-rank matrix B; the storage location of the LoRA weight data is determined; the low-rank matrix A and the low-rank matrix B are stored in units of pages according to the storage location; the mapping relationship between the LoRA weight data and its corresponding storage page is recorded; the reasoning system of the large language model calls the LoRA weight data according to the mapping relationship to perform reasoning calculations, and manages the LoRA weights through paging storage and mapping, which can improve the utilization rate of video memory and computing efficiency, support flexible expansion of large-scale models, adapt to multiple hardware environments, and optimize reasoning performance.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the present application will become more apparent by describing in detail the exemplary embodiments thereof with reference to the accompanying drawings. The accompanying drawings described below are only some embodiments of the present application, and it is clear to a person skilled in the art that other accompanying drawings can be obtained from these accompanying drawings without creative effort.

[0025] Figure 1 It is a structural diagram of LoRA weight in the prior art.

[0026] Figure 2 This is a schematic diagram of the storage of LoRA weights in the prior art.

[0027] Figure 3 is a flowchart of a LoRA weight management method for a large language model according to another exemplary embodiment.

[0028] Figure 4 is a schematic diagram of a LoRA weight management method for a large language model according to another exemplary embodiment.

[0029] Figure 5 is a schematic diagram of a LoRA weight management method for a large language model according to another exemplary embodiment.

[0030] Figure 6 is a flowchart of a LoRA weight management method for a large language model according to another exemplary embodiment.

[0031] Figure 7 is a schematic diagram of a LoRA weight management method for a large language model according to another exemplary embodiment.

[0032] Figure 8 It is a block diagram of a LoRA weight management device for a large language model according to an exemplary embodiment.

[0033] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment.

[0034] Fig.10 It is a block diagram of a computer-readable medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0037] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0038] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0039] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0040] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0041] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0042] After research, the applicant of this case found that in the original Linea structure, the module contained a weight tensor, and the calculation process obtained the output through matrix multiplication of input and weight (@ represents matrix multiplication, T represents matrix transposition): base_output=input@weight.T In the existing technology, the structure of LoRA Linear adds two new tensors on the basis of the original Linear: LoRA_A (low-rank matrix A) and LoRA_B (low-rank matrix B). The calculation process is to perform two additional small-scale matrix multiplications on the basis of base_output to calculate the LoRA output, and add it to base_output to get the final output result: LoRA_output=input@LoRA_A.T@LoRA_B.T output=base_output+LoRA_output In the LLM reasoning system, in order to support batch processing (i.e., calculating multiple different LoRA requests at the same time), each LoRALinear adds a batch dimension to store multiple LoRA models.

[0043] like Figure 1 As shown, an LLM model (base model) contains many Linea layers, while the LoRA model replaces all Linear layers in the basemodel with LoRA Linear. LoRA Linear includes the following three main weights: base_layer: the initial Linear weight; LoRA_A and LoRA_B: Newly added weights for LoRA.

[0044] The structure of a LoRA model is composed of a collection of all LoRA Linear weights. For the LoRA model without combined weights, the inference calculation process is as follows: Calculate the output of base_layer: base_output = base_layer(input) Calculate the output for each LoRA request one by one: LoRA_output_i=input@LoRA_A.weight[i].T@LoRA_B.weight[i].T Add base_output to the output of each LoRA request: output_i=base_output+LoRA_output_i This batch processing method requires a lot of computing resources, especially when there are multiple LoRA requests at the same time, which significantly increases the inference time and video memory requirements, limiting the inference efficiency of the LoRA model.

[0045] The applicant of this case found that in the LLM reasoning system, the rank of LoRA is usually not fixed, and may be 2, 4, 8, 16 or 32. In order to support reasoning of LoRA models of different ranks, a common processing method is to pad all LoRA models to the maximum supported rank. This makes the weight shape of all LoRA models consistent: LoRA_A:(max_LoRAs,rank,in_features) LoRA_B:(max_LoRAs,out_features,rank) like Figure 2 As shown in the figure, in the filling operation, 0 is used as the filling value (the gray part represents the filling), and these fillings will not affect the calculation results. However, this method has significant disadvantages: Video memory waste: For example, when the maximum supported rank is 32, and the actually calculated LoRA models are all rank=8, the video memory utilization is only 25%.

[0046] Additional computational burden: The padding operation adds unnecessary computational effort, further reducing efficiency.

[0047] In order to solve the above technical problems, this disclosure proposes a LoRA weight management method for a large language model. By designing a page-based weight management method, this solution has significant technical advantages in the following aspects: Improved video memory utilization: Based on the page management method, the weight of the LoRA model is stored and managed in pages, avoiding the waste of video memory caused by the traditional padding method. The size of each page is dynamically allocated according to actual needs, allowing the system to use video memory resources more accurately, thereby greatly improving video memory utilization.

[0048] Support dynamic rank adjustment: The page-based weight management method allows the system to flexibly support LoRA models of different ranks without the need to uniformly pad to the maximum rank. This flexibility not only reduces the memory usage, but also reduces unnecessary computing overhead, providing more efficient performance support for inference tasks.

[0049] Improve concurrency: By dividing the weights into pages and managing them independently, the system can handle more LoRA requests at the same time. This design significantly improves the maximum concurrency of the system and meets the high throughput requirements of large-scale models in practical applications.

[0050] Reduce computing resource requirements: The dynamic allocation and calling of weight pages allows the inference system to only load necessary weight data during LoRA request processing, avoiding redundant calculations and effectively reducing computing resource requirements.

[0051] Through the above optimization, the disclosed solution achieves comprehensive improvements in video memory utilization, system throughput, and resource consumption, and significantly enhances the efficiency and stability of the LLM reasoning system in practical applications.

[0052] The content of this application is described in detail below with the help of specific embodiments.

[0053] Figure 3 1 is a flow chart showing a method for managing LoRA weights for a large language model according to an exemplary embodiment. The method 30 for managing LoRA weights for a large language model at least includes steps S302 to S310.

[0054] like Figure 3 As shown, in S302, the reasoning system of the large language model obtains LoRA weight data, and the LoRA weight data includes: a low-rank matrix A and a low-rank matrix B. The low-rank matrices A and B are used to effectively fine-tune the pre-trained model based on the low-rank adaptation mechanism.

[0055] In S304, the storage location of the LoRA weight data is determined. The overall architecture of the large language model inference system, storage resources, and specific characteristics of the LoRA weights are comprehensively considered to ensure the efficiency and availability of data storage.

[0056] For example, the storage location of the LoRA weight data can be determined according to the system parameters of the large language model reasoning system, the storage page parameters, and the characteristics of the LoRA weight data. The weights are stored in pages according to the storage page parameters of the system and the characteristics of the LoRA weights. For example, based on the system parameters and storage page parameters of the large language model reasoning system, the LoRA weight matrix is ​​allocated to different storage pages according to its dimensions and characteristics, thereby reducing conflicts in the storage process and improving the utilization efficiency of memory resources.

[0057] In S306, the low-rank matrix A and the low-rank matrix B are stored in units of pages according to the storage locations.

[0058] More specifically, the storage location of the LoRA weight data is determined according to the system parameters of the large language model inference system, the storage page parameters, and the characteristics of the LoRA weight data, including: determining the storage page size; determining the storage location of the LoRA weight data according to the maximum number of LoRA layers of the large language model inference system, the storage page size, and the dimension of the LoRA weight data.

[0059] In one embodiment, the storage page size may be determined according to the matrix size of the tensor core in the reasoning system of the large language model, thereby achieving a balanced optimization of computing resources and storage resources.

[0060] In one embodiment, the position of the low-rank matrix A can be set to: (maximum number of LoRA layers, storage page size, input feature dimension of the low-rank matrix A): LoRA_A:(max_LoRAs2,page_size,in_features).

[0061] In one embodiment, the position of the low-rank matrix B can be set to: (maximum number of LoRA layers, output feature dimension of the low-rank matrix B, storage page size): LoRA_B:(max_LoRAs2,out_features,page_size).

[0062] Figure 4 is a schematic diagram of a LoRA weight management method for a large language model according to another exemplary embodiment.

[0063] max_LoRAs=8, max_rank=4, in_features=6, At this time, there are three LoRA models in the system, LoRA_0 rank is 4 (the entire first row), LoRA_1 rank is 2 (half of the second row, light gray), and LoRA_2 rank is 2 (half of the third row, dark gray); After the LoRA model padding in the prior art, the storage in the memory is as follows Figure 4 As shown, the gray part is padding and the white part is unused video memory. Overall, excluding the unused part, the effective utilization rate of this storage method is only 4 / 6.

[0064] According to the method in this application, the video memory distribution based on page management is as follows Figure 5 As shown, under the technical setting of this application: page_size=2, LoRA_0 rank is 4 and occupies two pages, LoRA_1 and LoRA_2 both have rank 2 and occupy one page. According to the technical settings in this application, the video memory that is not actually used is not included, and the actual effective video memory utilization rate is 100%. This method of this application greatly improves the video memory utilization rate.

[0065] In S308, the mapping relationship between the LoRA weight data and its corresponding storage page is recorded. For example, a page table can be set to record the mapping relationship between the low-rank matrix A and the low-rank matrix B and their corresponding storage pages. This mapping mechanism can ensure that the reasoning system can quickly and accurately access the corresponding weight data when needed, and avoid unnecessary data retrieval conflicts during the reasoning process.

[0066] In S310, the reasoning system of the large language model calls the LoRA weight data for reasoning calculation according to the mapping relationship. The reasoning system of the large language model calls the LoRA weight data page by page for reasoning calculation based on the mapping relationship; and summarizes the calculation results of all pages to generate a final reasoning result.

[0067] According to the mapping relationship, the inference system calls the LoRA weight data page by page and performs calculation operations such as matrix multiplication. During the calculation process of each page, the system merges the output of the low-rank matrices A and B with the base layer output of the pre-trained model to generate the final inference result. After the calculation results of all pages are aggregated, the final inference output is generated to meet the requirements of efficient and high-concurrency inference.

[0068] Through this page management and mapping mechanism, the large language model reasoning system can significantly improve reasoning efficiency, reduce video memory waste, increase system throughput, and support more concurrent reasoning tasks.

[0069] According to the LoRA weight management method for a large language model of the present application, LoRA weight data is obtained through the reasoning system of the large language model, and the LoRA weight data includes: a low-rank matrix A and a low-rank matrix B; the storage location of the LoRA weight data is determined; the low-rank matrix A and the low-rank matrix B are stored in units of pages according to the storage location; the mapping relationship between the LoRA weight data and its corresponding storage page is recorded; the reasoning system of the large language model calls the LoRA weight data according to the mapping relationship to perform reasoning calculations, and manages the LoRA weights through paging storage and mapping, which can improve video memory utilization and computing efficiency, support flexible expansion of large-scale models, adapt to multiple hardware environments, and optimize reasoning performance.

[0070] It should be clearly understood that the present application describes how to form and use specific examples, but the principles of the present application are not limited to any details of these examples. On the contrary, based on the teaching of the content disclosed in the present application, these principles can be applied to many other embodiments.

[0071] Figure 6 is a flowchart of a LoRA weight management method for a large language model according to another exemplary embodiment. Figure 6 The process 60 shown is for Figure 3 A detailed description of S310 "the reasoning system of the large language model calls the LoRA weight data according to the mapping relationship to perform reasoning calculation" in the process shown.

[0072] like Figure 6 As shown, in S602, the reasoning system of the large language model retrieves the low-rank matrix A and the low-rank matrix B in the LoRA weight data page by page. The reasoning system reads the data of the low-rank matrices A and B from the storage page by page according to the mapping relationship of the pages based on the pre-recorded page table information. This process ensures that each page of weight data can be quickly loaded into the system through an effective memory management mechanism, thereby optimizing the use of storage bandwidth and avoiding data access bottlenecks.

[0073] In S604, an inference calculation is performed based on the low-rank matrix A and the low-rank matrix B. Matrix multiplication may be performed on the input data with a corresponding page of the low-rank matrix A and a corresponding page of the low-rank matrix B.

[0074] The system performs matrix multiplication operations on the input data with the corresponding pages of low-rank matrices A and B. Specifically, the inference system first multiplies the current page of low-rank matrix A with the input data, and then multiplies the result with the current page of low-rank matrix B. This process utilizes low-rank adaptation technology, which significantly reduces the consumption of computing resources by adjusting low-rank matrices A and B while maintaining inference accuracy.

[0075] In S606 , the calculation results of all pages are summarized to generate a final inference result.

[0076] Since the calculation of the LoRA model involves multiple pages of data, the calculation results of each page will be completed one by one in the reasoning system. The reasoning system will obtain the calculation output of the current page at each step and summarize these results. Finally, the calculation results of all pages will be combined to generate the final reasoning result, which is the prediction or output of the input data after fine-tuning the large language model. Through this process, the large language model can efficiently complete the reasoning task while maximizing the use of storage and computing resources, improving the reasoning speed and throughput.

[0077] Figure 7 FIG. 1 is a schematic diagram of a LoRA weight management method for a large language model according to another exemplary embodiment. Figure 7 As shown, assuming a LoRA linear rank = 4, the LoRA output calculation formula is: LoRA_output = input @LoRA_A.T @LoRA_B.T After it is stored in two pages respectively, it can be calculated twice.

[0078] Calculation of the first page: LoRA_output_0=input@LoRA_A[block_table[0]].T@LoRA_B[block_table[0]].T Calculation of the second page: LoRA_output_1=input@LoRA_A[block_table[1]].T@LoRA_B[block_table[1]].T The final result is to add: LoRA_output=LoRA_output_0+LoRA_output_1 The essence of its computational equivalence is that a LoRA model with a larger rank can be divided into several LoRA models with smaller ranks. For the page_size setting, in the above example, page_size is set to 2. In practice, it is recommended to set page_size to at least 8. This is because in the CUDA system, tensor cores are used to accelerate matrix multiplication, but using tensor cores usually requires the size of the mxnxk matrix to be at least 8, otherwise padding is required to reduce the calculation speed.

[0079] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided in the present application are performed. The program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0080] In addition, it should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0081] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0082] Figure 8 is a block diagram of a LoRA weight management device for a large language model according to an exemplary embodiment. Figure 8 As shown, the LoRA weight management device 80 for a large language model includes: a data module 802 , a location module 804 , a storage module 806 , a mapping module 808 , and an inference module 810 .

[0083] The data module 802 is used for the inference system of the large language model to obtain LoRA weight data, and the LoRA weight data includes: a low-rank matrix A and a low-rank matrix B; The location module 804 is used to determine the storage location of the LoRA weight data; the location module 804 is also used to determine the storage location of the LoRA weight data based on the system parameters of the large language model inference system, the storage page parameters, and the characteristics of the LoRA weight data.

[0084] The storage module 806 is used to store the low-rank matrix A and the low-rank matrix B in units of pages according to the storage location; The mapping module 808 is used to record the mapping relationship between the LoRA weight data and its corresponding storage page; the mapping module 808 is also used to set a page table to record the mapping relationship between the low-rank matrix A and the low-rank matrix B and their corresponding storage pages.

[0085] The reasoning module 810 is used by the reasoning system of the large language model to call the LoRA weight data for reasoning calculation according to the mapping relationship. The reasoning module 810 is also used by the reasoning system of the large language model to call the LoRA weight data page by page for reasoning calculation based on the mapping relationship; and to summarize the calculation results of all pages to generate a final reasoning result.

[0086] According to the LoRA weight management device for a large language model of the present application, LoRA weight data is obtained through the reasoning system of the large language model, and the LoRA weight data includes: a low-rank matrix A and a low-rank matrix B; the storage location of the LoRA weight data is determined; the low-rank matrix A and the low-rank matrix B are stored in units of pages according to the storage location; the mapping relationship between the LoRA weight data and its corresponding storage page is recorded; the reasoning system of the large language model calls the LoRA weight data according to the mapping relationship to perform reasoning calculations, and manages the LoRA weights through paging storage and mapping, which can improve the utilization rate of video memory and computing efficiency, support flexible expansion of large-scale models, adapt to multiple hardware environments, and optimize reasoning performance.

[0087] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment.

[0088] Refer to the following Fig. 9 hereinafter describes an electronic device 900 according to this embodiment of the present application. Fig. 9 The electronic device 900 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0089] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), a display unit 940, etc.

[0090] The storage unit stores a program code, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 910 can perform the following steps: Figure 3 , Figure 6 Follow the steps shown in .

[0091] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 9201 and / or a cache memory unit 9202 , and may further include a read-only memory unit (ROM) 9203 .

[0092] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0093] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0094] The electronic device 900 may also communicate with one or more external devices 900' (e.g., keyboards, pointing devices, Bluetooth devices, etc.) so that a user can communicate with the device that interacts with the electronic device 900, and / or the electronic device 900 can communicate with any device that communicates with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 950. In addition, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 960. The network adapter 960 may communicate with other modules of the electronic device 900 through a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0095] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Fig.10 As shown, the technical solution according to the implementation mode of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation mode of the present application.

[0096] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0097] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0098] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0099] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by a device, the computer-readable medium realizes the following functions: the reasoning system of the large language model obtains LoRA weight data, and the LoRA weight data includes: low-rank matrix A and low-rank matrix B; determines the storage location of the LoRA weight data; stores the low-rank matrix A and the low-rank matrix B in units of pages according to the storage location; records the mapping relationship between the LoRA weight data and its corresponding storage page; the reasoning system of the large language model calls the LoRA weight data according to the mapping relationship to perform reasoning calculation.

[0100] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0101] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present application.

[0102] The exemplary embodiments of the present application are specifically shown and described above. It should be understood that the present application is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present application is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A LoRA weight management method for a large language model, characterized in that: include: The inference system of the large language model obtains LoRA weight data, and the LoRA weight data includes: a low-rank matrix A and a low-rank matrix B; Determine the storage location of the LoRA weight data; According to the storage location, the low-rank matrix A and the low-rank matrix B are stored in units of pages respectively; Record the mapping relationship between LoRA weight data and its corresponding storage page; The reasoning system of the large language model calls the LoRA weight data according to the mapping relationship to perform reasoning calculation.

2. The method according to claim 1, characterized in that Determining the storage location of the LoRA weight data includes: The storage location of the LoRA weight data is determined according to the system parameters of the large language model inference system, the storage page parameters, and the characteristics of the LoRA weight data.

3. The method according to claim 2, characterized in that Determining the storage location of the LoRA weight data according to the system parameters of the large language model inference system, the storage page parameters, and the characteristics of the LoRA weight data includes: Determine the storage page size; The storage location of the LoRA weight data is determined according to the maximum number of LoRA layers, the storage page size, and the dimension of the LoRA weight data of the large language model inference system.

4. The method according to claim 3, characterized in that Determine the storage page size, including: The storage page size is determined according to the matrix size of the tensor core in the reasoning system of the large language model.

5. The method according to claim 3, characterized in that Determining the storage location of the LoRA weight data according to the maximum number of LoRA layers of the large language model inference system, the storage page size, and the dimension of the LoRA weight data includes: The position of the low-rank matrix A is set to: (LoRA maximum number of layers, storage page size, input feature dimension of the low-rank matrix A); The position of the low-rank matrix B is set to: (maximum number of LoRA layers, output feature dimension of the low-rank matrix B, storage page size).

6. The method according to claim 1, characterized in that Record the mapping relationship between LoRA weight data and its corresponding storage page, including: A page table is set to record the mapping relationship between the low-rank matrix A and the low-rank matrix B and their corresponding storage pages.

7. The method according to claim 1, characterized in that The reasoning system of the large language model calls the LoRA weight data according to the mapping relationship to perform reasoning calculation, including: The reasoning system of the large language model calls the LoRA weight data page by page to perform reasoning calculation based on the mapping relationship; The calculation results of all pages are aggregated to generate the final inference result.

8. The method according to claim 7, characterized in that The reasoning system of the large language model calls the LoRA weight data page by page based on the mapping relationship to perform reasoning calculation, including: The reasoning system of the large language model retrieves the low-rank matrix A and the low-rank matrix B in the LoRA weight data page by page; Inference calculation is performed based on the low-rank matrix A and the low-rank matrix B.

9. The method according to claim 8, characterized in that Performing inference calculation based on the low-rank matrix A and the low-rank matrix B includes: Matrix multiplication of the input data with the corresponding page of low-rank matrix A and the corresponding page of low-rank matrix B.

10. A LoRA weight management device for a large language model, characterized in that: include: A data module, used for the inference system of the large language model to obtain LoRA weight data, wherein the LoRA weight data includes: a low-rank matrix A and a low-rank matrix B; A location module, used to determine the storage location of the LoRA weight data; A storage module, used to store the low-rank matrix A and the low-rank matrix B in units of pages according to the storage location; Mapping module, used to record the mapping relationship between LoRA weight data and its corresponding storage page; An inference module, an inference system for a large language model, calls the LoRA weight data according to the mapping relationship to perform inference calculation.

Images