Lora weight management method and device for large language model

Abstract

The present disclosure provides a LoRA weight management method and device for a large language model. The method comprises: an inference system of the large language model acquiring LoRA weight data, the LoRA weight data comprising: a low-rank matrix A and a low-rank matrix B; determining a storage location of the LoRA weight data; storing the low-rank matrix A and the low-rank matrix B in units of pages according to the storage location; recording a mapping relationship between the LoRA weight data and the corresponding storage page; and the inference system of the large language model calling the LoRA weight data according to the mapping relationship to perform inference calculation. The LoRA weight management method and device for a large language model disclosed in the present application can improve the utilization rate of video memory and the calculation efficiency by paging storage and mapping management of LoRA weight, support flexible expansion of large-scale models, adapt to multiple hardware environments, and optimize inference performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer information processing, and in particular, to a LoRA weight management method and device for large language models. BACKGROUND

[0002] In recent years, deep learning technology has made significant progress in natural language processing (NLP), computer vision (CV), and reinforcement learning (RL) fields. Large language models (LLMs) are an important application of deep learning, focusing on processing text data. These models have mastered rich language knowledge and semantic information through pre-training on large-scale corpora, and have performed well in downstream tasks such as text generation, machine translation, and question answering systems.

[0003] Although large language models have strong language understanding ability in the pre-training stage, they often need to be further fine-tuned to improve performance in specific tasks. Fine-tuning refers to retraining a small amount of data on the basis of a pre-trained model for specific tasks, so that the model is more adapted to task requirements. However, traditional fine-tuning methods require updating a large number of model parameters, consuming a large amount of computing resources and increasing the complexity of storage and deployment.

[0004] To solve these problems, LoRA (Low-Rank Adaptation) is proposed as an efficient fine-tuning technology. The core idea of LoRA is to add a small number of low-rank matrices to the pre-trained model to achieve rapid fine-tuning. This method avoids direct modification of large-scale model parameters, significantly reducing the resource consumption required for fine-tuning, while maintaining high performance improvement effect.

[0005] However, in the LoRA fine-tuning process, in order to add adjustments to the original model's calculation logic, additional structures need to be introduced based on the traditional model. These newly added low-rank matrices adjust a small number of parameters to supplement and optimize the output of the original model, thereby achieving fine-tuning effect. In batch processing LoRA requests, this process requires high computing resources and time, especially in the case of supporting multiple LoRA requests in parallel inference, the efficiency of the existing technology's calculation method becomes a key bottleneck limiting the inference performance of the LoRA model.

[0006] Therefore, a new LoRA weight management method and device for large language models are needed.

[0007] The above information disclosed in the background section is only used to strengthen the understanding of the background of the present application, and therefore it can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0008] Therefore, the application provides a LoRA weight management method and device for a large language model. The LoRA weight is managed through page storage and mapping, which can improve the utilization of video memory and computing efficiency, support flexible expansion of large-scale models, adapt to multiple hardware environments, and optimize inference performance.

[0009] Other characteristics and advantages of the application will become apparent from the detailed description that follows, or can be learned by practice of the application.

[0010] According to an aspect of the application, a LoRA weight management method for a large language model is provided. The method comprises: an inference system of the large language model obtaining LoRA weight data, the LoRA weight data comprising: a low-rank matrix A and a low-rank matrix B; determining a storage location of the LoRA weight data; storing the low-rank matrix A and the low-rank matrix B in units of pages according to the storage location; recording a mapping relationship between the LoRA weight data and its corresponding storage page; and the inference system of the large language model calling the LoRA weight data according to the mapping relationship for inference calculation.

[0011] In an exemplary embodiment of the application, determining the storage location of the LoRA weight data comprises: determining the storage location of the LoRA weight data according to system parameters of the inference system of the large language model, storage page parameters, and characteristics of the LoRA weight data.

[0012] In an exemplary embodiment of the application, determining the storage location of the LoRA weight data according to system parameters of the inference system of the large language model, storage page parameters, and characteristics of the LoRA weight data comprises: determining a storage page size; and determining the storage location of the LoRA weight data according to a maximum number of LoRA layers of the inference system of the large language model, the storage page size, and a dimension of the LoRA weight data.

[0013] In an exemplary embodiment of the application, determining the storage page size comprises: determining the storage page size according to a matrix size of a tensor core in the inference system of the large language model.

[0014] In an exemplary embodiment of the application, determining the storage location of the LoRA weight data according to a maximum number of LoRA layers of the inference system of the large language model, a storage page size, and a dimension of the LoRA weight data comprises: setting a position of the low-rank matrix A as (the maximum number of LoRA layers, the storage page size, an input feature dimension of the low-rank matrix A); and setting a position of the low-rank matrix B as (the maximum number of LoRA layers, an output feature dimension of the low-rank matrix B, the storage page size).

[0015] In an example embodiment of the present application, the mapping relationship between the LoRA weight data and the corresponding storage page is recorded, including: setting a page table to record the mapping relationship between the low-rank matrix A and the low-rank matrix B and the corresponding storage page.

[0016] In an example embodiment of the present application, the inference system of the large language model calls the LoRA weight data for inference calculation according to the mapping relationship, including: the inference system of the large language model calls the LoRA weight data page by page for inference calculation based on the mapping relationship; and the calculation results of all pages are summarized to generate a final inference result.

[0017] In an example embodiment of the present application, the inference system of the large language model calls the LoRA weight data page by page for inference calculation based on the mapping relationship, including: the inference system of the large language model calls the low-rank matrix A and the low-rank matrix B in the LoRA weight data page by page; and performs inference calculation based on the low-rank matrix A and the low-rank matrix B.

[0018] In an example embodiment of the present application, the inference calculation based on the low-rank matrix A and the low-rank matrix B includes: performing matrix multiplication of input data with the corresponding page of the low-rank matrix A and the corresponding page of the low-rank matrix B.

[0019] According to an aspect of the present application, a LoRA weight management device for a large language model is provided, which includes: a data module for an inference system of a large language model to obtain LoRA weight data, the LoRA weight data including: a low-rank matrix A and a low-rank matrix B; a position module for determining the storage position of the LoRA weight data; a storage module for storing the low-rank matrix A and the low-rank matrix B in page units according to the storage position; a mapping module for recording the mapping relationship between the LoRA weight data and the corresponding storage page; and an inference module for the inference system of the large language model to call the LoRA weight data for inference calculation according to the mapping relationship.

[0020] According to an aspect of the present application, an electronic device is provided, which includes: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0021] According to an aspect of the present application, a computer readable medium having a computer program stored thereon is provided, which, when executed by a processor, implements the method as described above.

[0022] According to the LoRA weight management method and device for a large language model provided in the present application, LoRA weight data is obtained through a reasoning system of the large language model, the LoRA weight data comprising a low-rank matrix A and a low-rank matrix B; a storage location of the LoRA weight data is determined; the low-rank matrix A and the low-rank matrix B are respectively stored in units of pages according to the storage location; a mapping relationship between the LoRA weight data and the corresponding storage page thereof is recorded; and the LoRA weight data is called by the reasoning system of the large language model according to the mapping relationship to perform reasoning calculation, so that the LoRA weight is managed through page storage and mapping, the utilization rate of the video memory and the calculation efficiency are improved, large-scale model flexible expansion is supported, multiple hardware environments are adapted, and the reasoning performance is optimized.

[0023] It should be understood that the foregoing general description and the following detailed description are only examples and are not limiting the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0025] Figure 1 is a structure diagram of the LoRA weight in the prior art.

[0026] Figure 2 is a storage diagram of the LoRA weight in the prior art.

[0027] Figure 3 is a flowchart of a LoRA weight management method for a large language model according to another example embodiment.

[0028] Figure 4 is a schematic diagram of a LoRA weight management method for a large language model according to another example embodiment.

[0029] Figure 5 is a schematic diagram of a LoRA weight management method for a large language model according to another example embodiment.

[0030] Figure 6 is a flowchart of a LoRA weight management method for a large language model according to another example embodiment.

[0031] Figure 7 is a schematic diagram of a LoRA weight management method for a large language model according to another example embodiment.

[0032] Figure 8 is a block diagram of a LoRA weight management apparatus for a large language model according to an exemplary embodiment.

[0033] Figure 9 is a block diagram of an electronic device according to an exemplary embodiment.

[0034] Figure 10 is a block diagram of a computer readable medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings; however, the example embodiments can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the specification. Like components will not be described repeatedly with like reference numerals.

[0036] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the application can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, and operations have not been shown or described in detail to avoid obscuring aspects of the application.

[0037] The block diagrams in the accompanying drawings are only block diagrams and do not necessarily correspond to physically independent entities. That is, the functional entities can be implemented in software, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0038] The flow diagrams in the accompanying drawings are only exemplary and do not necessarily include all of the content and operations / steps, and are not necessarily performed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so that the actual order of execution can be changed according to the actual situation.

[0039] It should be understood that although the terms first, second, third, etc. can be used herein to describe various components, these components should not be limited by these terms. These terms are used only to distinguish one component from another. Thus, a first component discussed below could be termed a second component without departing from the teachings of the present application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0040] Those skilled in the art can understand that the modules or flows in the drawings are not necessarily required for implementing the present application, and therefore cannot be used to limit the protection scope of the present application.

[0041] The applicant has found that in the original Linea structure, the module contains a weight tensor, and the calculation process obtains the output by matrix multiplication of input and weight (@ represents matrix multiplication, and T represents matrix transpose):

[0042] base_output=input@weight.T

[0043] In the prior art, the structure of LoRA Linear is to add two new tensors, LoRA_A (low-rank matrix A) and LoRA_B (low-rank matrix B), on the basis of the original Linear. The calculation process is to additionally perform two small-scale matrix multiplications on the basis of base_output to calculate LoRA output, and add it to base_output to obtain the final output result:

[0044] LoRA_output=input@LoRA_A.T@LoRA_B.T

[0045] output=base_output+LoRA_output

[0046] In the LLM inference system, in order to support batch processing (i.e., simultaneously calculating multiple different LoRA requests), each LoRA Linear adds a batch dimension for storing multiple LoRA models.

[0047] As shown in Figure 1 , an LLM model (base model) contains many Linea layers, and the LoRA model replaces all the Linear layers in the base model with LoRA Linear. LoRA Linear includes the following three main weights:

[0048] base_layer: the weight of the original Linear;

[0049] LoRA_A and LoRA_B: New weights added to LoRA.

[0050] The structure of a LoRA model consists of the set of all LoRA Linear weights. For a LoRA model without weight merging, the inference computation process is as follows:

[0051] Calculate the output of base_layer:

[0052] base_output = base_layer(input)

[0053] Calculate the output for each LoRA request individually:

[0054] LoRA_output_i=input@LoRA_A.weight[i].T@LoRA_B.weight[i].T

[0055] Add base_output to the output of each LoRA request:

[0056] output_i=base_output+LoRA_output_i

[0057] This batch processing method requires a lot of computing resources, especially when there are multiple LoRA requests at the same time, which significantly increases the inference time and memory requirements, limiting the inference efficiency of the LoRA model.

[0058] The applicant in this case discovered that in LLM inference systems, the rank of LoRA is typically not fixed, and can be 2, 4, 8, 16, or 32, among others. To support inference for LoRA models with different ranks, a common approach is to pad all LoRA models to the maximum supported rank. This ensures that the weight shape of all LoRA models remains consistent.

[0059] LoRA_A:(max_LoRAs,rank,in_features)

[0060] LoRA_B:(max_LoRAs,out_features,rank)

[0061] like Figure 2 As shown, in the padding operation, 0 is used as the padding value (the gray area represents the padding), and this padding does not affect the calculation results. However, this method has a significant drawback:

[0062] Memory waste: For example, when the maximum supported rank is 32, but the actual calculated LoRA model is all rank=8, the memory utilization rate is only 25%.

[0063] Additional computational burden: The padding operation increases unnecessary computation, further reducing efficiency.

[0064] To solve the above technical problems, the present disclosure proposes a LoRA weight management method for large language models. By designing a page-based weight management method, the present scheme has the following significant technical advantages:

[0065] Memory utilization rate improvement: Based on the page-based management method, the weights of the LoRA model are stored and managed in pages, avoiding the memory waste caused by the traditional padding method. The size of each page is dynamically allocated according to actual needs, enabling the system to use memory resources more accurately, thereby significantly improving memory utilization.

[0066] Support for dynamic rank adjustment: The page-based weight management method allows the system to flexibly support LoRA models of different ranks without uniform padding to the maximum rank. This flexibility not only reduces memory usage, but also reduces unnecessary computational overhead, providing more efficient performance support for inference tasks.

[0067] Improve concurrency: By dividing the weights into pages and managing them independently, the system can handle more LoRA requests simultaneously. This design significantly improves the maximum concurrency of the system, meeting the high throughput requirements of large-scale models in practical applications.

[0068] Reduce computational resource demand: The dynamic allocation and invocation of weight pages enable the inference system to load only the necessary weight data during LoRA request processing, avoiding redundant computation and effectively reducing the demand for computational resources.

[0069] Through the above optimization, the present scheme comprehensively improves memory utilization, system throughput, and resource consumption, significantly enhancing the efficiency and stability of LLM inference systems in practical applications.

[0070] The content of the present application will be described in detail below with the help of specific embodiments.

[0071] Figure 3 is a flowchart of a LoRA weight management method for large language models according to an exemplary embodiment. The LoRA weight management method for large language models 30 includes at least steps S302 to S310.

[0072] As Figure 3As shown, in S302, the inference system of the large language model obtains LoRA weight data, which includes a low-rank matrix A and a low-rank matrix B. The low-rank matrices A and B are used to effectively fine-tune the pre-trained model based on a low-rank adaptation mechanism.

[0073] In S304, the storage location of the LoRA weight data is determined. The overall architecture of the large language model inference system, storage resources, and specific characteristics of the LoRA weight are considered comprehensively to ensure the efficiency and availability of data storage.

[0074] For example, the storage location of the LoRA weight data can be determined according to the system parameters of the large language model inference system, the storage page parameters, and the characteristics of the LoRA weight data. The weights are stored in pages according to the system storage page parameters and the characteristics of the LoRA weight. For example, based on the system parameters and storage page parameters of the large language model inference system, the LoRA weight matrix is allocated to different storage pages according to its dimensions and characteristics, thereby reducing conflicts during storage and improving the utilization efficiency of memory resources.

[0075] In S306, the low-rank matrix A and the low-rank matrix B are stored in pages according to the storage location.

[0076] More specifically, the storage location of the LoRA weight data is determined according to the system parameters of the large language model inference system, the storage page parameters, and the characteristics of the LoRA weight data, including determining the storage page size and determining the storage location of the LoRA weight data according to the maximum number of LoRA layers of the large language model inference system, the storage page size, and the dimensions of the LoRA weight data.

[0077] In one embodiment, the storage page size can be determined according to the matrix size of the tensor core in the inference system of the large language model, thereby achieving balanced optimization of computing resources and storage resources.

[0078] In one embodiment, the location of the low-rank matrix A can be set as (the maximum number of LoRA layers, the storage page size, and the input feature dimension of the low-rank matrix A):

[0079] LoRA_A:(max_LoRAs2,page_size,in_features).

[0080] In one embodiment, the location of the low-rank matrix B can be set as (the maximum number of LoRA layers, the output feature dimension of the low-rank matrix B, and the storage page size):

[0081] LoRA_B:(max_LoRAs2,out_features,page_size).

[0082] Figure 4 This is a schematic diagram illustrating a LoRA weight management method for a large language model according to another exemplary embodiment.

[0083] max_LoRAs=8, max_rank=4, in_features=6,

[0084] At this point, there are three LoRA models in the system: LoRA_0 has a rank of 4 (the entire first row), LoRA_1 has a rank of 2 (half of the second row, light gray), and LoRA_2 has a rank of 2 (half of the third row, dark gray).

[0085] After padding according to the existing LoRA model, the storage in memory is as follows: Figure 4 As shown, gray represents padding, and white represents unused video memory. Overall, excluding unused memory, the effective utilization rate of this storage method is only 4 / 6.

[0086] According to the method in this application, the memory distribution based on page-based management is as follows: Figure 5 As shown, under the technical settings of this application:

[0087] page_size=2, LoRA_0 rank is 4, occupying two pages.

[0088] LoRA_1 and LoRA_2 both have a rank of 2 and occupy one page.

[0089] According to the technical settings in this application, which do not include unused video memory, the actual effective video memory utilization rate is 100%. This method in this application greatly improves the video memory utilization rate.

[0090] In S308, the mapping relationship between LoRA weight data and its corresponding storage page is recorded. For example, a page table can be set up to record the mapping relationship between the low-rank matrix A and the low-rank matrix B and their corresponding storage pages. This mapping mechanism ensures that the inference system can quickly and accurately access the corresponding weight data when needed, and avoids unnecessary data retrieval conflicts during the inference process.

[0091] At S310, the inference system of the large language model calls the LoRA weight data for inference calculation according to the mapping relationship. The inference system of the large language model calls the LoRA weight data page by page for inference calculation based on the mapping relationship; and the calculation results of all pages are summarized to generate a final inference result.

[0092] According to the mapping relationship, the inference system calls the LoRA weight data page by page, and performs calculation operations such as matrix multiplication. In the calculation process of each page, the system combines the outputs of the low-rank matrices A and B with the base layer output of the pre-trained model to generate a final inference result. After the calculation results of all pages are summarized, a final inference output is generated to meet the requirements of efficient and high-concurrency inference.

[0093] Through this page management and mapping mechanism, the large language model inference system can significantly improve inference efficiency, reduce memory waste, improve system throughput, and support higher-concurrency inference tasks.

[0094] According to the LoRA weight management method for a large language model of the present application, the LoRA weight data is obtained by the inference system of the large language model, the LoRA weight data includes a low-rank matrix A and a low-rank matrix B; the storage location of the LoRA weight data is determined; the low-rank matrix A and the low-rank matrix B are stored in units of pages according to the storage location; the mapping relationship between the LoRA weight data and its corresponding storage page is recorded; and the inference system of the large language model calls the LoRA weight data for inference calculation according to the mapping relationship. By paging storage and mapping management of the LoRA weight, the memory utilization and calculation efficiency can be improved, large-scale model flexible expansion can be supported, multiple hardware environments can be adapted, and inference performance can be optimized.

[0095] It should be clearly understood that the present application describes how to form and use specific examples, but the principles of the present application are not limited to any details of these examples. On the contrary, based on the teachings of the disclosure of the present application, these principles can be applied to many other embodiments.

[0096] Figure 6 is a flowchart of a LoRA weight management method for a large language model according to another exemplary embodiment. Figure 6 The flowchart 60 shown is an elaboration of Figure 3 The detailed description of S310 "the inference system of the large language model calls the LoRA weight data for inference calculation" in the flowchart shown.

[0097] As Figure 6As shown, in S602, the inference system of the large language model retrieves the low-rank matrix A and the low-rank matrix B in the LoRA weight data page by page. The inference system reads the data of the low-rank matrix A and B from the storage according to the mapping relationship of the pages according to the page table information recorded in advance. This process ensures that the weight data of each page can be quickly loaded into the system through an effective memory management mechanism, thereby optimizing the use of storage bandwidth and avoiding data access bottlenecks.

[0098] In S604, inference calculation is performed based on the low-rank matrix A and the low-rank matrix B. Matrix multiplication can be performed on the input data and the corresponding pages of the low-rank matrix A and the low-rank matrix B.

[0099] The system performs matrix multiplication operation on the input data and the corresponding pages of the low-rank matrix A and the low-rank matrix B. Specifically, the inference system first performs multiplication operation on the current page of the low-rank matrix A and the input data, and then performs multiplication operation on the obtained result and the current page of the low-rank matrix B. This process utilizes the low-rank adaptation technology, significantly reduces the consumption of computing resources while maintaining the inference accuracy.

[0100] In S606, the calculation results of all pages are summarized to generate the final inference result.

[0101] Since the calculation of the LoRA model involves data of multiple pages, the calculation result of each page is completed one by one in the inference system. The inference system obtains the calculation output of the current page in each step and summarizes these results. Finally, the calculation results of all pages are combined to generate the final inference result, which is the prediction or output of the input data by the large language model after fine-tuning. Through this process, the large language model can efficiently complete the inference task while maximizing the use of storage and computing resources, improving the inference speed and throughput.

[0102] Figure 7 is a schematic diagram of a LoRA weight management method for a large language model according to another exemplary embodiment. As shown in Figure 7 Assuming that a LoRA linear rank=4, the output calculation formula of the LoRA is:

[0103] LoRA_output = input @ LoRA_A.T @ LoRA_B.T

[0104] When they are stored in two pages respectively, they can be divided into two calculations.

[0105] First page calculation:

[0106] LoRA_output_0 = input@LoRA_A[block_table[0]].T@LoRA_B[block_table[0]].T

[0107] The calculation of the second page:

[0108] LoRA_output_1 = input@LoRA_A[block_table[1]].T@LoRA_B[block_table[1]].T

[0109] The final result is to add them together:

[0110] LoRA_output = LoRA_output_0 + LoRA_output_1

[0111] The essence of its calculation equivalence is that a LoRA model with a larger rank can be divided into several LoRA models with smaller ranks

[0112] For the setting of page_size, in the above example, page_size is set to 2. In practice, page_size can be recommended to be set to at least 8. This is because in the cuda system, matrix multiplication will use tensor core for acceleration, but the use of tensor core usually requires the size of the mxnxk three matrices to be at least 8, otherwise padding is required, which reduces the calculation speed.

[0113] Those skilled in the art can understand that all or part of the steps of the above embodiments are implemented as computer programs executed by the CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided by the present application are executed. The program can be stored in a computer readable storage medium, which can be a read-only memory, a magnetic disk or an optical disk, etc.

[0114] In addition, it should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not for limiting purposes. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0115] The following is a device embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0116] Figure 8This is a block diagram illustrating a LoRA weight management device for a large language model according to an exemplary embodiment. Figure 8 As shown, the LoRA weight management device 80 for large language models includes: a data module 802, a location module 804, a storage module 806, a mapping module 808, and an inference module 810.

[0117] Data module 802 is used by the inference system of a large language model to obtain LoRA weight data, which includes: low-rank matrix A and low-rank matrix B;

[0118] The location module 804 is used to determine the storage location of the LoRA weight data; the location module 804 is also used to determine the storage location of the LoRA weight data based on the system parameters of the large language model inference system, the storage page parameters, and the characteristics of the LoRA weight data.

[0119] Storage module 806 is used to store the low-rank matrix A and the low-rank matrix B in pages according to the storage location;

[0120] The mapping module 808 is used to record the mapping relationship between LoRA weight data and its corresponding storage page; the mapping module 808 is also used to set up a page table to record the mapping relationship between the low-rank matrix A and the low-rank matrix B and their corresponding storage pages.

[0121] The inference module 810 is used by the inference system of the large language model to call the LoRA weight data for inference calculation according to the mapping relationship. The inference module 810 is also used by the inference system of the large language model to call the LoRA weight data page by page for inference calculation based on the mapping relationship; and to summarize the calculation results of all pages to generate the final inference result.

[0122] According to the LoRA weight management device for large language models disclosed in this application, LoRA weight data is obtained through the inference system of the large language model. The LoRA weight data includes a low-rank matrix A and a low-rank matrix B. The storage location of the LoRA weight data is determined. The low-rank matrix A and the low-rank matrix B are stored in pages according to the storage location. The mapping relationship between the LoRA weight data and its corresponding storage page is recorded. The inference system of the large language model calls the LoRA weight data for inference calculation according to the mapping relationship. By managing LoRA weights through paging storage and mapping, the utilization of video memory and computational efficiency can be improved, large-scale model flexible expansion can be supported, multiple hardware environments can be adapted, and inference performance can be optimized.

[0123] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0124] The electronic device 900 according to this embodiment of the present application will be described below with reference to Figure 9 FIG. 9. Figure 9 The electronic device 900 shown is merely an example and should not limit the scope of functionality or use of the embodiments of the present application.

[0125] As shown in Figure 9 FIG. 9, the electronic device 900 is in the form of a general computing device. Components of the electronic device 900 can include, but are not limited to, at least one processing unit 910, at least one memory unit 920, a bus 930 that connects different system components including the memory unit 920 and the processing unit 910, a display unit 940, and the like.

[0126] The memory unit stores program codes that can be executed by the processing unit 910, so that the processing unit 910 performs the steps described in the present specification according to various exemplary embodiments of the present application. For example, the processing unit 910 can perform the steps as shown in Figure 3 Figure 6

[0127] The memory unit 920 can include a readable medium in the form of a volatile memory unit, such as a random access memory (RAM) 9201 and / or a cache memory unit 9202, and can further include a read-only memory (ROM) 9203.

[0128] The memory unit 920 can further include a program / utility 9204 having a set of program modules 9205, including but not limited to, an operating system, one or more application programs, other program modules, and program data, and each of these examples, or some combination thereof, can include implementation of a network environment.

[0129] The bus 930 can represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processing bus, or a local bus using any of a variety of bus architectures by using any of a variety of bus architectures.

[0130] ​​The electronic device 900 can also communicate with one or more external devices 900' such as a keyboard, a pointing device, a Bluetooth device, etc.; devices that enable a user to interact with the electronic device 900; and / or any devices (e.g., a router, a modem, a peer device or other computing device) that enable the electronic device 900 to communicate with one or more other computing devices. Such communication can occur via an input / output (I / O) interface 950. Still yet, the electronic device 900 can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or the Internet) via a network adapter 960. The network adapter 960 can communicate with the other components of the electronic device 900 via the bus 930. It should be understood that, although not shown, other hardware and / or software components could be used in conjunction with the electronic device 900. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0131] From the description given above, those skilled in the art will readily perceive that the example embodiments described herein can be implemented by software and / or by hardware coupled with software. Accordingly, the terms "processor" and "controller" should be interpreted broadly to encompass a general purpose processor, a central processing unit, a microprocessor, a microcomputer, a microcomputer core of a processor (e.g., a processor core of a multi-core module), a microcontroller, a digital signal processor (DSP), a graphics processing unit (GPU), a reduced instruction set computer (RISC) processor, a very large scale integration (VLSI) processor, a superconducting Figure 10 As shown, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or on a network, and includes a plurality of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to perform the above-mentioned method according to the embodiments of the present application.

[0132] The software product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electrical, a magnetic, an optical, an electromagnetic, an infrared, or a semiconductor system, device or apparatus, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or a flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0133] The computer readable storage medium can include a data signal transported over a carrier wave and can be baseband or propagated along with carriers. The program code embodied on the computer readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, and the like, or any suitable combination of the foregoing.

[0134] The program code can be executed by one or more programmable processors, which can be hardware, software, firmware, or any combination thereof. The program code can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0135] The computer readable medium described above can carry one or more programs, when the one or more programs are executed by the device, the computer readable medium enables the following functions: the large language model inference system obtains LoRA weight data, the LoRA weight data includes: a low rank matrix A and a low rank matrix B; determines the storage location of the LoRA weight data; stores the low rank matrix A and the low rank matrix B in page units respectively according to the storage location; records the mapping relationship between the LoRA weight data and the corresponding storage page; the large language model inference system calls the LoRA weight data according to the mapping relationship to perform inference calculation.

[0136] Those skilled in the art can understand that the above modules can be distributed in the device according to the description of the embodiments, and can also be changed in one or more devices different from the embodiments. The modules of the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0137] Those skilled in the art can easily understand, through the above description of the embodiments, that the example embodiments described herein can be implemented by software or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present application.

[0138] The example embodiments of the present application are specifically shown and described above. It should be understood that the present application is not limited to the detailed structure, arrangement or implementation method described herein; on the contrary, the present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A LoRA weight management method for large language models, characterized in that, include: The reasoning system of the large language model acquires LoRA weight data, which includes: low-rank matrix A and low-rank matrix B; The storage page size is determined based on the matrix size of the tensor core in the large language model inference system. The storage location of the LoRA weight data is determined based on the maximum number of LoRA layers in the large language model inference system, the storage page size, and the dimension of the LoRA weight data. This includes setting the position of the low-rank matrix A as (maximum number of LoRA layers, storage page size, and input feature dimension of low-rank matrix A), and setting the position of the low-rank matrix B as (maximum number of LoRA layers, output feature dimension of low-rank matrix B, and storage page size). The low-rank matrix A and the low-rank matrix B are stored in pages according to the storage location. Record the mapping relationship between LoRA weight data and its corresponding storage page; The reasoning system of the large language model calls the LoRA weight data to perform reasoning calculations based on the mapping relationship.

2. The method as described in claim 1, characterized in that, Record the mapping relationship between LoRA weight data and its corresponding storage pages, including: A page table is set up to record the mapping relationship between the low-rank matrix A and the low-rank matrix B and their corresponding storage pages.

3. The method as described in claim 1, characterized in that, The inference system of the large language model calls the LoRA weight data according to the mapping relationship to perform inference calculations, including: The reasoning system of the large language model calls the LoRA weight data page by page based on the mapping relationship to perform reasoning calculations; The calculation results from all pages are aggregated to generate the final inference result.

4. The method as described in claim 3, characterized in that, The inference system of the large language model calls the LoRA weight data page by page based on the mapping relationship to perform inference calculations, including: The reasoning system of the large language model retrieves the low-rank matrix A and the low-rank matrix B from the LoRA weight data page by page; Inference calculations are performed based on the low-rank matrix A and the low-rank matrix B.

5. The method as described in claim 4, characterized in that, Inference calculations are performed based on the low-rank matrix A and the low-rank matrix B, including: Matrix multiplication of the input data with the corresponding pages of low-rank matrix A and low-rank matrix B.

6. A LoRA weight management device for large language models, characterized in that, include: The data module is used by the inference system of the large language model to obtain LoRA weight data, which includes: low-rank matrix A and low-rank matrix B; The location module is used to determine the storage page size based on the matrix size of the tensor core in the inference system of the large language model, and to determine the storage location of the LoRA weight data based on the maximum number of LoRA layers in the inference system of the large language model, the storage page size, and the dimension of the LoRA weight data. This includes setting the position of the low-rank matrix A as (maximum number of LoRA layers, storage page size, and input feature dimension of low-rank matrix A), and setting the position of the low-rank matrix B as (maximum number of LoRA layers, output feature dimension of low-rank matrix B, and storage page size). A storage module is used to store the low-rank matrix A and the low-rank matrix B in pages according to the storage location. The mapping module is used to record the mapping relationship between LoRA weight data and its corresponding storage pages; The inference module is used by the inference system of the large language model to call the LoRA weight data for inference calculation based on the mapping relationship.

Images