Lora weight fusion method and device of large language model inference system for generating text

Abstract

The application relates to a LoRA weight fusion method and device for a large language model inference system. The method comprises the following steps: a large language model inference system acquires a plurality of LoRA weight data; a fusion ratio corresponding to the plurality of LoRA weight data is determined; the plurality of LoRA weight data are subjected to splicing processing or cutting processing based on the fusion ratio, so as to generate LoRA fusion weight; the inference system of the large language model acquires input data; and inference calculation is performed on the LoRA fusion weight based on the input data. The LoRA weight fusion method and device for the large language model inference system are suitable for a scene requiring dynamic multi-style fusion, can improve the utilization rate of display memory, and meet the dual requirements of flexibility and performance of the large language model inference system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer information processing, and in particular, to a LoRA weight fusion method and device for a large language model inference system for generating text. BACKGROUND

[0002] In recent years, deep learning technology has made significant progress in natural language processing (NLP), computer vision (CV), and reinforcement learning (RL) fields. Among them, large language models (LLMs) as a kind of deep learning model are mainly used for processing text data. These models learn rich language knowledge and semantic information by pre-training on large-scale corpus, and are widely used in text generation, machine translation, question answering systems and other downstream tasks.

[0003] Although large language models have strong language understanding ability in the pre-training stage, actual needs for specific tasks still need to be further improved through fine-tuning. Fine-tuning refers to optimizing training on a small amount of labeled data for specific tasks based on pre-trained models, so that the model better adapts to the data characteristics and needs of specific tasks, and thus achieves better actual performance.

[0004] In order to reduce the computational cost of fine-tuning and improve efficiency, an efficient model fine-tuning method, LoRA (Low-Rank Adaptation), has been proposed in recent years. The core idea of LoRA is to add a low-rank matrix to the pre-trained model to realize fine-tuning, and only a small number of parameters need to be adjusted to significantly improve the model performance.

[0005] However, in practical applications, the style fusion of multiple LoRA models has become a new challenge. For example, by training LoRA models on different style datasets, the model can generate text of a specific style, but existing methods are difficult to dynamically adjust the style proportion or generate text of multiple styles at the same time. The current LLM inference system does not support the function of dynamically fusing multiple LoRA models, which limits its application potential in multi-style text generation.

[0006] Therefore, a new LoRA weight fusion method and device for a large language model inference system are needed.

[0007] The above information disclosed in the BACKGROUND section is only intended to strengthen the understanding of the background of the present application, and therefore it can include information that does not constitute the prior art known to those of ordinary skill in the art. SUMMARY

[0008] Therefore, the application provides a LoRA weight fusion method and device for a large language model inference system, which is suitable for scenarios requiring dynamic multi-style fusion, can improve the utilization of video memory, and meets the dual requirements of flexibility and performance of the large language model inference system.

[0009] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0010] According to an aspect of the present application, a LoRA weight fusion method for a large language model inference system is provided, which comprises: a large language model inference system obtaining a plurality of LoRA weight data; determining a fusion ratio corresponding to the plurality of LoRA weight data; based on the fusion ratio, performing splicing processing or segmentation processing on the plurality of LoRA weight data to generate LoRA fusion weight; the inference system of the large language model obtains input data; and based on the input data, calling the LoRA fusion weight for inference calculation.

[0011] In an exemplary embodiment of the present application, determining the fusion ratio corresponding to the plurality of LoRA weight data comprises: determining the fusion ratio corresponding to the plurality of LoRA weight data according to a preset task requirement.

[0012] In an exemplary embodiment of the present application, based on the fusion ratio, the plurality of LoRA weight data is spliced or segmented to generate LoRA fusion weight, which comprises: based on the rank dimension of LoRA weight data, the plurality of LoRA weight data is spliced or segmented; and according to the fusion ratio, the data after splicing or segmentation is fused to generate the LoRA fusion weight.

[0013] In an exemplary embodiment of the present application, based on the rank dimension of LoRA weight data, the plurality of LoRA weight data is spliced or segmented to generate the LoRA processing weight, which comprises: when the rank dimension of LoRA weight data is less than a dimension threshold, the plurality of LoRA weight data is spliced to be summarized as the LoRA processing weight.

[0014] In an exemplary embodiment of the present application, according to the fusion ratio, the data after splicing or segmentation is fused to generate the LoRA fusion weight, which comprises: according to the fusion ratio, the LoRA processing weight is matrix spliced according to the rank dimension to generate the LoRA fusion weight.

[0015] In an example embodiment of the present application, the rank dimension of the LoRA weight data is used to splice or split the plurality of LoRA weight data to generate the LoRA processing weight, including: when the rank dimension of the LoRA weight data is greater than the dimension threshold, the LoRA weight data is split into a plurality of LoRA processing weights.

[0016] In an example embodiment of the present application, the data after splicing or splitting is fused according to the fusion ratio to generate the LoRA fusion weight, and further includes: after splitting, the plurality of LoRA processing weights and the corresponding fusion ratio are stored based on the page management method to generate the LoRA fusion weight.

[0017] In an example embodiment of the present application, the LoRA fusion weight is called for inference calculation based on the input data, including: the input data and the fusion weight are used for inference calculation to generate a LoRA result; and the LoRA result and the output result of the large language model are added to generate an inference result.

[0018] In an example embodiment of the present application, the input data and the fusion weight are used for inference calculation to generate a LoRA result, including: after splitting, the input data is used for inference calculation with the plurality of LoRA processing weights and the corresponding fusion ratio to generate a LoRA result.

[0019] According to an aspect of the present application, a LoRA weight fusion device for a large language model inference system is provided, which includes: a data module for obtaining a plurality of LoRA weight data by the large language model inference system; a ratio module for determining a fusion ratio corresponding to the plurality of LoRA weight data; a processing module for splicing or splitting the plurality of LoRA weight data based on the fusion ratio to generate a LoRA fusion weight; an input module for obtaining input data by the inference system of the large language model; and an inference module for calling the LoRA fusion weight for inference calculation based on the input data.

[0020] According to an aspect of the present application, an electronic device is provided, which includes: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0021] According to an aspect of the present application, a computer readable medium having a computer program stored thereon is provided, and the program is executed by a processor to implement the method as described above.

[0022] According to the LoRA weight fusion method and device for the large language model inference system provided in the application, a plurality of LoRA weight data is obtained through the large language model inference system; a fusion ratio corresponding to the plurality of LoRA weight data is determined; the plurality of LoRA weight data is spliced or cut based on the fusion ratio to generate LoRA fusion weight; the inference system of the large language model obtains input data; and the LoRA fusion weight is called for inference calculation based on the input data. The method is suitable for a scene requiring dynamic multi-style fusion, can improve the utilization rate of video memory, and meets the dual requirements of flexibility and performance of the large language model inference system.

[0023] It should be understood that the foregoing general description and the following detailed description are only examples and are not limiting the application. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0025] Figure 1 FIG. 1 is a flowchart illustrating a LoRA weight fusion method for a large language model inference system according to an example embodiment.

[0026] Figure 2 FIG. 2 is a flowchart illustrating a LoRA weight fusion method for a large language model inference system according to another example embodiment.

[0027] Figure 3 FIG. 3 is a schematic diagram illustrating a LoRA weight fusion method for a large language model inference system according to another example embodiment.

[0028] Figure 4 FIG. 4 is a schematic diagram illustrating a LoRA weight fusion method for a large language model inference system according to another example embodiment.

[0029] Figure 5 FIG. 5 is a flowchart illustrating a LoRA weight fusion method for a large language model inference system according to another example embodiment.

[0030] Figure 6 FIG. 6 is a schematic diagram illustrating a LoRA weight fusion method for a large language model inference system according to another example embodiment.

[0031] Figure 7FIG. 1 is a schematic diagram of a LoRA weight fusion method for a large language model inference system according to another exemplary embodiment.

[0032] Figure 8 FIG. 1 is a schematic diagram of a LoRA weight fusion method for a large language model inference system according to another exemplary embodiment.

[0033] Figure 9 FIG. 1 is a schematic diagram of a LoRA weight fusion method for a large language model inference system according to another exemplary embodiment.

[0034] Figure 10 FIG. 1 is a schematic diagram of a LoRA weight fusion method for a large language model inference system according to another exemplary embodiment. DETAILED DESCRIPTION

[0035] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the figures, and descriptions of the same or similar elements can be incorporated throughout the description by reference to the same or similar reference numerals.

[0036] In addition, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the application can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, implementations, and operations have not been shown or described in detail to avoid obscuring aspects of the application.

[0037] The block diagrams in the drawings show functions and functionality as they can be implemented in software / firmware. However, doing this is only an example and the functions shown can equally be implemented in hardware and / or in a processor and / or controller device. Further, the steps or functions illustrated in the flow charts can be carried out in any order unless it is expressly stated otherwise in the description.

[0038] The flow charts shown in the drawings are only exemplary and do not necessarily have to include all content and operations / steps and do not necessarily have to be carried out in the order described. For example, some operations / steps can be broken down further, and some operations / steps can be combined or partially combined, so that the order of actual execution can be changed depending on the actual situation.

[0039] It should be understood that although the terms first, second, third, etc. can be used herein to describe various components, these components should not be limited by these terms. These terms are used only to distinguish one component from another. Thus, a first component discussed below could be termed a second component without departing from the teachings of the present application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0040] Those skilled in the art can understand that the modules or flows in the drawings are not necessarily required for implementing the present application, and therefore cannot be used to limit the protection scope of the present application.

[0041] Figure 1 is a flowchart of a LoRA weight fusion method for a large language model inference system according to an exemplary embodiment. The LoRA weight fusion method for a large language model inference system 10 at least includes steps S102-S110.

[0042] As shown in S102, the large language model inference system obtains a plurality of LoRA weight data. Figure 1

[0043] In actual application process, the large language model inference system obtains a plurality of LoRA weight data through loading or initialization operation. These weight data can come from different styles of LoRA model, for example, LoRA model weights trained for specific data sets such as humor style, satire style, etc. Each LoRA weight data contains a low-rank matrix representation of the original model parameters, with independent style characteristics.

[0044] In S104, the fusion ratio corresponding to the plurality of LoRA weight data is determined. For example, the fusion ratio corresponding to the plurality of LoRA weight data can be determined according to the preset task requirement.

[0045] In one specific embodiment, for example, the inference system determines the fusion ratio of the plurality of LoRA weight data according to the preset task requirement. The fusion ratio can be dynamically calculated based on user input, task description, target style requirement, etc. For example, when generating text, 80% of humor style and 20% of satire style are required, and the corresponding weight ratio is 0.8 and 0.2.

[0046] For example, the ratio can also be stored as a corresponding fusion coefficient for direct calling in subsequent calculation.

[0047] In S106, the plurality of LoRA weight data is spliced or cut based on the fusion ratio to generate LoRA fusion weight.

[0048] ​It is worth mentioning that LoRA Linear adds LoRA-related calculations on the basis of the original Base Linear. The calculation process of BaseLinear is as follows:

[0049] base_output=input @ weight.T

[0050] In the LoRA Linear structure, two new tensors, LoRA_A and LoRA_B, are added based on the original weight tensor. The calculation method is to calculate the multiplication of the two small matrices, then add the output result of LoRA to the output result of BaseLinear to get the final output result:

[0051] LoRA_output=input@LoRA_A.T@LoRA_B.T#(@ represents matrix multiplication,.T represents matrix transpose)

[0052] output=base_output+LoRA_output

[0053] Among them, the shape of each tensor is as follows:

[0054] plaintext

[0055] input: (batch_size, in_features)

[0056] weight: (out_features, in_features)

[0057] base_output: (batch_size, out_features)

[0058] LoRA_A: (rank, in_features)

[0059] LoRA_B: (out_features, rank)

[0060] LoRA_output: (batch_size, out_features)

[0061] output: (batch_size, out_features)

[0062] Because the shape of base_output and LoRA_output is the same, they can be directly added.

[0063] Accordingly, it can be known that the LoRA weight has a splittable characteristic in the rank dimension. Specifically, a LoRA model with a larger rank can be split into several smaller LoRA models in the rank dimension. By respectively calculating the output results of these split LoRA models and adding them together, the final result is equivalent to the calculation result without splitting.

[0064] For example:

[0065] LoRA_output_0=input@LoRA_A_0.T@LoRA_B_0.T

[0066] LoRA_output_1=input@LoRA_A_1.T@LoRA_B_1.T

[0067] LoRA_output=LoRA_output_0+LoRA_output_1

[0068] Assuming that a LoRA model is split into two smaller models in equal proportions, the shapes of the tensors are as follows:

[0069] plaintext

[0070] LoRA_A_0: (rank / 2, in_features)

[0071] LoRA_B_0: (out_features, rank / 2)

[0072] LoRA_A_1: (rank / 2, in_features)

[0073] LoRA_B_1: (out_features, rank / 2)

[0074] LoRA_output_0: (batch_size, out_features)

[0075] LoRA_output_1: (batch_size, out_features)

[0076] Since the rank dimension only appears in the intermediate results of LoRA calculation, splitting the rank dimension will not affect the final calculation result.

[0077] The splicing and splitting of LoR weights in the rank dimension are closely related. Specifically, if multiple small LoRA models need to be calculated simultaneously, they can be spliced into a larger LoRA model in the rank dimension, and then the result is derived through a unified calculation process.

[0078] plaintext

[0079] LoRA_A_combined = concat(LoRA_A_0, LoRA_A_1, dim = 0) # Splice in rank dimension

[0080] LoRA_B_combined = concat(LoRA_B_0, LoRA_B_1, dim = 1) # Splice in rank dimension

[0081] LoRA_output_combined = input @ LoRA_A_combined.T @ LoRA_B_combined.T

[0082] Through splicing, the calculation of multiple LoRA models can be completed at once, reducing the complexity of calculating each model individually, while ensuring that the results are consistent with those obtained by calculating each model individually and then adding them together.

[0083] Through the above characteristics, the flexibility of the LoRA model is greatly improved, which can be divided into sub-models in the rank dimension when needed, and multiple small models can be spliced into a unified large model in the task scenario to achieve more efficient inference calculation.

[0084] In the present application, the plurality of LoRA weight data can be spliced or split based on the rank dimension of the LoRA weight data, and the spliced or split data can be fused according to the fusion ratio to generate the LoRA fusion weight.

[0085] The plurality of LoRA weight data is spliced or split to generate a unified LoRA fusion weight.

[0086] In one embodiment, the splicing of the weights of different LoRA models in the rank dimension can be used to form a unified weight matrix. In the splicing process, appropriate padding can be performed to ensure the consistency of the dimensions after splicing.

[0087] In another embodiment, if the rank of a certain LoRA model is large, it can be divided into multiple smaller rank sub-weight matrices in the rank dimension. The divided sub-weights can participate in subsequent fusion calculation respectively, improving the memory utilization efficiency.

[0088] More specifically, the spliced or divided LoRA weight matrix can be weighted and fused according to a preset fusion ratio. The weighting process can be completed through matrix multiplication operation, ensuring that LoRA weights of different styles participate in the calculation according to the ratio.

[0089] In S108, the inference system of the large language model obtains input data. The inference system of the large language model receives input data from the user end, such as the starting text or context prompt in the text generation task. The input data will be the basis for inference calculation and directly affect the final output result.

[0090] In S110, the LoRA fusion weight is called for inference calculation based on the input data. For example, the input data and the fusion weight are subjected to inference calculation to generate a LoRA result; and the LoRA result and the output result of the large language model are added to generate an inference result.

[0091] According to the LoRA weight fusion method for the large language model inference system of the present application, multiple LoRA weight data are obtained by the large language model inference system; a fusion ratio corresponding to the multiple LoRA weight data is determined; the multiple LoRA weight data are subjected to splicing processing or division processing based on the fusion ratio to generate a LoRA fusion weight; input data are obtained by the inference system of the large language model; and the LoRA fusion weight is called for inference calculation based on the input data. This method is suitable for scenarios requiring dynamic multi-style fusion, can improve the memory utilization rate, and meets the dual requirements of flexibility and performance of the large language model inference system.

[0092] It should be clearly understood that the present application describes how to form and use specific examples, but the principles of the present application are not limited to any details of these examples. On the contrary, based on the teachings of the disclosure of the present application, these principles can be applied to many other embodiments.

[0093] Figure 2 is a flowchart of a LoRA weight fusion method for a large language model inference system according to another exemplary embodiment. Figure 2 The flowchart 20 shown is an embodiment of the LoRA weight fusion method for the large language model inference system. Figure 1 The detailed description of the dimension splicing method in the flowchart shown.

[0094] As shown in the flowchart shown in Figure 2As shown in S202, when the rank dimension of the LoRA weight data is less than a dimension threshold, the multiple LoRA weight data are concatenated. First, the rank dimension of the acquired LoRA weight data is determined; if it is determined that the rank dimension of the weight data is less than a preset dimension threshold, then it is concatenated.

[0095] In S204, the LoRA processing weights are concatenated into a matrix according to the rank dimension according to the fusion ratio to generate the LoRA fusion weights.

[0096] In S206, the input data and the fusion weights are used for inference calculation to generate a LoRA result. The input data and the generated LoRA fusion weights are then used for inference calculation to generate the LoRA calculation result. In this process, the concatenated fusion weights can effectively combine the characteristics of different LoRA models, achieving diverse inference outputs.

[0097] In S208, the LoRA result and the output of the large language model are added together to generate the inference result. The LoRA calculation result is added to the original output of the large language model to generate the final inference result. Through this weighted fusion method, the inference system can not only retain the general knowledge of the large language model, but also improve the performance of specific tasks through the characteristics of the LoRA model.

[0098] In a specific embodiment, taking into account the concatenation of LoRA weights in the rank dimension as described above, multiple LoRA weights can be concatenated in the rank dimension and then calculated.

[0099] Figure 3 This is the storage format for fusing a LoRA model with rank=2 and a LoRA model with rank=1. Yellow represents the weights of LoRA_model_0 (rank=2), pink represents the weights of LoRA_model_1 (rank=1), white represents unused GPU memory, and gray represents padding. Furthermore, a tensor of shape (max_LoRAs, rank) can be defined to represent the weight of each LoRA model in the fusion process. Because it's concatenated along the rank dimension, it has rank elements.

[0100] According to the implementation method of this application, the specific calculation process is as follows: Figure 4 As shown, compared with the calculation method in the prior art, the two separate LoRA models in this application can be concatenated and only need to be calculated once. At the same time, after calculating fused_LoRA_A_out, it is necessary to multiply by the routing weight fused_weight.

[0101] Figure 5 is a flowchart of a LoRA weight fusion method for a large language model inference system according to another exemplary embodiment. Figure 5 The flowchart 50 shown is a process of Figure 1 The detailed description of the dimension splitting method in the flowchart is shown.

[0102] As shown in Figure 5 In S502, when the rank dimension of the LoRA weight data is greater than the dimension threshold, the LoRA weight data is split into multiple LoRA processing weights.

[0103] First, the rank dimension of the LoRA weight data is judged. If the rank dimension is greater than the set dimension threshold, the LoRA weight data is split into multiple smaller LoRA processing weight matrices.

[0104] Through the splitting process, the complexity of a single weight matrix is reduced, the storage and computing resources required for inference calculation are reduced, and the foundation for subsequent flexible management is laid.

[0105] In S504, after the splitting process, multiple LoRA processing weights and their corresponding fusion ratios are stored based on the page management method to generate the LoRA fusion weight. After the splitting is completed, the page management method is used to effectively store and manage the multiple LoRA processing weights and their corresponding fusion ratios.

[0106] The advantages of page management can realize dynamic loading of weights and on-demand allocation of computing resources. In a multi-task scenario, it supports quick switching of different weight combinations.

[0107] In S506, the input data is respectively inferred with the multiple LoRA processing weights and their corresponding fusion ratios to generate the LoRA result. The system respectively infers the input data with the multiple LoRA processing weights and their corresponding fusion ratios to gradually generate respective LoRA calculation results.

[0108] In one specific embodiment, the LoRA model that needs to be fused can be split into several LoRA models of equal rank size in the rank dimension and combined with page management for calculation.

[0109] Based on the page management method, a LoRA model with a large rank can be divided into several smaller rank LoRA models and stored in different pages.

[0110] In the embodiments of the present application, according to the page storageFigure 6 As shown, the storage in the three pages is respectively, since the storage in each page is necessarily the same LoRA model, so when storing LoRA_weight, the shape is defined as (num_pages, 1).

[0111] The specific calculation process is as shown in Figure 7 As shown, the result corresponding to the LoRA model in each page is calculated respectively, and then a summation is performed at the end. It is worth mentioning that although the following calculation process is calculated separately, it can be calculated in parallel when using a GPU.

[0112] In the present application, although the two methods seem to be opposite, they can actually accelerate the calculation. This is because when multiple LoRA models are fused, the rank of the LoRA model may be different, and whether the LLM system needs to fuse multiple LoRAs during inference is dynamically determined (some inference requests need to fuse multiple LoRA models, while some do not). In the technical solution provided by the present application, the method of splitting or splicing can be flexibly selected for dynamic fusion according to the actual scene.

[0113] In the present application, the core of the two methods is the same in that the calculation process is unified:

[0114] Based on the splicing method, when multiple LoRA models need to be fused, they are spliced into a large LoRA model in the rank dimension. By uniformly processing all LoRA models (for example, padding to the same rank), the calculation process is unified.

[0115] Based on the splitting method, whether or not fusion is needed, each LoRA model will be split into equal-sized sub-LoRA models according to the rank dimension, thereby achieving unified calculation.

[0116] Although the two methods differ in operation, their common goal is to optimize the fusion calculation of multiple LoRA models through a unified calculation process.

[0117] Those skilled in the art can understand that all or part of the steps of the above embodiments are implemented as computer programs executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided by the present application are executed. The program can be stored in a computer readable storage medium, which can be a read-only memory, a magnetic disk or an optical disk, etc.

[0118] In addition, it should be noted that the above-described figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not for limiting purposes. It is easy to understand that the processes shown in the above-described figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0119] The following is a device embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0120] Figure 8 is a block diagram of a LoRA weight fusion device for a large language model inference system according to an exemplary embodiment. As shown in Figure 8 , the LoRA weight fusion device 80 for the large language model inference system includes a data module 802, a proportion module 804, a processing module 806, an input module 808, and an inference module 810.

[0121] The data module 802 is used to obtain a plurality of LoRA weight data for the large language model inference system;

[0122] The proportion module 804 is used to determine the fusion proportion corresponding to the plurality of LoRA weight data. The proportion module 804 is also used to determine the fusion proportion corresponding to the plurality of LoRA weight data according to the preset task requirement.

[0123] The processing module 806 is used to perform splicing processing or segmentation processing on the plurality of LoRA weight data based on the fusion proportion to generate LoRA fusion weight. The processing module 806 is also used to perform splicing processing or segmentation processing on the plurality of LoRA weight data based on the rank dimension of the LoRA weight data. The data after splicing processing or segmentation processing is fused according to the fusion proportion to generate the LoRA fusion weight.

[0124] The input module 808 is used to obtain input data for the inference system of the large language model;

[0125] The inference module 810 is used to call the LoRA fusion weight for inference calculation based on the input data. The inference module 810 is also used to perform inference calculation on the input data and the fusion weight to generate LoRA result. The inference result is added to the output result of the large language model to generate an inference result.

[0126] According to the LoRA weight fusion device for the large language model inference system provided in the application, a plurality of LoRA weight data are acquired through the large language model inference system; a fusion ratio corresponding to the plurality of LoRA weight data is determined; the plurality of LoRA weight data are spliced or divided based on the fusion ratio to generate LoRA fusion weight; the inference system of the large language model acquires input data; and the LoRA fusion weight is called for inference calculation based on the input data. The method is suitable for a scene requiring dynamic multi-style fusion, can improve the utilization rate of the display memory, and meets the dual requirements of flexibility and performance of the large language model inference system.

[0127] Figure 9 is a block diagram of an electronic device according to an exemplary embodiment.

[0128] The electronic device 900 according to this embodiment of the application will be described below with reference to Figure 9 Figure 9 The displayed electronic device 900 is merely an example and should not impose any limitation on the functions and use range of the embodiments of the application.

[0129] As shown in Figure 9 , the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 can include, but are not limited to, at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), a display unit 940, and the like.

[0130] The storage unit stores program codes which can be executed by the processing unit 910, so that the processing unit 910 performs the steps described in the specification according to various exemplary embodiments of the application. For example, the processing unit 910 can perform the steps as shown in Figure 1 , Figure 2 , Figure 5 .

[0131] The storage unit 920 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 9201 and / or a cache memory unit 9202, and can further include a read-only memory (ROM) 9203.

[0132] The storage unit 920 can further include program / utilities 9204 having a set of (at least one) program modules 9205, such as an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which can include the implementation of a network environment.

[0133] ​Bus 930 can be one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, a graphics bus, a processor or local bus using any of a variety of bus architectures, and the like.

[0134] Electronic device 900 can also communicate with one or more external devices 900' such as a keyboard or pointing device, a Bluetooth device, etc. using one or more communication interfaces 950. Communication interfaces 950 can include, without limitation, a modem, a network interface card (e.g. an Ethernet card), a wireless network interface card, etc. One or more communication interfaces 950 can enable electronic device 900 to communicate with one or more devices using electrical, electromagnetic, or optical packets in a wired or wireless communication scheme. Communication interfaces 950 can also include one or more devices that can interpret instructions from a user in a manner that can be understood by electronic device 900, such as a speech synthesizer or screen reader.

[0135] From the foregoing description, it will be apparent to a person skilled in the art that the example embodiments described herein can be implemented in software and / or hardware. As such, the technology according to embodiments of the present application can be embodied as a software product or a software program product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB, a mobile hard disk, etc.) or a network, and includes a plurality of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to perform the above-described methods according to embodiments of the present application. Figure 10

[0136] The software product can employ any combination of one or more computer readable media. The computer readable media can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0137] ​The computer readable storage medium can include a data signal transported over a carrier wave and can be baseband or propagated along with carriers. The program code embodied on the computer readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, and the like, or any suitable combination of the foregoing.

[0138] The program code can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device such as through the Internet using an Internet Service Provider.

[0139] The computer readable medium described above can carry one or more programs, when the one or more programs are executed by the device, the computer readable medium can realize the following functions: the large language model inference system obtains a plurality of LoRA weight data; determine the fusion ratio corresponding to the plurality of LoRA weight data; based on the fusion ratio, the plurality of LoRA weight data are spliced or cut, and the LoRA fusion weight is generated; the inference system of the large language model obtains the input data; based on the input data, the LoRA fusion weight is called for inference calculation.

[0140] Those skilled in the art can understand that the above-mentioned modules can be distributed in the device according to the description of the embodiment, and can also be changed in one or more devices different from the embodiment. The modules of the above-mentioned embodiment can be combined into one module, or can be further split into a plurality of sub-modules.

[0141] Those skilled in the art can easily understand, through the above description of the embodiments, that the example embodiments described herein can be implemented by software or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present application.

[0142] The example embodiments of the present application are specifically shown and described above. It should be understood that the present application is not limited to the detailed structure, arrangement or implementation method described herein; on the contrary, the present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A LoRA weight fusion method for a large language model inference system for text generation, characterized in that, include: The large language model inference system acquires multiple LoRA weight data; Determine the fusion ratio corresponding to multiple LoRA weighted data; Based on the fusion ratio, the multiple LoRA weight data are concatenated or segmented to generate LoRA fusion weights. This includes: when the rank dimension of the LoRA weight data is less than a dimension threshold, the multiple LoRA weight data are concatenated to form LoRA processing weights; then, according to the fusion ratio, the LoRA processing weights are matrix-concatenated according to the rank dimension to generate the LoRA fusion weights; and when the rank dimension of the LoRA weight data is greater than a dimension threshold, the LoRA weight data is segmented into multiple LoRA processing weights; after the segmentation, multiple LoRA processing weights and their corresponding fusion ratios are stored using a page-based management method to generate the LoRA fusion weights. The reasoning system of a large language model acquires input data; Inference calculations are performed by calling the LoRA fusion weights based on the input data.

2. The method as described in claim 1, characterized in that, Determine the fusion ratio corresponding to multiple LoRA weighted data, including: The fusion ratio corresponding to the multiple LoRA weight data is determined according to the preset task requirements.

3. The method as described in claim 1, characterized in that, Inference calculations are performed based on the input data and the LoRA fusion weights, including: The input data and the fusion weights are used to perform inference calculations to generate LoRA results; The LoRA results and the output of the large language model are added together to generate the inference result.

4. The method as described in claim 3, characterized in that, The input data and the fusion weights are used to perform inference calculations to generate LoRA results, including: After the segmentation process, the input data is inferred and calculated with multiple LoRA processing weights and their corresponding fusion ratios to generate LoRA results.

5. A LoRA weight fusion device for a large language model inference system for text generation, characterized in that, include: The data module is used by the large language model inference system to obtain multiple LoRA weight data; The scaling module is used to determine the fusion ratio corresponding to multiple LoRA weight data; The processing module is used to concatenate or segment the multiple LoRA weight data based on the fusion ratio to generate LoRA fusion weights. This includes: concatenating the multiple LoRA weight data when the rank dimension of the LoRA weight data is less than a dimension threshold, summarizing them into LoRA processing weights, and then concatenating the LoRA processing weights according to the rank dimension according to the fusion ratio to generate the LoRA fusion weights; and segmenting the LoRA weight data when the rank dimension of the LoRA weight data is greater than a dimension threshold, splitting it into multiple LoRA processing weights, and then storing the multiple LoRA processing weights and their corresponding fusion ratios based on a page-based management method after the segmentation process to generate the LoRA fusion weights. The input module is used by the inference system of a large language model to obtain input data; The inference module is used to perform inference calculations by calling the LoRA fusion weights based on the input data.

Images