Low-rank adapter hot-update method and apparatus, storage medium, and electronic device

Abstract

The present application relates to the technical field of artificial intelligence, and discloses a low-rank adapter hot-update method and apparatus, a storage medium, and an electronic device. The method comprises: downloading a new-version low-rank adapter from an adapter management end and saving same in an adapter repository; copying a low-rank adapter control block; and performing state migration processing on the basis of the low-rank adapter control block, so as to hot-update an old-version low-rank adapter using the new-version low-rank adapter. The present application achieves rapid hot-updating of a low-rank adapter.

Description

Low-rank adapter hot-update method, apparatus, storage medium and electronic device

[0001] This application claims priority to Chinese Patent Application No. 202411444259.5, filed on October 15, 2024, entitled “Low-rank adapter hot-update method, apparatus, storage medium and electronic device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, specifically to a method, apparatus, storage medium, and electronic device for hot-updating a low-rank adapter. Background Technology

[0003] With the development of technology, various large models are widely used. However, large models have a large number of parameters, and the cost of model training and deployment is high. As a result, the low-rank adaptation (LoRA) technique has emerged. Based on low-rank adaptation, only the base large model needs to be fine-tuned for a specific task to obtain a low-rank adapter (LoRA Adapter). Then, the low-rank adapter and the base large model are combined to form a low-rank adapted large model (LLM LoRA Adapter), which can effectively handle specific tasks.

[0004] With the large-scale deployment of low-rank adapters, updating these low-rank adapters often requires stopping the online service of large models or low-rank adapters (LoRA Adapters). Re-releasing low-rank adapted large models after updating low-rank adapters will interfere with the online inference process of the models. Technical issues

[0005] The current approach is not conducive to the rapid updating of low-rank adapters and cannot effectively support the rapid and flexible version iteration of large models, resulting in weak scalability and maintainability of large models. Technical solutions

[0006] This application provides a low-rank adapter hot update scheme, which can effectively realize the fast hot update of low-rank adapters, effectively support the fast and flexible version iteration of large models, and improve the scalability and maintainability of large models.

[0007] The embodiments of this application provide the following technical solutions:

[0008] According to one embodiment of this application, a method for hot-updating a low-rank adapter includes: downloading a new version of the low-rank adapter corresponding to an old version of the low-rank adapter from an adapter management terminal; storing the new version of the low-rank adapter in an adapter repository in local storage; copying and inheriting the low-rank adapter control block of the old version of the low-rank adapter; and performing state transition processing according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

[0009] In some embodiments of this application, the step of performing state transition processing according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository includes: removing the old version of the low-rank adapter from multiple adapter storage locations, including the local storage, main memory, and graphics processor memory; and querying and scheduling the new version of the low-rank adapter to the graphics processor memory in sequence from the graphics processor memory, the main memory, to the local storage, based on the low-rank adapter missing interrupt / low-rank adapter missing exception of the old version of the low-rank adapter and the low-rank adapter control block.

[0010] In some embodiments of this application, the step of performing state transition processing according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository includes: determining the location corresponding to the state from the local storage, main memory, and graphics processor memory based on the state in the low-rank adapter control block to obtain a target location; removing the old version of the low-rank adapter from the target location; and updating the new version of the low-rank adapter in the adapter repository to the target location.

[0011] In some embodiments of this application, the step of performing state transition processing based on the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository includes: determining the location corresponding to the state from the local storage, main memory, and graphics processor memory based on the state in the low-rank adapter control block to obtain a target location; changing the information in the low-rank adapter control block corresponding to the old version of the low-rank adapter at the target location to the information corresponding to the new version of the low-rank adapter; and removing the old version of the low-rank adapter from the target location.

[0012] In some embodiments of this application, before copying and inheriting the low-rank adapter control block of the old version low-rank adapter, the method further includes: verifying the new version low-rank adapter to obtain a verification result; and determining whether to copy and inherit the low-rank adapter control block of the old version low-rank adapter based on the verification result.

[0013] In some embodiments of this application, the local storage includes multiple adapter storage locations, including local storage, main memory, and graphics processor memory. The graphics processor memory stores a set of low-rank adapters used for large models, and the set of low-rank adapters includes multiple candidate low-rank adapters. The method further includes: determining that the access time of each candidate low-rank adapter is within a first time range or a second time range to obtain a target time range, wherein the first time range is further away from the current time than the second time range; calculating the scheduling priority of each candidate low-rank adapter according to a first and a second round-off weight corresponding to the target time range; and if an adapter needs to be swapped out, swapping each candidate low-rank adapter out of the graphics processor memory according to its scheduling priority.

[0014] In some embodiments of this application, calculating the scheduling priority of each candidate low-rank adapter based on the first selection weight corresponding to the first adapter set and the second selection weight corresponding to the second adapter set includes: calculating the first priority of each candidate low-rank adapter based on the interval duration corresponding to each candidate low-rank adapter, wherein the interval duration is the time elapsed between the current inference time and the last access time of the candidate low-rank adapter; calculating the second priority of each candidate low-rank adapter based on the access count corresponding to each candidate low-rank adapter, wherein the access count is the number of times the candidate low-rank adapter was accessed before the current inference time; and calculating the scheduling priority corresponding to each candidate low-rank adapter based on the first priority, the first selection weight, the second priority, and the second selection weight corresponding to each candidate low-rank adapter.

[0015] In some embodiments of this application, the step of calculating the first priority of each candidate low-rank adapter based on the interval duration corresponding to each candidate low-rank adapter includes: calculating the first priority of each candidate low-rank adapter according to the formula Priority1 = e -ax The first priority of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1).

[0016] In some embodiments of this application, the step of calculating the second priority of each candidate low-rank adapter based on the number of accesses corresponding to each candidate low-rank adapter includes: calculating the second priority of each candidate low-rank adapter according to the formula Priority2 = 1 - e -by The second priority of the candidate low-rank adapter is calculated, where b is the second coefficient, b>0, y is the access count corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1).

[0017] In some embodiments of this application, the step of calculating the scheduling priority corresponding to each candidate low-rank adapter based on the first priority, the first selection weight, the second priority, and the second selection weight corresponding to each candidate low-rank adapter includes: calculating the scheduling priority Priority3 corresponding to the candidate low-rank adapter according to the formula Priority3=μ[α(Priority1)+β(Priority2)], where α and β are the first selection weight and the second selection weight, respectively, α,β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

[0018] In some embodiments of this application, the graphics processor memory is divided into multiple storage spaces, each used to store different parameters; the multiple storage spaces include an adapter storage space, which stores multiple candidate low-rank adapters in pages; the step of swapping out each candidate low-rank adapter from the graphics processor memory according to its scheduling priority includes: determining the candidate low-rank adapter to be swapped out according to its scheduling priority; obtaining the address set of the candidate low-rank adapter in the adapter storage space according to the low-rank adapter control block corresponding to the candidate low-rank adapter to be swapped out, and obtaining a target address set; determining the page of the candidate low-rank adapter to be swapped out in the adapter storage space according to the target address set, and obtaining a target page; and swapping out the candidate low-rank adapter to be swapped out from the target page in the adapter storage space.

[0019] According to one embodiment of this application, a low-rank adapter hot-update apparatus includes: a download module for downloading a new version of a low-rank adapter corresponding to an old version of the low-rank adapter from an adapter management terminal; a storage module for storing the new version of the low-rank adapter in an adapter repository in local storage; a copy module for copying a low-rank adapter control block that inherits from the old version of the low-rank adapter; and an update module for performing state transition processing based on the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

[0020] In some embodiments of this application, the update module is configured to: remove the old version low-rank adapter from multiple adapter storage locations, the multiple adapter storage locations including the local storage, main memory, and graphics processor memory; and, based on the low-rank adapter missing interrupt / low-rank adapter missing exception of the old version low-rank adapter and the low-rank adapter control block, sequentially query and schedule the new version low-rank adapter to the graphics processor memory from the graphics processor memory, the main memory, and the local storage.

[0021] In some embodiments of this application, the update module is configured to: determine the location corresponding to the state from the local storage, main memory, and graphics processor memory based on the state in the low-rank adapter control block, and obtain a target location; remove the old version of the low-rank adapter from the target location; and update the new version of the low-rank adapter in the adapter repository to the target location.

[0022] In some embodiments of this application, the update module is configured to: determine the location corresponding to the state from the local storage, main memory, and graphics processor memory based on the state in the low-rank adapter control block, and obtain a target location; change the information in the low-rank adapter control block corresponding to the old version low-rank adapter in the target location to the information corresponding to the new version low-rank adapter; and remove the old version low-rank adapter from the target location.

[0023] In some embodiments of this application, before copying and inheriting the low-rank adapter control block of the old version low-rank adapter, the device further includes a verification module for: verifying the new version low-rank adapter and obtaining a verification result; and determining whether to copy and inherit the low-rank adapter control block of the old version low-rank adapter based on the verification result.

[0024] In some embodiments of this application, the local storage includes multiple adapter storage locations, including the local storage, main memory, and graphics processor memory. The graphics processor memory stores a set of low-rank adapters for use with large models, and the set of low-rank adapters includes multiple candidate low-rank adapters. The device further includes a scheduling module for: determining whether the access time of each candidate low-rank adapter is within a first time range or a second time range to obtain a target time range, wherein the first time range is further away from the current time than the second time range; calculating the scheduling priority of each candidate low-rank adapter according to a first and a second round-off weight corresponding to the target time range; and if an adapter needs to be swapped out, swapping each candidate low-rank adapter out of the graphics processor memory according to its scheduling priority.

[0025] In some embodiments of this application, the scheduling module is configured to: calculate a first priority for each candidate low-rank adapter based on the interval duration corresponding to each candidate low-rank adapter, wherein the interval duration is the time elapsed between the current inference time and the last access time of the candidate low-rank adapter; calculate a second priority for each candidate low-rank adapter based on the number of accesses corresponding to each candidate low-rank adapter, wherein the number of accesses is the number of times the candidate low-rank adapter was accessed before the current inference time; and calculate a scheduling priority corresponding to each candidate low-rank adapter based on the first priority, the first selection weight, the second priority, and the second selection weight corresponding to each candidate low-rank adapter.

[0026] In some embodiments of this application, the scheduling module is used to: according to the formula Priority1 = e -ax The first priority of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1).

[0027] In some embodiments of this application, the scheduling module is used to: according to the formula Priority2 = 1 - e -by The second priority of the candidate low-rank adapter is calculated, where b is the second coefficient, b>0, y is the access count corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1).

[0028] In some embodiments of this application, the scheduling module is used to: calculate the scheduling priority Priority3 corresponding to the candidate low-rank adapter according to the formula Priority3=μ[α(Priority1)+β(Priority2)], where α and β are the first and second selection weights respectively, α,β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

[0029] In some embodiments of this application, the graphics processor memory is divided into multiple storage spaces, each used to store different parameters; the multiple storage spaces include an adapter storage space, which stores the multiple candidate low-rank adapters in pages; the scheduling module is configured to: determine the candidate low-rank adapter to be swapped out according to the scheduling priority of each candidate low-rank adapter; obtain the address set of the candidate low-rank adapter to be swapped out in the adapter storage space according to the low-rank adapter control block corresponding to the candidate low-rank adapter to be swapped out, and obtain a target address set; determine the page of the candidate low-rank adapter to be swapped out in the adapter storage space according to the target address set, and obtain a target page; swap out the candidate low-rank adapter to be swapped out from the target page in the adapter storage space.

[0030] According to another embodiment of this application, a storage medium stores a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the methods described in the embodiments of this application.

[0031] According to another embodiment of this application, an electronic device may include: a memory storing a computer program; and a processor reading the computer program stored in the memory to execute the methods described in the embodiments of this application.

[0032] According to another embodiment of this application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in the embodiments of this application. Beneficial effects

[0033] In this embodiment, a new version of the low-rank adapter corresponding to the old version of the low-rank adapter is downloaded from the adapter management terminal; the new version of the low-rank adapter is stored in the adapter repository in local storage; the low-rank adapter control block that inherits from the old version of the low-rank adapter is copied; and state transition processing is performed according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

[0034] In this way, after downloading the new version of the low-rank adapter corresponding to the old version, the new version of the low-rank adapter is saved in the adapter repository in local storage. Then, the low-rank adapter control block of the old version of the low-rank adapter is copied. Based on the copied low-rank adapter control block, the old version of the low-rank adapter is hot-updated using the new version of the low-rank adapter in the adapter repository through state transition. The low-rank adapter can be updated without stopping the online service of the large model, without redeploying the large model, and without redeploying the low-rank adapter. This achieves fast hot updates of the low-rank adapter, effectively supporting the rapid and flexible version iteration of the large model and improving the scalability and maintainability of the large model. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 shows a flowchart of a low-rank adapter hot update method according to an embodiment of this application.

[0037] Figure 2 shows a flowchart of a low-rank adapter hot update according to another embodiment of this application.

[0038] Figure 3 shows a schematic diagram of low-rank adapter management according to an embodiment of this application.

[0039] Figure 4 shows a flowchart of the state transition of a low-rank adapter according to an embodiment of this application.

[0040] Figure 5 shows a schematic diagram of low-rank adapter partitioning according to an embodiment of this application.

[0041] Figure 6 shows a schematic diagram of a low-rank adapter storage according to an embodiment of this application.

[0042] Figure 7 shows a block diagram of a low-rank adapter hot-update device according to an embodiment of this application.

[0043] Figure 8 shows a block diagram of an electronic device according to an embodiment of this application.

[0044] Implementation methods of this application

[0045] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present disclosure and are not intended to limit the present disclosure. Furthermore, the embodiments provided below are some embodiments for implementing the present disclosure, and not all embodiments for implementing the present disclosure. Unless otherwise specified, the technical solutions described in the embodiments of the present disclosure can be implemented in any combination.

[0046] It should be noted that, in the embodiments of this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of circuitry, processors, programs, or software, etc.) in the method or apparatus that includes that element.

[0047] For example, the low-rank adapter hot update method provided in this disclosure includes a series of steps, but the low-rank adapter hot update method provided in this disclosure is not limited to the steps described. Similarly, the low-rank adapter hot update device provided in this disclosure includes a series of units, but the device provided in this disclosure is not limited to the units explicitly described, but may also include units that need to be set up for obtaining relevant information or processing based on the information.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure.

[0049] It is understood that in the specific implementation of this application, relevant data is involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0050] Figure 1 schematically illustrates a flowchart of a low-rank adapter hot-update method according to an embodiment of this application. The entity performing the low-rank adapter hot-update method can be a device or a server.

[0051] As shown in Figure 1, the low-rank adapter hot update method may include steps S110 to S140.

[0052] Step S110: Download the new version of the low-rank adapter corresponding to the old version of the low-rank adapter from the adapter management terminal; Step S120: Save the new version of the low-rank adapter in the adapter repository in local storage; Step S130: Copy the low-rank adapter control block that inherits from the old version of the low-rank adapter; Step S140: Perform state transition processing according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

[0053] The adapter management terminal can be a management terminal used to publish low-rank adapters. Referring to Figure 2, the execution entity 210 can download the new version of the low-rank adapter corresponding to the old version from the adapter management terminal 220 and save the new version of the low-rank adapter in the adapter repository in the local storage. The local storage is other storage besides the CPU's main memory and the graphics processor's GPU memory, such as a local disk or other storage.

[0054] In the embodiments of this application, referring to Figure 2, all low-rank adapters are maintained in local storage. Some highly important low-rank adapters can be maintained simultaneously in local storage, main memory, and GPU memory through manual configuration or by manually initiating a low-rank adapter missing interruption / exception. Low-rank adapters needed for subsequent batches are scheduled from local storage to main memory. Furthermore, low-rank adapters needed for the current batch are scheduled from main memory to GPU memory (e.g., in case of a low-rank adapter missing interruption / exception). When the large model processes a specific inference task, it uses the adapter weights of the low-rank adapters corresponding to that specific inference task in GPU memory, along with other basic model weights, for inference. In other words, the low-rank adapters are combined with the large model for inference.

[0055] For each low-rank adapter, a corresponding low-rank adapter control block (LACB) records the adapter's state and attributes. In other words, each low-rank adapter has a low-rank adapter control block that inherits its state and attribute information. By copying this low-rank adapter control block that inherits from the old version of the low-rank adapter, the state and attribute information of the old version of the low-rank adapter can be obtained through the copied low-rank adapter control block.

[0056] In one example, the state of a low-rank adapter can include: Created (creating the LoRA Adapter Control Block (LACB), entering the scheduling process), Updating (the low-rank adapter is in the process of upgrading, and in this state, the upgrading process can be restricted from being interrupted), Updated (the low-rank adapter has been upgraded and can transition to the Created state, ready to execute the subsequent scheduling process), Waiting (the low-rank adapter is in the adapter repository and is ready to be scheduled into main memory at any time), Ready (the low-rank adapter is in main memory and is ready to be scheduled into GPU memory at any time), and Running (the low-rank adapter is in GPU memory and is ready to be combined with the large model (LLM Base Model) for inference at any time). It is understandable that in other examples, additional states can be added for finer-grained management based on business needs.

[0057] Multiple low-rank adapters in the same state are divided into corresponding state pools. Referring to Figure 3, multiple low-rank adapters in the Running state in GPU memory are divided into a Running Pool 300. Similarly, multiple low-rank adapters in the Waiting state in local storage can be divided into a Waiting Pool, and multiple low-rank adapters in the Ready state in main memory can be divided into a Ready Pool. Each low-rank adapter 310 has a corresponding Low-Rank Adapter Control Block (LACB) 310 that inherits its state and attribute information. For example, referring to Figure 3, the Low-Rank Adapter Control Block (LACB) can record, but is not limited to, the low-rank adapter's ID, version, page table information, state, and timestamp. The Fork method allows you to copy the low-rank adapter control block of the old version of the low-rank adapter by inheritance (Fork / clone). Therefore, you can obtain information such as the state and attributes of the old version of the low-rank adapter in different states through the copied low-rank adapter control block.

[0058] State transition processing is performed based on information such as the state and attributes in the low-rank adapter control block. The old version of the low-rank adapter is hot-updated using the new version of the low-rank adapter in the adapter repository. Hot updates of the low-rank adapter can be performed based on the state information of the low-rank adapter control block. The low-rank adapter can be updated without stopping the online inference service of large models.

[0059] The low-rank adapter (LoRA Adapter) + base model (LLM Base Model) can typically provide two inference service methods: (1) merging the low-rank adapter into the base model and then using the merged base model for inference; (2) directly performing inference using the base model + low-rank adapter without merging. In the embodiment of this application, there is no need to redeploy the base model or the low-rank adapter. The new version of the low-rank adapter can be seamlessly integrated into the current batch of base model inference processes, resulting in minimal user perception.

[0060] In this way, after downloading the new version of the low-rank adapter corresponding to the old version, the new version of the low-rank adapter is saved in the adapter repository in local storage. Then, the low-rank adapter control block of the old version of the low-rank adapter is copied. Based on the copied low-rank adapter control block, the old version of the low-rank adapter is hot-updated using the new version of the low-rank adapter in the adapter repository through state transition. The low-rank adapter can be updated without stopping the online service of the large model, without redeploying the large model, and without redeploying the low-rank adapter. This achieves rapid hot updates of the low-rank adapter, effectively supports the rapid and flexible version iteration of the large model, and improves the scalability and maintainability of the large model.

[0061] The following describes further optional embodiments of the steps performed during low-rank adapter hot updates in the embodiment shown in Figure 1.

[0062] In one embodiment, the state transition process based on the low-rank adapter control block to hot-update the old low-rank adapter using the new version of the low-rank adapter in the adapter repository may include: removing the old low-rank adapter from multiple adapter storage locations, including the local storage, main memory, and graphics processor memory; and querying and scheduling the new low-rank adapter to the graphics processor memory sequentially from the graphics processor memory, the main memory, to the local storage based on the low-rank adapter's missing low-rank interrupt / missing low-rank exception and the low-rank adapter control block.

[0063] In this embodiment, during state transition, the old version of the low-rank adapter is first removed from local storage, main memory, and graphics processor memory. Then, based on the low-rank adapter's low-rank adapter fault / low-rank adapter failure message and the low-rank adapter control block, the new version of the low-rank adapter is sequentially queried and scheduled from graphics processor memory, main memory, to local storage to the graphics processor memory. Specifically, the low-rank adapter fault / low-rank adapter failure message refers to the LoRA Adapter failure / LoRA Adapter interruption message.

[0064] For example, referring to Figure 4, after removing the old version of LoRA Adapter7 from local storage, main memory, and GPU memory, S7 (A7) indicates that when inference task 7 (Sequence 7) wants to start inference using LoRA Adapter7, step S410 is used to query the low-rank adapter from the running pool. If LoRA Adapter7 is not found in the running pool in GPU memory, a low-rank adapter fault / low-rank adapter exception 1 (Adapter Fault 1) is initiated. Then, step S420 is executed to query the low-rank adapter from the ready pool. Conversely, if LoRA Adapter7 is found in the running pool in GPU memory, step S430 is executed to run iterative-level inference, so that the large model uses LoRA Adapter7 in the running pool in GPU memory for the current batch of inference.

[0065] During step S420, if LoRA Adapter7 is not found in the Ready Pool in Main Memory, a Low-Rank Adapter Fault 2 is triggered. Then, step S440 is executed to fetch the low-rank adapter from the Waiting Pool. If step S420 finds LoRA Adapter7 in the Ready Pool in Main Memory, step S450 is executed to load the low-rank adapter from the Ready Pool into the Running Pool, and LoRA Adapter7 is scheduled from the Ready Pool to the Running Pool.

[0066] After performing the query in step S440, if LoRA Adapter7 is found in the waiting pool in local storage, then step S460 loads the low-rank adapter from the waiting pool to the ready pool, scheduling LoRA Adapter7 from the waiting pool to the ready pool. Further, step S450 can be performed to schedule LoRA Adapter7 from the ready pool to the running pool. This allows new versions of LoRA Adapter7 to be scheduled to the running pool level by level.

[0067] In addition, during the hierarchical scheduling of the new version of the low-rank adapter, the low-rank adapter control block inheriting the new version of the low-rank adapter can be updated in the running pool, ready pool, and waiting pool in the graphics processor memory, main memory, and local storage respectively, based on the copied low-rank adapter control block.

[0068] It should be noted that Large Model (LLM) inference services typically use continuous batching technology for request batching. Continuous batching is a technique that allows one or more inference requests to be combined into a single batch and the batch size to be dynamically adjusted as needed to maximize throughput.

[0069] Specifically, as shown in Figure 4, for a continuous batching request, continuous batching can be organized at the sequence level according to the large rectangles corresponding to T1 to T7 in Figure 4, and processed on a sequence-by-sequence basis. There are a total of 7 sequences within this large rectangle, where Si represents the i-th sequence; for example, S1 represents sequence 1. Furthermore, the batch size is 4. For example, each column corresponding to T1 to T7 within this large rectangle includes 4 smaller rectangles, indicating that the batch size for one batch is 4.

[0070] Furthermore, during the inference process, the large model represents the status of each token at the iteration level, as shown in Figure 4, where each small rectangle within the large rectangle represents a token. Specifically, the [plain white background rectangle] represents Prompt Tokens, which are the user's prompts input to the large model, broken down into smaller units of tokens. The [diagonal background rectangle] represents Generated Tokens, which are the tokens derived from the output text generated by the large model based on the input Prompt Tokens. The [horizontal background rectangle] represents End of Sequence Tokens, indicating the end of a sequence. The [horizontal and vertical background rectangles] represent Adapter Fault / Low-Rank Adapter Fault.

[0071] For example, in Figure 4, S1(A1) [plain white background rectangle] represents the prompt tokens in Sequence 1 (Adapter 1) during inference using the low-rank adapter 1; S1(A1) [diagonal background rectangle] represents the output text tokens in Sequence 1 (Adapter 1) during inference using the low-rank adapter 1; END [horizontal background rectangle] after S1(A1) represents the end of sequence token; S7(A7) [horizontal and vertical background rectangles] indicates that when Sequence 7 of the large model is inference and wants to start iterative-level inference using the low-rank adapter 7, the query in step S410 finds that the low-rank adapter 7 is not in the GPU memory running pool, thus triggering a low-rank adapter fault / low-rank adapter failure. Initiating a low-rank adapter fault / low-rank adapter failure triggers the scheduling process described in the aforementioned embodiments.

[0072] In one embodiment, the step of performing state transition processing according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository may further include:

[0073] Based on the state in the low-rank adapter control block, determine the location corresponding to the state from the local storage, main memory, and graphics processor memory to obtain the target location; remove the old version of the low-rank adapter from the target location; and update the new version of the low-rank adapter in the adapter repository to the target location.

[0074] In this embodiment, the target location is obtained by determining the location corresponding to the state in the low-rank adapter control block from the local storage, main memory, and graphics processor memory. For example, if the state in the copied low-rank adapter control block is "Waiting," the location corresponding to "Waiting" is the "Waiting Pool" in the local storage. Therefore, the target location is the "Waiting Pool" in the local storage.

[0075] Then, the old version of the low-rank adapter is removed from the target location, and the new version of the low-rank adapter in the adapter repository is updated to the target location. In this way, the adapter hot update can be performed with less user awareness. In this embodiment, the user will only be briefly aware of it when the target location includes the running pool in the GPU memory.

[0076] Furthermore, in one embodiment, the step of performing state transition processing according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository may include:

[0077] Based on the state in the low-rank adapter control block, determine the location corresponding to the state from the local storage, main memory, and graphics processor memory to obtain the target location; change the information in the low-rank adapter control block corresponding to the old version low-rank adapter in the target location to the information corresponding to the new version low-rank adapter; remove the old version low-rank adapter from the target location.

[0078] In this embodiment, based on the state in the low-rank adapter control block, the location corresponding to the state is determined from local storage, main memory, and GPU memory. After obtaining the target location, the information in the low-rank adapter control block corresponding to the old version low-rank adapter at the target location is changed to the information corresponding to the new version low-rank adapter. Then, the old version low-rank adapter is removed from the target location. Specifically, the pointer representing the real address of the old version low-rank adapter in the low-rank adapter control block corresponding to the old version low-rank adapter is modified to point to the address of the new version low-rank adapter. When the target location is the running pool in GPU memory, the old version low-rank adapter can be retained to ensure that the inference process of the large model is not interrupted by the upgrade process. When the new version low-rank adapter is created, a corresponding low-rank adapter control block is created in the corresponding storage space (e.g., main memory, GPU memory, etc.), and the page table information in the low-rank adapter control block is updated. The page table information is the address set of the storage space (e.g., main memory, GPU memory, etc.) of the new version low-rank adapter, and the address set in the page table information points to the specific address of the new version low-rank adapter in the storage space. This approach can further reduce the user's perception of hot updates for low-rank adapters.

[0079] In a further embodiment, before copying and inheriting the low-rank adapter control block of the old version low-rank adapter, the method further includes: verifying the new version low-rank adapter to obtain a verification result; and determining whether to copy and inherit the low-rank adapter control block of the old version low-rank adapter based on the verification result.

[0080] After downloading the new version of the low-rank adapter, it is verified. If the verification passes, the low-rank adapter control block of the old version is copied and inherited. Conversely, if the verification fails, the low-rank adapter control block of the old version is not copied and inherited. This ensures the reliability of adapter hot updates. The verification methods for the new version of the low-rank adapter include, but are not limited to, MD5 checksum verification, compatibility verification with large models, and verification of the correctness of inference results.

[0081] Furthermore, in one embodiment, the local storage includes multiple adapter storage locations, including the local storage, main memory, and graphics processor memory. The graphics processor memory stores a set of low-rank adapters for use with large models, and the set of low-rank adapters includes multiple candidate low-rank adapters.

[0082] The method may further include: determining that the access time of each candidate low-rank adapter is within a first time range or a second time range to obtain a target time range, wherein the first time range is further away from the current time than the second time range; calculating the scheduling priority of each candidate low-rank adapter according to a first discard weight and a second discard weight corresponding to the target time range; and if an adapter needs to be swapped out, swapping each candidate low-rank adapter out of the graphics processor memory according to the scheduling priority of each candidate low-rank adapter.

[0083] When a low-rank adapter needs to be scheduled into the GPU memory, if GPU memory is limited, existing low-rank adapters in GPU memory must be swapped out. For example, when a low-rank adapter fault occurs, the corresponding low-rank adapter needs to be scheduled from main memory into GPU memory. If there is no free space in GPU memory, some low-rank adapters can be swapped out from GPU memory to main memory based on the scheduling priority of each candidate low-rank adapter to free up space for the low-rank adapter required by the current inference request. The GPU memory required by the low-rank adapter needs to be locked in advance. Frequent swapping of low-rank adapters consumes a lot of resources, so choosing which low-rank adapter to swap out becomes extremely important.

[0084] To increase GPU utilization and improve inference efficiency during inference, large models utilize the aforementioned batching mechanism as much as possible. The iteration-level scheduling policies of mainstream large model inference services (Text-Generation-Inference, vLLM, TensorRT-LLM, etc.) typically support continuous batching. This technique does not disrupt the order of request processing, meaning that requests within the same batch are only temporally related, not logically related. The low-rank adapter set used by the request set belonging to the same batch must be loaded into GPU memory simultaneously during token generation for this round of iteration-level inference.

[0085] The technical characteristics of continuous batching mean that the set of low-rank adapters that have just been accessed in this time-related request set are likely to be used again in the next token generation iteration "unless the end-of-sequence marker or the maximum generation length limit is encountered". Therefore, the set of low-rank adapters that have just been accessed must be retained for a short period of time (not allowed to be swapped out of GPU memory) in order to successfully complete the request inference process of this batch. That is, in this scenario, it is desirable for the access time of the low-rank adapters to be swapped out in the near future to be as far away from the current time as possible (approximately LRU algorithm, with time factor taking priority). The low-rank adapters to be swapped out in the far future are expected not to be accessed in the next batch (batch inference process) to reduce the overhead caused by temporarily loading low-rank adapters (for example, LFU algorithm can be used for approximate prediction, with frequency factor taking priority and frequency decaying over time).

[0086] In this embodiment of the application, the time range is divided into a first time range or a second time range according to the order of access time (for example, as shown in Figure 5, the time before the current time is divided into a first time range LRU with Decay and a second time range LFU with Decay at point A); the scheduling priority of each candidate low-rank adapter is calculated according to the first and second pick-up weights corresponding to the target time range where the access time of each candidate low-rank adapter is located; if an adapter needs to be swapped out, each candidate low-rank adapter is swapped out from the graphics processor memory according to the scheduling priority of each candidate low-rank adapter. For example, one or more candidate low-rank adapters with the lowest scheduling priority can be swapped out from the graphics processor memory, which can reliably solve the problems existing under the technical characteristics of continuous batching and further improve the reliability of adapter hot update.

[0087] In one embodiment, calculating the scheduling priority of each candidate low-rank adapter based on the first and second round-off weights corresponding to the target time range may include:

[0088] Based on the interval duration corresponding to each candidate low-rank adapter, the first priority of each candidate low-rank adapter is calculated, wherein the interval duration is the time elapsed between the current inference time and the last access time of the candidate low-rank adapter; based on the access count corresponding to each candidate low-rank adapter, the second priority of each candidate low-rank adapter is calculated, wherein the access count is the number of times the candidate low-rank adapter was accessed before the current inference time; based on the first priority, the first pick-up weight, the second priority, and the second pick-up weight corresponding to each candidate low-rank adapter, the scheduling priority corresponding to each candidate low-rank adapter is calculated.

[0089] Specifically, in one approach, calculating the first priority of each candidate low-rank adapter based on the interval duration corresponding to each candidate low-rank adapter may include: using the formula Priority1 = e -ax The first priority of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1).

[0090] The first priority of the candidate low-rank adapter is calculated based on the formula. x represents the time elapsed between the current time of the current iteration-level inference and the last access time of the adapter. a is used as a weight to control the rate of decrease over time. The larger a is, the faster the rate of decrease.

[0091] In one approach, calculating the second priority of each candidate low-rank adapter based on the number of accesses corresponding to each candidate low-rank adapter includes: using the formula Priority2 = 1 - e -by The second priority of the candidate low-rank adapter is calculated, where b is the second coefficient, b>0, y is the access count corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1).

[0092] According to the formula Priority2 = 1 - e -by The second priority of the candidate low-rank adapter is calculated, where y represents the number of times the candidate low-rank adapter is accessed before the current iteration-level inference. b is used as a weight to control the rate at which the priority decreases with the number of accesses. The larger b is, the faster the rate of increase.

[0093] In one approach, the step of calculating the scheduling priority corresponding to each candidate low-rank adapter based on the first priority, the first selection weight, the second priority, and the second selection weight corresponding to each candidate low-rank adapter includes: calculating the scheduling priority Priority3 corresponding to the candidate low-rank adapter according to the formula Priority3=μ[α(Priority1)+β(Priority2)], where α and β are the first selection weight and the second selection weight, respectively, α,β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

[0094] Referring to Figure 5, α and β can be the selection weights for the first time-range LRU with Decay and the second time-range LFU with Decay, respectively, satisfying α, β ∈ [0, 1] and α + β = 1. The dividing point A in Figure 5 (an empirical value, or given by a dynamic formula) indicates that if the candidate low-rank adapter is in the first time-range LRU with Decay, then α > β; if the candidate low-rank adapter is in the second time-range LFU with Decay, then α < β. μ represents the weight of external or manual intervention (for example, this parameter can be used to ensure that an important candidate low-rank adapter remains resident in GPU Memory for real-time inference without being swapped out to Main Memory). α and β can be specified empirically through program logic or calculated dynamically.

[0095] In one further embodiment, the graphics processor memory is divided into multiple storage spaces, each used to store different parameters. These multiple storage spaces include an adapter storage space, which stores the multiple candidate low-rank adapters in pages. Furthermore, the multiple storage spaces also include a model weight storage space, a key-value cache storage space, and other parameter storage spaces.

[0096] Referring to Figure 6, in this embodiment, the GPU memory is divided into multiple storage spaces, each used to store different parameters. One of these storage spaces is an adapter storage space, which stores the multiple candidate low-rank adapters in pages. Furthermore, this application allows for adapter updates to be performed solely within this adapter storage space. Compared to related technologies where model weights (Base Model), adapter weights (Adapter), KV cache, and other parameters are stored in a unified space, this embodiment's space-divided storage method facilitates the management and hot updates of low-rank adapters.

[0097] Furthermore, in the foregoing embodiments, the step of swapping out each of the candidate low-rank adapters from the graphics processor memory may specifically include: swapping out each of the candidate low-rank adapters from the graphics processor memory according to the scheduling priority of each candidate low-rank adapter, including: determining the candidate low-rank adapter to be swapped out according to the scheduling priority of each candidate low-rank adapter; obtaining the address set of the candidate low-rank adapter to be swapped out in the adapter storage space according to the low-rank adapter control block corresponding to the candidate low-rank adapter to be swapped out, and obtaining a target address set; determining the page of the candidate low-rank adapter to be swapped out in the adapter storage space according to the target address set, and obtaining a target page; swapping out the candidate low-rank adapter to be swapped out from the target page in the adapter storage space.

[0098] The adapter storage space stores multiple candidate low-rank adapters in pages. When a candidate low-rank adapter is created, a corresponding low-rank adapter control block is created in the corresponding storage space (such as graphics processor memory), and the page table information in the low-rank adapter control block is updated. The page table information is the address set of the candidate low-rank adapter in the storage space (such as main memory, graphics processor memory, etc.), and the address set in the page table information points to the specific address of the candidate low-rank adapter in the storage space.

[0099] Therefore, after determining the candidate low-rank adapters to be replaced based on the scheduling priority of each candidate low-rank adapter (for example, determining one or more candidate low-rank adapters with the lowest scheduling priority as candidate low-rank adapters to be replaced), the address set of the candidate low-rank adapters to be replaced in the adapter storage space can be obtained according to the low-rank adapter control block corresponding to the candidate low-rank adapters to be replaced, and this address set is used as the target address set.

[0100] Furthermore, based on this target address set, the page of the candidate low-rank adapter to be swapped out in the adapter storage space can be determined, and this page is the target page. Therefore, the candidate low-rank adapter to be swapped out can be accurately swapped out from this target page in the adapter storage space.

[0101] It should be noted that the Graphics Processing Unit (GPU) in this article can be technically replaced by other devices that can accelerate model computation, such as Tensor Processing Unit (TPU) and Neural Processing Unit (NPU). Similarly, GPU memory can be technically replaced by the memory of a corresponding device that can accelerate model computation. Furthermore, the technical solution described in this article can also be expressed using a computer cluster.

[0102] To facilitate better implementation of the low-rank adapter hot-update method provided in this application, this application also provides a low-rank adapter hot-update apparatus based on the above-described low-rank adapter hot-update method. The meanings of the terms used are the same as in the low-rank adapter hot-update method described above, and specific implementation details can be found in the description of the method embodiments. Figure 7 shows a block diagram of a low-rank adapter hot-update apparatus according to an embodiment of this application.

[0103] As shown in Figure 7, the low-rank adapter hot update device 500 may include: a download module 510, used to download the new version of the low-rank adapter corresponding to the old version of the low-rank adapter from the adapter management terminal; a storage module 520, used to store the new version of the low-rank adapter in the adapter repository in local storage; a copy module 530, used to copy the low-rank adapter control block that inherits from the old version of the low-rank adapter; and an update module 540, used to perform state transition processing according to the low-rank adapter control block, so as to use the new version of the low-rank adapter in the adapter repository to hot update the old version of the low-rank adapter.

[0104] In some embodiments of this application, the update module is configured to: remove the old version low-rank adapter from multiple adapter storage locations, the multiple adapter storage locations including the local storage, main memory, and graphics processor memory; and, based on the low-rank adapter missing interrupt / low-rank adapter missing exception of the old version low-rank adapter and the low-rank adapter control block, sequentially query and schedule the new version low-rank adapter to the graphics processor memory from the graphics processor memory, the main memory, and the local storage.

[0105] In some embodiments of this application, the update module is configured to: determine the location corresponding to the state from the local storage, main memory, and graphics processor memory based on the state in the low-rank adapter control block, and obtain a target location; remove the old version of the low-rank adapter from the target location; and update the new version of the low-rank adapter in the adapter repository to the target location.

[0106] In some embodiments of this application, the update module is configured to: determine the location corresponding to the state from the local storage, main memory, and graphics processor memory based on the state in the low-rank adapter control block, and obtain a target location; change the information in the low-rank adapter control block corresponding to the old version low-rank adapter in the target location to the information corresponding to the new version low-rank adapter; and remove the old version low-rank adapter from the target location.

[0107] In some embodiments of this application, before copying and inheriting the low-rank adapter control block of the old version low-rank adapter, the device further includes a verification module for: verifying the new version low-rank adapter and obtaining a verification result; and determining whether to copy and inherit the low-rank adapter control block of the old version low-rank adapter based on the verification result.

[0108] In some embodiments of this application, the local storage includes multiple adapter storage locations, including the local storage, main memory, and graphics processor memory. The graphics processor memory stores a set of low-rank adapters for use with large models, and the set of low-rank adapters includes multiple candidate low-rank adapters. The device further includes a scheduling module for: determining whether the access time of each candidate low-rank adapter is within a first time range or a second time range to obtain a target time range, wherein the first time range is further away from the current time than the second time range; calculating the scheduling priority of each candidate low-rank adapter according to a first and a second round-off weight corresponding to the target time range; and if an adapter needs to be swapped out, swapping each candidate low-rank adapter out of the graphics processor memory according to its scheduling priority.

[0109] In some embodiments of this application, the scheduling module is configured to: calculate a first priority for each candidate low-rank adapter based on the interval duration corresponding to each candidate low-rank adapter, wherein the interval duration is the time elapsed between the current inference time and the last access time of the candidate low-rank adapter; calculate a second priority for each candidate low-rank adapter based on the number of accesses corresponding to each candidate low-rank adapter, wherein the number of accesses is the number of times the candidate low-rank adapter was accessed before the current inference time; and calculate a scheduling priority corresponding to each candidate low-rank adapter based on the first priority, the first selection weight, the second priority, and the second selection weight corresponding to each candidate low-rank adapter.

[0110] In some embodiments of this application, the scheduling module is used to: according to the formula Priority1 = e -ax The first priority of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1).

[0111] In some embodiments of this application, the scheduling module is used to: according to the formula Priority2 = 1 - e -by The second priority of the candidate low-rank adapter is calculated, where b is the second coefficient, b>0, y is the access count corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1).

[0112] In some embodiments of this application, the scheduling module is used to: calculate the scheduling priority Priority3 corresponding to the candidate low-rank adapter according to the formula Priority3=μ[α(Priority1)+β(Priority2)], where α and β are the first and second selection weights respectively, α,β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

[0113] In some embodiments of this application, the graphics processor memory is divided into multiple storage spaces, each used to store different parameters; the multiple storage spaces include an adapter storage space, which stores the multiple candidate low-rank adapters in pages; the scheduling module is configured to: determine the candidate low-rank adapter to be swapped out according to the scheduling priority of each candidate low-rank adapter; obtain the address set of the candidate low-rank adapter to be swapped out in the adapter storage space according to the low-rank adapter control block corresponding to the candidate low-rank adapter to be swapped out, and obtain a target address set; determine the page of the candidate low-rank adapter to be swapped out in the adapter storage space according to the target address set, and obtain a target page; swap out the candidate low-rank adapter to be swapped out from the target page in the adapter storage space.

[0114] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0115] Furthermore, this application also provides an electronic device, as shown in FIG8, which is a block diagram of an electronic device according to an embodiment of this application. Specifically:

[0116] The electronic device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will understand that the electronic device structure shown in FIG8 does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0117] The processor 601 is the control center of the electronic device. It connects to various parts of the computer device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 602, and by calling data stored in the memory 602, it performs various functions of the computer device and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and application programs, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 601.

[0118] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0119] The electronic device also includes a power supply 603 that supplies power to the various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 603 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0120] The electronic device may also include an input unit 604, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0121] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 601 in the electronic device loads the executable files corresponding to the processes of one or more computer programs into the memory 602 according to the following instructions, and the processor 601 runs the computer programs stored in the memory 602, thereby realizing the various functions in the foregoing embodiments of this application. For example, the processor 601 can perform the following steps:

[0122] Download the new version of the low-rank adapter corresponding to the old version of the low-rank adapter from the adapter management terminal; save the new version of the low-rank adapter in the adapter repository in local storage; copy the low-rank adapter control block that inherits from the old version of the low-rank adapter; perform state transition processing according to the low-rank adapter control block, so as to use the new version of the low-rank adapter in the adapter repository to hot update the old version of the low-rank adapter.

[0123] In some embodiments of this application, the step of performing state transition processing according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository includes: removing the old version of the low-rank adapter from multiple adapter storage locations, including the local storage, main memory, and graphics processor memory; and querying and scheduling the new version of the low-rank adapter to the graphics processor memory in sequence from the graphics processor memory, the main memory, to the local storage, based on the low-rank adapter missing interrupt / low-rank adapter missing exception of the old version of the low-rank adapter and the low-rank adapter control block.

[0124] In some embodiments of this application, the step of performing state transition processing according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository includes: determining the location corresponding to the state from the local storage, main memory, and graphics processor memory based on the state in the low-rank adapter control block to obtain a target location; removing the old version of the low-rank adapter from the target location; and updating the new version of the low-rank adapter in the adapter repository to the target location.

[0125] In some embodiments of this application, the step of performing state transition processing based on the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository includes: determining the location corresponding to the state from the local storage, main memory, and graphics processor memory based on the state in the low-rank adapter control block to obtain a target location; changing the information in the low-rank adapter control block corresponding to the old version of the low-rank adapter at the target location to the information corresponding to the new version of the low-rank adapter; and removing the old version of the low-rank adapter from the target location.

[0126] In some embodiments of this application, before copying and inheriting the low-rank adapter control block of the old version low-rank adapter, the method further includes: verifying the new version low-rank adapter to obtain a verification result; and determining whether to copy and inherit the low-rank adapter control block of the old version low-rank adapter based on the verification result.

[0127] In some embodiments of this application, the local storage includes multiple adapter storage locations, including local storage, main memory, and graphics processor memory. The graphics processor memory stores a set of low-rank adapters used for large models, and the set of low-rank adapters includes multiple candidate low-rank adapters. The method further includes: determining that the access time of each candidate low-rank adapter is within a first time range or a second time range to obtain a target time range, wherein the first time range is further away from the current time than the second time range; calculating the scheduling priority of each candidate low-rank adapter according to a first and a second round-off weight corresponding to the target time range; and if an adapter needs to be swapped out, swapping each candidate low-rank adapter out of the graphics processor memory according to its scheduling priority.

[0128] In some embodiments of this application, calculating the scheduling priority of each candidate low-rank adapter based on the first selection weight corresponding to the first adapter set and the second selection weight corresponding to the second adapter set includes: calculating the first priority of each candidate low-rank adapter based on the interval duration corresponding to each candidate low-rank adapter, wherein the interval duration is the time elapsed between the current inference time and the last access time of the candidate low-rank adapter; calculating the second priority of each candidate low-rank adapter based on the access count corresponding to each candidate low-rank adapter, wherein the access count is the number of times the candidate low-rank adapter was accessed before the current inference time; and calculating the scheduling priority corresponding to each candidate low-rank adapter based on the first priority, the first selection weight, the second priority, and the second selection weight corresponding to each candidate low-rank adapter.

[0129] In some embodiments of this application, the step of calculating the first priority of each candidate low-rank adapter based on the interval duration corresponding to each candidate low-rank adapter includes: calculating the first priority of each candidate low-rank adapter according to the formula Priority1 = e -ax The first priority of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1).

[0130] In some embodiments of this application, the step of calculating the second priority of each candidate low-rank adapter based on the number of accesses corresponding to each candidate low-rank adapter includes: calculating the second priority of each candidate low-rank adapter according to the formula Priority2 = 1 - e -by The second priority of the candidate low-rank adapter is calculated, where b is the second coefficient, b>0, y is the access count corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1).

[0131] In some embodiments of this application, the step of calculating the scheduling priority corresponding to each candidate low-rank adapter based on the first priority, the first selection weight, the second priority, and the second selection weight corresponding to each candidate low-rank adapter includes: calculating the scheduling priority Priority3 corresponding to the candidate low-rank adapter according to the formula Priority3=μ[α(Priority1)+β(Priority2)], where α and β are the first selection weight and the second selection weight, respectively, α,β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

[0132] In some embodiments of this application, the graphics processor memory is divided into multiple storage spaces, each used to store different parameters; the multiple storage spaces include an adapter storage space, which stores multiple candidate low-rank adapters in pages; the step of swapping out each candidate low-rank adapter from the graphics processor memory according to its scheduling priority includes: determining the candidate low-rank adapter to be swapped out according to its scheduling priority; obtaining the address set of the candidate low-rank adapter in the adapter storage space according to the low-rank adapter control block corresponding to the candidate low-rank adapter to be swapped out, and obtaining a target address set; determining the page of the candidate low-rank adapter to be swapped out in the adapter storage space according to the target address set, and obtaining a target page; and swapping out the candidate low-rank adapter to be swapped out from the target page in the adapter storage space.

[0133] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0134] Therefore, embodiments of this application also provide a storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the methods provided in embodiments of this application.

[0135] The storage medium can be a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0136] Since the computer program stored in the storage medium can execute the steps of any of the methods provided in the embodiments of this application, the beneficial effects that the methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0137] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0138] It should be understood that this application is not limited to the embodiments described above and shown in the accompanying drawings, but various modifications and changes can be made without departing from its scope.

Claims

1. A method for hot updating a low-rank adapter, wherein, include: Download the new version of the low-rank adapter corresponding to the old version of the low-rank adapter from the adapter management terminal; The new version of the low-rank adapter is stored in the adapter repository in local storage; Copy the low-rank adapter control block that inherits from the old version of the low-rank adapter; The state transition process is performed according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

2. The method according to claim 1, wherein, The step of performing state transition processing according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository includes: Remove the older low-rank adapter from multiple adapter storage locations, including local storage, main memory, and graphics processor memory; Based on the missing low-rank adapter interrupt / missing low-rank adapter exception of the old version low-rank adapter and the low-rank adapter control block, the new version low-rank adapter is sequentially queried and scheduled from the graphics processor memory, the main memory to the local storage to the graphics processor memory.

3. The method according to claim 1, wherein, The step of performing state transition processing according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository includes: Based on the state in the low-rank adapter control block, the location corresponding to the state is determined from the local storage, main memory, and graphics processor memory to obtain the target location; Remove the old version low-rank adapter from the target location; Update the new version of the low-rank adapter in the adapter repository to the target location.

4. The method according to claim 1, wherein, The step of performing state transition processing according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository includes: Based on the state in the low-rank adapter control block, the location corresponding to the state is determined from the local storage, main memory, and graphics processor memory to obtain the target location; Change the information in the low-rank adapter control block corresponding to the old version low-rank adapter at the target location to the information corresponding to the new version low-rank adapter; Remove the old version low-rank adapter from the target location.

5. The method according to claim 1, wherein, Prior to copying the low-rank adapter control block inherited from the older low-rank adapter, the method further includes: The new version of the low-rank adapter was verified, and the verification results were obtained. Based on the verification results, determine whether to copy and inherit the low-rank adapter control block of the old version low-rank adapter.

6. The method according to claim 1, wherein, The local storage includes multiple adapter storage locations, including the local storage, main memory, and graphics processor memory. The graphics processor memory stores a set of low-rank adapters for use with large models, and the set of low-rank adapters includes multiple candidate low-rank adapters. The method further includes: The access time of each candidate low-rank adapter is determined to be within a first time range or a second time range to obtain a target time range, wherein the first time range is further away from the current time than the second time range; The scheduling priority of each candidate low-rank adapter is calculated based on the first and second selection weights corresponding to the target time range. If an adapter needs to be swapped out, each of the candidate low-rank adapters is swapped out of the graphics processor memory according to its scheduling priority.

7. The method according to claim 6, wherein, The step of calculating the scheduling priority of each candidate low-rank adapter based on the first and second round-off weights corresponding to the target time range includes: The first priority of each candidate low-rank adapter is calculated based on the interval duration corresponding to each candidate low-rank adapter, wherein the interval duration is the time elapsed between the current inference time and the last access time of the candidate low-rank adapter; The second priority of each candidate low-rank adapter is calculated based on the number of accesses corresponding to each candidate low-rank adapter, wherein the number of accesses is the number of times the candidate low-rank adapter was accessed before this round of inference; The scheduling priority of each candidate low-rank adapter is calculated based on the first priority, the first rejection weight, the second priority, and the second rejection weight.

8. The method according to claim 7, wherein, The step of calculating the first priority of each candidate low-rank adapter based on the interval duration corresponding to each candidate low-rank adapter includes: According to the formula Priority1 = e -ax The first priority of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1).

9. The method according to claim 7, wherein, The step of calculating the second priority of each candidate low-rank adapter based on the number of accesses corresponding to each candidate low-rank adapter includes: According to the formula Priority2 = 1 - e -by The second priority of the candidate low-rank adapter is calculated, where b is the second coefficient, b>0, y is the access count corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1).

10. The method according to claim 7, wherein, The step of calculating the scheduling priority corresponding to each candidate low-rank adapter based on the first priority, the first discard weight, the second priority, and the second discard weight corresponding to each candidate low-rank adapter includes: The scheduling priority 3 corresponding to the candidate low-rank adapter is calculated according to the formula Priority3=μ[α(Priority1)+β(Priority2)], where α and β are the first and second pick-up weights, respectively, α,β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

11. The method according to claim 6, wherein, The graphics processor memory is divided into multiple storage spaces, each used to store different parameters; the multiple storage spaces include an adapter storage space, in which multiple candidate low-rank adapters are stored in pages. The step of swapping out each candidate low-rank adapter from the graphics processor memory according to the scheduling priority of each candidate low-rank adapter includes: The candidate low-rank adapter to be replaced is determined based on the scheduling priority of each candidate low-rank adapter. Based on the low-rank adapter control block corresponding to the candidate low-rank adapter to be replaced, the address set of the candidate low-rank adapter to be replaced in the adapter storage space is obtained, and the target address set is obtained. Based on the target address set, determine the page of the candidate low-rank adapter to be swapped out in the adapter storage space to obtain the target page; The candidate low-rank adapter to be swapped out is swapped out from the target page in the adapter storage space.

12. The method according to claim 11, wherein, The multiple storage spaces also include model weight storage space, KV cache storage space, and other parameter storage space.

13. The method according to claim 5, wherein, The step of determining whether to copy and inherit the low-rank adapter control block of the old version low-rank adapter based on the verification result includes: If the verification result is passed, then it is determined to copy the low-rank adapter control block that inherits the old version of the low-rank adapter; If the verification result is unsuccessful, then it is determined that the low-rank adapter control block inherited from the old version low-rank adapter will not be copied.

14. The method according to claim 4, wherein, The step of changing the information in the low-rank adapter control block corresponding to the old version low-rank adapter at the target location to the information corresponding to the new version low-rank adapter includes: Modify the pointer representing the real address of the old version low-rank adapter in the low-rank adapter control block corresponding to the old version low-rank adapter in the target location to point to the address of the new version low-rank adapter.

15. A low-rank adapter hot-renewal device, wherein, include: The download module is used to download the new version of the low-rank adapter corresponding to the old version of the low-rank adapter from the adapter management terminal; A storage module is used to store the new version of the low-rank adapter in the adapter repository in local storage; The copy module is used to copy the low-rank adapter control block that inherits from the old version of the low-rank adapter; The update module is used to perform state transition processing according to the low-rank adapter control block, so as to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

16. The apparatus according to claim 15, wherein, The update module is configured to: remove the old version low-rank adapter from multiple adapter storage locations, including the local storage, main memory, and graphics processor memory; Based on the missing low-rank adapter interrupt / missing low-rank adapter exception of the old version low-rank adapter and the low-rank adapter control block, the new version low-rank adapter is sequentially queried and scheduled from the graphics processor memory, the main memory to the local storage to the graphics processor memory.

17. The apparatus according to claim 15, wherein, The update module is configured to: determine the location corresponding to the state from the local storage, main memory, and graphics processor memory based on the state in the low-rank adapter control block, and obtain the target location; and remove the old version of the low-rank adapter from the target location. Update the new version of the low-rank adapter in the adapter repository to the target location.

18. The apparatus according to claim 15, wherein, The update module is configured to: determine the position corresponding to the state from the local storage, main memory, and graphics processor memory based on the state in the low-rank adapter control block, and obtain the target position; Change the information in the low-rank adapter control block corresponding to the old version low-rank adapter at the target location to the information corresponding to the new version low-rank adapter; Remove the old version low-rank adapter from the target location.

19. A storage medium, wherein, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the method described in any one of claims 1 to 14.

20. An electronic device, wherein, include: Memory, which stores computer programs; A processor reads a computer program stored in memory to execute the method according to any one of claims 1 to 14.

Images