Low-rank adapter hot update method and device, storage medium and electronic equipment

Abstract

The invention discloses a low-rank adapter hot update method and device, a storage medium and electronic equipment, and relates to the technical field of artificial intelligence, and the method comprises the steps: downloading a new-version low-rank adapter corresponding to an old-version low-rank adapter from an adapter management end; storing the new version of low-rank adapter in an adapter warehouse in a local storage; replicating a low rank adapter control block that inherits the old version of low rank adapter; and performing state transition processing according to the low-rank adapter control block so as to perform hot update on the old-version low-rank adapter by using the new-version low-rank adapter in the adapter warehouse. According to the method, the low-rank adapter can be updated without stopping the online service of the large model, the large model does not need to be published again, the low-rank adapter does not need to be published again, rapid hot updating of the low-rank adapter is achieved, rapid and flexible version iteration of the large model is effectively supported, and the expandability and maintainability of the large model are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a low-rank adapter hot update method, device, storage medium and electronic device. Background Art

[0002] With the development of technology, various large models have been widely used. However, large models have large parameters and the cost of model training and deployment is high. Therefore, the technology of low-rank adaptation (LoRA) has emerged. Based on low-rank adaptation, only the base large model needs to be fine-tuned for specific tasks to obtain a low-rank adapter (LoRA Adapter). After that, the low-rank adapter is combined with the base large model into a low-rank adaptation large model (LLM LoRA Adapter) to effectively handle specific tasks.

[0003] With the large-scale application and deployment of low-rank adapters, the update of these low-rank adapters often requires stopping the online service of large models or low-rank adapters (LoRA Adapters). After updating the low-rank adapters, re-publishing the low-rank adaptation to the large model will interfere with the online reasoning process of the model. The current method is not convenient for the rapid update of low-rank adapters, and cannot effectively support the rapid and flexible version iteration of large models, resulting in weak scalability and maintainability of large models. Summary of the invention

[0004] The embodiment of the present application provides a low-rank adapter hot update solution, which can effectively realize the rapid hot update of the low-rank adapter, effectively support the rapid and flexible version iteration of the large model, and improve the scalability and maintainability of the large model.

[0005] The embodiments of the present application provide the following technical solutions:

[0006] According to one embodiment of the present application, a low-rank adapter hot update method includes: downloading a new version of a low-rank adapter corresponding to an old version of a low-rank adapter from an adapter management end; saving the new version of the low-rank adapter in an adapter repository in local storage; copying and inheriting the low-rank adapter control block of the old version of the low-rank adapter; performing state migration processing according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

[0007] In some embodiments of the present application, the state migration processing is performed according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository, including: removing the old version of the low-rank adapter from multiple adapter storage locations, the multiple adapter storage locations including the local storage, main memory, and graphics processor memory; according to the lack of low-rank adapter interruption / lack of low-rank adapter exception of the old version of the low-rank adapter and the low-rank adapter control block, querying and scheduling the new version of the low-rank adapter to the graphics processor memory step by step from the graphics processor memory, the main memory to the local storage.

[0008] In some embodiments of the present application, the state migration processing is performed according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository, including: according to the state in the low-rank adapter control block, determining the location corresponding to the state from the local storage, main memory, and graphics processor memory to obtain the target location; removing the old version of the low-rank adapter from the target location; and updating the new version of the low-rank adapter in the adapter repository to the target location.

[0009] In some embodiments of the present application, the state migration processing is performed according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository, including: according to the state in the low-rank adapter control block, determining the location corresponding to the state from the local storage, main memory, and graphics processor memory to obtain the target location; changing the information in the low-rank adapter control block corresponding to the old version of the low-rank adapter in the target location to the information corresponding to the new version of the low-rank adapter; and removing the old version of the low-rank adapter from the target location.

[0010] In some embodiments of the present application, before copying and inheriting the low-rank adapter control block of the old version of the low-rank adapter, the method also includes: verifying the new version of the low-rank adapter to obtain a verification result; and determining whether to copy and inherit the low-rank adapter control block of the old version of the low-rank adapter based on the verification result.

[0011] In some embodiments of the present application, the local includes multiple adapter storage locations, and the multiple adapter storage locations include the local storage, main memory and graphics processor memory. The graphics processor memory stores a set of low-rank adapters for use with large models, and the low-rank adapter set includes multiple candidate low-rank adapters; the method also includes: determining that the access time of each of the candidate low-rank adapters is within the first time range or the second time range, and obtaining a target time range, wherein the first time range is farther away from the current time than the second time range; calculating the scheduling priority of each of the candidate low-rank adapters according to the first cut-off weight and the second cut-off weight corresponding to the target time range; if the adapter needs to be swapped out, swapping each of the candidate low-rank adapters out of the graphics processor memory according to the scheduling priority of each of the candidate low-rank adapters.

[0012] In some embodiments of the present application, the scheduling priority of each candidate low-rank adapter is calculated based on the first cut-off weight corresponding to the first adapter set and the second cut-off weight corresponding to the second adapter set, including: calculating the first priority of each candidate low-rank adapter based on the interval duration corresponding to each candidate low-rank adapter, wherein the interval duration is the time elapsed between the time of this round of reasoning and the time of the last visit to the candidate low-rank adapter; calculating the second priority of each candidate low-rank adapter based on the number of visits corresponding to each candidate low-rank adapter, wherein the number of visits is the number of times the candidate low-rank adapter is visited before this round of reasoning; calculating the scheduling priority corresponding to each candidate low-rank adapter based on the first priority, the first cut-off weight, the second priority and the second cut-off weight corresponding to each candidate low-rank adapter.

[0013] In some embodiments of the present application, the first priority of each candidate low-rank adapter is calculated according to the interval duration corresponding to each candidate low-rank adapter, including: according to the formula Priority1=e -ax The first priority Priority1 of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1].

[0014] In some embodiments of the present application, the second priority of each candidate low-rank adapter is calculated according to the number of accesses corresponding to each candidate low-rank adapter, including: according to the formula Priority2=1-e -by The second priority Priority2 of the candidate low-rank adapter is calculated, where b is the second coefficient, b>0, y is the number of visits corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1].

[0015] In some embodiments of the present application, the scheduling priority corresponding to each candidate low-rank adapter is calculated according to the first priority, the first trade-off weight, the second priority and the second trade-off weight corresponding to each candidate low-rank adapter, including: calculating the scheduling priority Priority3 corresponding to the candidate low-rank adapter according to the formula Priority3=μ[α(Priority1)+β(Priority2)], wherein α, β are the first trade-off weight and the second trade-off weight respectively, α, β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

[0016] In some embodiments of the present application, the graphics processor memory is divided into multiple storage spaces, and the multiple storage spaces are used to store different parameters respectively; the multiple storage spaces include an adapter storage space, and the multiple candidate low-rank adapters are stored in pages in the adapter storage space; swapping out each candidate low-rank adapter from the graphics processor memory according to the scheduling priority of each candidate low-rank adapter includes: determining the candidate low-rank adapter to be swapped out according to the scheduling priority of each candidate low-rank adapter; obtaining an address set of the candidate low-rank adapter to be swapped out in the adapter storage space according to the low-rank adapter control block corresponding to the candidate low-rank adapter to be swapped out, and obtaining a target address set; determining the paging of the candidate low-rank adapter to be swapped out in the adapter storage space according to the target address set, and obtaining a target page; swapping out the candidate low-rank adapter to be swapped out from the target page in the adapter storage space.

[0017] According to one embodiment of the present application, a low-rank adapter hot update device includes: a download module for downloading a new version of a low-rank adapter corresponding to an old version of a low-rank adapter from an adapter management end; a save module for saving the new version of the low-rank adapter in an adapter repository in local storage; a copy module for copying a low-rank adapter control block that inherits the old version of the low-rank adapter; and an update module for performing state migration processing according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

[0018] In some embodiments of the present application, the update module is used to: remove the old version of the low-rank adapter from multiple adapter storage locations, the multiple adapter storage locations including the local storage, the main memory, and the graphics processor memory; according to the lack of low-rank adapter interrupt / lack of low-rank adapter exception and the low-rank adapter control block of the old version of the low-rank adapter, query and schedule the new version of the low-rank adapter to the graphics processor memory step by step from the graphics processor memory, the main memory to the local storage.

[0019] In some embodiments of the present application, the update module is used to: determine the location corresponding to the state from the local storage, main memory, and graphics processor memory according to the state in the low-rank adapter control block to obtain the target location; remove the old version of the low-rank adapter from the target location; and update the new version of the low-rank adapter in the adapter repository to the target location.

[0020] In some embodiments of the present application, the update module is used to: determine the location corresponding to the state from the local storage, main memory, and graphics processor memory according to the state in the low-rank adapter control block to obtain the target location; change the information in the low-rank adapter control block corresponding to the old version of the low-rank adapter in the target location to the information corresponding to the new version of the low-rank adapter; and remove the old version of the low-rank adapter from the target location.

[0021] In some embodiments of the present application, before copying and inheriting the low-rank adapter control block of the old version of the low-rank adapter, the device also includes a verification module for: verifying the new version of the low-rank adapter to obtain a verification result; and determining whether to copy and inherit the low-rank adapter control block of the old version of the low-rank adapter based on the verification result.

[0022] In some embodiments of the present application, the local includes multiple adapter storage locations, and the multiple adapter storage locations include the local storage, main memory and graphics processor memory. The graphics processor memory stores a set of low-rank adapters for use with large models, and the low-rank adapter set includes multiple candidate low-rank adapters; the device also includes a scheduling module, which is used to: determine whether the access time of each of the candidate low-rank adapters is within the first time range or the second time range, and obtain a target time range, wherein the first time range is farther away from the current time than the second time range; calculate the scheduling priority of each of the candidate low-rank adapters according to the first cut-off weight and the second cut-off weight corresponding to the target time range; if the adapter needs to be swapped out, swap each of the candidate low-rank adapters out of the graphics processor memory according to the scheduling priority of each of the candidate low-rank adapters.

[0023] In some embodiments of the present application, the scheduling module is used to: calculate the first priority of each candidate low-rank adapter according to the interval duration corresponding to each candidate low-rank adapter, wherein the interval duration is the time elapsed between the time of this round of reasoning and the time of the last visit to the candidate low-rank adapter; calculate the second priority of each candidate low-rank adapter according to the number of visits corresponding to each candidate low-rank adapter, wherein the number of visits is the number of times the candidate low-rank adapter is visited before this round of reasoning; calculate the scheduling priority corresponding to each candidate low-rank adapter according to the first priority, the first cut-off weight, the second priority and the second cut-off weight corresponding to each candidate low-rank adapter.

[0024] In some embodiments of the present application, the scheduling module is used to: according to the formula Priority1=e -ax The first priority Priority1 of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1].

[0025] In some embodiments of the present application, the scheduling module is used to: according to the formula Priority2=1-e -by The second priority Priority2 of the candidate low-rank adapter is calculated, where b is the second coefficient, b>0, y is the number of visits corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1].

[0026] In some embodiments of the present application, the scheduling module is used to: calculate the scheduling priority Priority3 corresponding to the candidate low-rank adapter according to the formula Priority3=μ[α(Priority1)+β(Priority2)], wherein α and β are the first trade-off weight and the second trade-off weight, respectively, α, β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

[0027] In some embodiments of the present application, the graphics processor memory is divided into multiple storage spaces, and the multiple storage spaces are used to store different parameters respectively; the multiple storage spaces include an adapter storage space, and the multiple candidate low-rank adapters are stored in pages in the adapter storage space; the scheduling module is used to: determine the candidate low-rank adapter to be swapped out according to the scheduling priority of each of the candidate low-rank adapters; obtain the address set of the candidate low-rank adapter to be swapped out in the adapter storage space according to the low-rank adapter control block corresponding to the candidate low-rank adapter to be swapped out, and obtain a target address set; determine the paging of the candidate low-rank adapter to be swapped out in the adapter storage space according to the target address set, and obtain a target page; swap out the candidate low-rank adapter to be swapped out from the target page in the adapter storage space.

[0028] According to another embodiment of the present application, a storage medium stores a computer program thereon, and when the computer program is executed by a processor of a computer, the computer executes the method described in the embodiment of the present application.

[0029] According to another embodiment of the present application, an electronic device may include: a memory storing a computer program; and a processor reading the computer program stored in the memory to execute the method described in the embodiment of the present application.

[0030] According to another embodiment of the present application, a computer program product or a computer program includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0031] In an embodiment of the present application, a new version of a low-rank adapter corresponding to an old version of a low-rank adapter is downloaded from an adapter management end; the new version of the low-rank adapter is saved in an adapter repository in local storage; the low-rank adapter control block that inherits the old version of the low-rank adapter is copied; and state migration processing is performed according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

[0032] In this way, after downloading the new version of the low-rank adapter corresponding to the old version of the low-rank adapter, the new version of the low-rank adapter is saved in the adapter warehouse in the local storage, and then the low-rank adapter control block of the old version of the low-rank adapter is copied. According to the copied low-rank adapter control block, the new version of the low-rank adapter in the adapter warehouse is used to hot-update the old version of the low-rank adapter through state migration. The low-rank adapter can be updated without stopping the online service of the large model, without re-releasing the large model, and without re-releasing the low-rank adapter, thereby realizing rapid hot update of the low-rank adapter, effectively supporting the fast and flexible version iteration of the large model, and improving the scalability and maintainability of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 A flowchart of a low-rank adapter hot update method according to an embodiment of the present application is shown.

[0035] Figure 2 A low-rank adapter hot update flow chart according to another embodiment of the present application is shown.

[0036] Figure 3 A schematic diagram of low-rank adapter management according to an embodiment of the present application is shown.

[0037] Figure 4 A low-rank adapter state migration flowchart according to an embodiment of the present application is shown.

[0038] Figure 5 A schematic diagram of low-rank adapter partitioning according to an embodiment of the present application is shown.

[0039] Figure 6 A schematic diagram of low-rank adapter storage according to an embodiment of the present application is shown.

[0040] Figure 7 A block diagram of a low-rank adapter hot update device according to an embodiment of the present application is shown.

[0041] Figure 8 A block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0042] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are only used to explain the present disclosure and are not intended to limit the present disclosure. In addition, the embodiments provided below are partial embodiments for implementing the present disclosure, rather than providing all embodiments for implementing the present disclosure. In the absence of conflict, the technical solutions recorded in the embodiments of the present disclosure can be implemented in any combination.

[0043] It should be noted that, in the embodiments of the present disclosure, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or apparatus including a series of elements includes not only the elements explicitly recorded, but also includes other elements not explicitly listed, or also includes elements inherent to the implementation of the method or apparatus. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other related elements in the method or apparatus including the element (such as a step in the method or a unit in the apparatus, for example, a unit may be a part of a circuit, a part of a processor, a part of a program or software, etc.).

[0044] For example, the low-rank adapter hot update method provided in the embodiment of the present disclosure includes a series of steps, but the low-rank adapter hot update method provided in the embodiment of the present disclosure is not limited to the recorded steps. Similarly, the low-rank adapter hot update device provided in the embodiment of the present disclosure includes a series of units, but the device provided in the embodiment of the present disclosure is not limited to including the units explicitly recorded, and may also include units that need to be set to obtain relevant information or perform processing based on the information.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure.

[0046] It is understandable that in the specific implementation of this application, relevant data is involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0047] Figure 1 The flowchart of the low-rank adapter hot update method according to an embodiment of the present application is schematically shown. The execution subject of the low-rank adapter hot update method can be a device or a server.

[0048] like Figure 1 As shown, the low-rank adapter hot update method may include steps S110 to S140.

[0049] Step S110, downloading a new version of the low-rank adapter corresponding to the old version of the low-rank adapter from the adapter management end; Step S120, saving the new version of the low-rank adapter in the adapter repository in the local storage; Step S130, copying and inheriting the low-rank adapter control block of the old version of the low-rank adapter; Step S140, performing state migration processing according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

[0050] The adapter manager can be used to publish low-rank adapters. Figure 2 , the execution subject 210 can download the new version of the low-rank adapter corresponding to the old version of the low-rank adapter from the adapter management terminal 220, and save the new version of the low-rank adapter in the adapter warehouse in the local storage (Local Store). The local storage is other storage other than the main memory (Main Memory) of the CPU and the GPU memory (GPU Memory) of the graphics processor, and the local storage can be, for example, a local disk or other storage.

[0051] In the embodiments of this application, refer to Figure 2 All low-rank adapters are maintained in local storage (Local Store). Some high-importance low-rank adapters can be maintained in local storage, main memory (Main Memory) and GPU memory (GPU Memory) at the same time through manual configuration or manual initiation of low-rank adapter missing interruption / low-rank adapter missing exception. The low-rank adapters required for subsequent batches (Future Batch) will be scheduled from local storage to main memory (Main Memory). Furthermore, the low-rank adapters required for the current batch (Current Memory) will be scheduled from main memory to GPU memory (for example, if a low-rank adapter missing interruption / low-rank adapter missing exception occurs). When processing a specific reasoning task, the large model will use the adapter weights of the low-rank adapter corresponding to the specific reasoning task in the GPU memory and other basic model weights for reasoning, that is, the low-rank adapter is combined with the large model for reasoning.

[0052] For each low-rank adapter, the status and attributes of the low-rank adapter are recorded through the corresponding low-rank adapter control block (LACB, LoRA AdapterControl Block), that is, each low-rank adapter has a low-rank adapter control block that inherits its status and attributes. By copying the low-rank adapter control block that inherits the old version of the low-rank adapter, the status and attributes of the old version of the low-rank adapter can be obtained through the copied low-rank adapter control block.

[0053] In one example, the state of the low-rank adapter may include: Created (creating a low-rank adapter control block (LACB, LoRA Adapter Control Block) and entering the scheduling process), Updating (the low-rank adapter is in the process of upgrading, and the upgrade process can be restricted from being interrupted in this state), Upgrade completed (Updated, the low-rank adapter upgrade is completed and can be transferred to the Created state, and the subsequent scheduling process can be executed), Candidate (Waiting, the low-rank adapter is in the adapter warehouse and can be scheduled to the main memory at any time), Ready (the low-rank adapter is in the main memory and is ready to be scheduled to the GPU memory at any time), Running (the low-rank adapter is in the GPU memory and can be combined with the large model (LLM Base Model) for reasoning at any time). It can be understood that in other examples, some states can also be added according to business conditions for refined management.

[0054] Multiple low-rank adapters in the same state are divided into corresponding state pools, see Figure 3 , multiple low-rank adapters in the running state in the GPU memory are divided into a running pool (Running Pool) 300. Similarly, multiple low-rank adapters in the waiting state in the local storage can be divided into a candidate pool (WaitingPool), and multiple low-rank adapters in the ready state in the main memory can be divided into a ready pool (ReadyPool). Each low-rank adapter 310 has a corresponding low-rank adapter control block (LACB) 310 that inherits its status and attributes. For example, see Figure 3 The low-rank adapter control block (LACB, LoRA Adapter Control Block) may record, including but not limited to, the identification (ID), version (version), page table information (Page Table), state (state) and timestamp (timestamp) of the low-rank adapter. The low-rank adapter control block of the old version of the low-rank adapter may be copied and inherited (Fork / clone) through the Fork method, so that the old version of the low-rank adapter in different states can obtain its status and attributes and other information through the copied low-rank adapter control block.

[0055] State migration processing is performed based on the status and attributes in the low-rank adapter control block. The old version of the low-rank adapter is hot-updated using the new version of the low-rank adapter in the adapter repository. The low-rank adapter can be hot-updated based on the status information of the low-rank adapter control block. The low-rank adapter can be updated without stopping the large model online inference service.

[0056] Among them, the low-rank adapter (LoRA Adapter) + base large model (LLM Base Model) can usually provide two specific reasoning service modes: (1) merge the low-rank adapter into the base large model, and then use the merged large model for reasoning; (2) the base large model + low-rank adapter directly performs reasoning without the need for a merging action. In the embodiment of the present application, there is no need to republish the large model or the low-rank adapter. The new version of the low-rank adapter can be seamlessly replaced in the current batch of large model reasoning process, and the user perception is extremely low.

[0057] In this way, after downloading the new version of the low-rank adapter corresponding to the old version of the low-rank adapter, the new version of the low-rank adapter is saved in the adapter warehouse in the local storage, and then the low-rank adapter control block of the old version of the low-rank adapter is copied. According to the copied low-rank adapter control block, the new version of the low-rank adapter in the adapter warehouse is used to hot-update the old version of the low-rank adapter through state migration. The low-rank adapter can be updated without stopping the online service of the large model, without re-releasing the large model, and without re-releasing the low-rank adapter, thereby realizing rapid hot update of the low-rank adapter, effectively supporting the rapid and flexible version iteration of the large model, and improving the scalability and maintainability of the large model.

[0058] Described below Figure 1 When performing a low-rank adapter hot update under the embodiment, further optional specific embodiments are provided for each step performed.

[0059] In one embodiment, the state migration processing performed according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository may include: removing the old version of the low-rank adapter from multiple adapter storage locations, the multiple adapter storage locations including the local storage, main memory, and graphics processor memory; according to the lack of low-rank adapter interruption / lack of low-rank adapter exception of the old version of the low-rank adapter and the low-rank adapter control block, querying and scheduling the new version of the low-rank adapter to the graphics processor memory step by step from the graphics processor memory, the main memory to the local storage.

[0060] In this embodiment, when performing state migration, first, remove the old version of the low-rank adapter from the local storage, main memory, and graphics processor memory. Then, according to the low-rank adapter interruption / low-rank adapter exception (Adapter Fault) of the old version of the low-rank adapter and the low-rank adapter control block, the new version of the low-rank adapter is queried and scheduled to the graphics processor memory step by step from the graphics processor memory, main memory to the local storage, wherein the low-rank adapter interruption / low-rank adapter exception is specifically a message of the lack of LoRA Adapter exception / LoRA Adapter interruption.

[0061] For example, see Figure 4 After removing the old version of LoRAAdapter7 from local storage, main memory and GPU memory, S7 (A7) indicates that when reasoning task 7 (Sequence7) is to use LoRA Adapter7 to start reasoning, step S410 is used to query the low-rank adapter from the running pool (Fetch Adapter from Running Pool). If it is found that LoRA Adapter7 is not in the running pool (Running Pool) in the GPU memory (GPU Memory), a low-rank adapter interruption / low-rank adapter exception 1 (Adapter Fault1) is initiated, and then step S420 is executed to query the low-rank adapter from the ready pool (Fetch Adapter from Ready Pool); on the contrary, if LoRA Adapter7 is found in the running pool (Running Pool) 1 in the GPU memory (GPU Memory), step S430 is executed to run iterative-level inference (running Iteration-Level inference), so that the large model uses the LoRA Adapter7 in the Running Pool in the GPU Memory (GPU Memory) for reasoning of the current batch.

[0062] When step S420 performs the query, if it is found that LoRA Adapter7 is not in the Ready Pool in the main memory (Main Memory), a low-rank adapter interruption / low-rank adapter exception 2 (Adapter Fault2) is initiated, and then step S440 is executed to query the low-rank adapter from the candidate pool (Fetch Adapter from WaitingPool). If step S420 finds that LoRA Adapter7 is in the Ready Pool in the main memory (Main Memory), step S450 is executed to load the low-rank adapter from the Ready Pool to the Running Pool, and the LoRA Adapter7 Ready Pool (ReadyPool) is scheduled to the Running Pool (Running Pool).

[0063] After executing step S440 for query, if it is found that LoRA Adapter7 is in the candidate pool (Waiting Pool) in the local storage, then execute step S460 to load the low-rank adapter from the candidate pool to the preparation pool, and schedule LoRAAdapter7 from the candidate pool (Waiting Pool) to the preparation pool (Ready Pool) 1, and further execute step S450 to schedule LoRA Adapter7 from the preparation pool (Ready Pool) to the running pool (Running Pool) 1. Thus, the new version of LoRA Adapter7 can be scheduled to the running pool (Running Pool) step by step.

[0064] In addition, in the process of scheduling the new version of the low-rank adapter step by step, the low-rank adapter control block that inherits the new version of the low-rank adapter can be updated in the running pool (Running Pool), ready pool (ReadyPool) and candidate pool (Waiting Pool) in the graphics processor memory, main memory and local storage according to the copied low-rank adapter control block.

[0065] It should be noted that the reasoning service of the large model (LLM) usually uses continuous batching technology for request batching. Continuous batching is a technology that allows one or more reasoning requests to be combined into a single batch and dynamically adjusts the batch size as needed to maximize throughput.

[0066] Specifically, Figure 4As shown, for continuous batching requests, continuous batching can be performed as follows Figure 4 The large rectangular boxes corresponding to T1 to T7 in the diagram are organized at the sequence level and processed in units of sequences. There are a total of 7 sequences in the large rectangular box, where Si represents the i-th sequence, for example, S1 represents sequence 1. In addition, the batch size is 4, for example, each column corresponding to T1 to T7 in the large rectangular box includes 4 small rectangular boxes, indicating that the batch size of a batch is 4.

[0067] Furthermore, during the reasoning process, the large model uses iteration-level to represent the working status of each token, such as Figure 4 Each small rectangular box in the large rectangular box is a token. Among them, [rectangular box with pure white background] represents prompt information tokens, that is, the prompt information (Prompt) input to the large model in the user's request is decomposed into smaller units of tokens. [Rectangular box with diagonal background] represents output text tokens (Generated Tokens), that is, the output text generated by the large model according to the input prompt information tokens is decomposed into tokens. [Rectangular box with horizontal background] represents the end of sequence token, that is, the token indicating the end of a sequence. [Rectangular box with horizontal and vertical lines] represents a low-rank adapter interruption / low-rank adapter exception (Adapter Fault).

[0068] For example, Figure 4In the figure, S1(A1) [rectangular box with pure white background] indicates the prompt information token (Sequence 1(Adapter 1) Prompt Tokens) in sequence 1 in which the large model uses the low-rank adapter 1 for reasoning; S1(A1) [rectangular box with diagonal background] indicates the output text token (Sequence 1(Adapter 1) Generated Tokens) in the sequence in which the large model uses the low-rank adapter 1 for reasoning; END[rectangular box with horizontal background] after S1(A1) indicates the end of sequence token; S7(A7)[rectangular box with horizontal and vertical background] indicates that when the large model is reasoning about sequence 7 (Sequence 7), and is about to use the low-rank adapter 7 (Adapter 7) to start iteration-level reasoning, after the query in step S410, it is found that the low-rank adapter 7 (Adapter 7) is not in the running pool (RunningPool) of GPU Memory, so a low-rank adapter missing interrupt / low-rank adapter missing exception (Adapter Fault) is initiated. After a missing low-rank adapter interruption / missing low-rank adapter exception (Adapter Fault) is initiated, the scheduling process described in the above embodiment is triggered.

[0069] In one embodiment, the state migration processing is performed according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository, and may also include:

[0070] According to the state in the low-rank adapter control block, determine the location corresponding to the state from the local storage, main memory and graphics processor memory to obtain the target location; remove the old version of the low-rank adapter from the target location; and update the new version of the low-rank adapter in the adapter warehouse to the target location.

[0071] In this embodiment, according to the state in the low-rank adapter control block, the position corresponding to the state is determined from the local storage, main memory and graphics processor memory to obtain the target position. For example, the state in the copied low-rank adapter control block is candidate (Waiting), and the position corresponding to candidate (Waiting) is the candidate pool (Waiting Pool) in the local storage, so that the target position is the candidate pool (Waiting Pool) in the local storage.

[0072] Then, the old version of the low-rank adapter is removed from the target location, and the new version of the low-rank adapter in the adapter warehouse is updated to the target location. As a result, the adapter hot update can be performed with lower user perception. Only when the target location includes the running pool (Running Pool) in the GPU memory, it will be briefly perceived by the user in the manner of this embodiment.

[0073] Further, in one embodiment, the state migration processing according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository may include:

[0074] According to the state in the low-rank adapter control block, determine the location corresponding to the state from the local storage, main memory and graphics processor memory to obtain the target location; change the information in the low-rank adapter control block corresponding to the old version of the low-rank adapter in the target location to the information corresponding to the new version of the low-rank adapter; remove the old version of the low-rank adapter from the target location.

[0075] In this embodiment, according to the state in the low-rank adapter control block, the position corresponding to the state is determined from the local storage, main memory and graphics processor memory. After obtaining the target position, the information in the low-rank adapter control block corresponding to the old version of the low-rank adapter in the target position is changed to the information corresponding to the new version of the low-rank adapter, and then the old version of the low-rank adapter is removed from the target position. Specifically, the pointer representing the real address of the old version of the low-rank adapter in the low-rank adapter control block corresponding to the old version of the low-rank adapter is modified to point to the address of the new version of the low-rank adapter. When the target position is the running pool in the GPU memory, the old version of the low-rank adapter can be retained to ensure that the reasoning process of the large model is not interrupted by the upgrade process; wherein, when the new version of the low-rank adapter is created, the corresponding low-rank adapter control block will be created in the corresponding storage space (such as main memory, graphics processor memory, etc.), and the page table information (Page Table) in the low-rank adapter control block will be updated. The page table information is the address set of the storage space (such as main memory, graphics processor memory, etc.) of the new version of the low-rank adapter, and the address set in the page table information points to the specific address of the new version of the low-rank adapter in the storage space. In this way, the user perception of low-rank adapter hot updates can be further reduced.

[0076] Furthermore, in one embodiment, before copying and inheriting the low-rank adapter control block of the old version of the low-rank adapter, the method also includes: verifying the new version of the low-rank adapter to obtain a verification result; and determining whether to copy and inherit the low-rank adapter control block of the old version of the low-rank adapter based on the verification result.

[0077] After downloading the new version of the low-rank adapter, verify the new version of the low-rank adapter to obtain the verification result. If the verification result is passed, it can be determined to copy and inherit the low-rank adapter control block of the old version of the low-rank adapter. On the contrary, if the verification result is not passed, it can be determined not to copy and inherit the low-rank adapter control block of the old version of the low-rank adapter. In this way, the reliability of hot updates of the adapter can be ensured. Among them, the methods for verifying the new version of the low-rank adapter include but are not limited to MD5 verification, compatibility verification with large models, and correctness verification of inference results.

[0078] Further, in one embodiment, the local includes multiple adapter storage locations, the multiple adapter storage locations include the local storage, the main memory and the graphics processor memory, the graphics processor memory stores a low-rank adapter set for use with the large model, and the low-rank adapter set includes multiple candidate low-rank adapters;

[0079] The method may also include: determining whether the access time of each of the candidate low-rank adapters is within the first time range or the second time range to obtain a target time range, wherein the first time range is farther away from the current time than the second time range; calculating the scheduling priority of each of the candidate low-rank adapters according to the first cut-off weight and the second cut-off weight corresponding to the target time range; if the adapter needs to be swapped out, swapping each of the candidate low-rank adapters out of the graphics processor memory according to the scheduling priority of each of the candidate low-rank adapters.

[0080] When a low-rank adapter needs to be scheduled into the GPU memory, if the GPU memory is tight, the low-rank adapter already in the GPU memory needs to be swapped out from the GPU memory. For example, when a low-rank adapter interruption / low-rank adapter exception (Adapter Fault) occurs, the corresponding low-rank adapter needs to be scheduled from the main memory to the GPU memory. At this time, if there is no free space in the GPU memory, some low-rank adapters can be swapped out from the GPU memory to the main memory according to the scheduling priority of each candidate low-rank adapter, so as to make room to swap in the low-rank adapter required for the current inference request into the GPU memory. The GPU memory required by the low-rank adapter needs to be locked in advance. If the low-rank adapter is swapped out frequently, it consumes a lot of resources. Therefore, when it is necessary to swap out the low-rank adapter already in the GPU memory from the GPU memory, it becomes extremely important to select which low-rank adapter to swap out.

[0081] In order to increase the utilization of the graphics processing unit (GPU) and improve the inference efficiency during the inference process, large models will use the aforementioned batch mechanism as much as possible. The iteration-level scheduling policy of the inference services of mainstream large models (Text-Generation-Inference, vLLM, TensorRT-LLM, etc.) usually supports continuous batching technology. This technology does not disrupt the order of request processing, which means that the requests in the same batch are only temporally related, not logically related. The set of low-rank adapters used by the set of requests belonging to the same batch must be loaded into the GPU Memory at the same time during the generation of tokens for this round of iteration-level reasoning.

[0082] The technical characteristics of continuous batching make it possible for the low-rank adapter set just accessed by this batch of temporally associated request sets to be used in the next token generation iteration "without encountering the sequence end marker or the maximum generation length limit". Therefore, the low-rank adapter set just accessed must be retained for a short period of time (GPU Memory is not allowed to be swapped out) in order to successfully complete the request reasoning process of this batch. That is, this scenario expects the access time of the low-rank adapter to be swapped out in the near future to be as far away from the current time as possible (approximately LRU algorithm, time factor priority), and the low-rank adapter to be swapped out in the long term is not expected to be accessed during the next batch (batch reasoning process) to reduce the overhead caused by temporarily loading the low-rank adapter (for example, the LFU algorithm can be used to make approximate predictions, with frequency factors taking priority and frequency decaying over time).

[0083] In this embodiment of the present application, the access time is divided into the first time range or the second time range (for example, Figure 5As shown, the time before the current time is divided from A into a first time range LRU with Decay and a second time range LFU with Decay); the scheduling priority of each candidate low-rank adapter is calculated according to the first cut-off weight and the second cut-off weight corresponding to the target time range in which the access time of each candidate low-rank adapter is located; if the adapter needs to be swapped out, each candidate low-rank adapter is swapped out from the graphics processor memory according to the scheduling priority of each candidate low-rank adapter. For example, one or more candidate low-rank adapters with the lowest scheduling priority can be swapped out from the graphics processor memory, which can reliably solve the problems existing in the above-mentioned technical characteristics of continuous batching and can further improve the reliability of adapter hot updates.

[0084] In one embodiment, calculating the scheduling priority of each of the candidate low-rank adapters according to the first cut-off weight and the second cut-off weight corresponding to the target time range may include:

[0085] According to the interval duration corresponding to each candidate low-rank adapter, the first priority of each candidate low-rank adapter is calculated, wherein the interval duration is the time elapsed between the time of this round of reasoning and the time of the last visit to the candidate low-rank adapter; according to the number of visits corresponding to each candidate low-rank adapter, the second priority of each candidate low-rank adapter is calculated, wherein the number of visits is the number of times the candidate low-rank adapter is visited before this round of reasoning; according to the first priority, the first cut-off weight, the second priority and the second cut-off weight corresponding to each candidate low-rank adapter, the scheduling priority corresponding to each candidate low-rank adapter is calculated.

[0086] Specifically, in one embodiment, the first priority of each candidate low-rank adapter is calculated according to the interval duration corresponding to each candidate low-rank adapter, which may include: according to the formula Priority1=e -ax The first priority Priority1 of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1].

[0087] The first priority Priority1 of the candidate low-rank adapter is calculated according to this formula. x represents the time between the current time (Current Time) of this round (Iteration-Level) reasoning and the last access time (AccessTime) of the Adapter. a is used as the weight to control the speed of decrease over time. The larger a is, the faster the decrease speed is.

[0088] In one embodiment, the second priority of each candidate low-rank adapter is calculated according to the number of accesses corresponding to each candidate low-rank adapter, including: according to the formula Priority2=1-e -by The second priority Priority2 of the candidate low-rank adapter is calculated, where b is the second coefficient, b>0, y is the number of visits corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1].

[0089] According to the formula Priority2 = 1-e -by The second priority Priority2 of the candidate low-rank adapter is calculated, where y represents the number of times the candidate low-rank adapter is accessed before this round (Iteration-Level) of reasoning. b is used as the weight to control the speed of decrease with the number of visits. The larger b is, the faster the increase is.

[0090] In one method, the scheduling priority corresponding to each candidate low-rank adapter is calculated according to the first priority, the first cut-off weight, the second priority and the second cut-off weight corresponding to each candidate low-rank adapter, including: calculating the scheduling priority Priority3 corresponding to the candidate low-rank adapter according to the formula Priority3=μ[α(Priority1)+β(Priority2)], wherein α and β are the first cut-off weight and the second cut-off weight respectively, α, β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

[0091] See also Figure 5 , α, β can be the weights of LRU with Decay in the first time range and LFU with Decay in the second time range, respectively, and need to satisfy α, β∈[0,1] and α+β=1. Figure 5 The middle demarcation point A (empirical value, or given by a dynamic formula), if the candidate low-rank adapter is in the first time range LRU with Decay, then α>β, if the candidate low-rank adapter is in the second time range LFU with Decay, then α<β. μ represents the external or manual intervention weight (for example, this parameter can be used to ensure that an important candidate low-rank adapter always resides in GPU Memory for real-time reasoning without being swapped out to Main Memory). α, β can be specified by program logic based on experience, or calculated by a dynamic formula.

[0092] Furthermore, in one embodiment, the graphics processor memory is divided into multiple storage spaces, and the multiple storage spaces are respectively used to store different parameters; the multiple storage spaces include an adapter storage space, and the multiple candidate low-rank adapters are stored in pages in the adapter storage space. Furthermore, the multiple storage spaces also include a model weight storage space, a KV cache storage space, and other parameter storage spaces.

[0093] See also Figure 6 In this embodiment, the GPU memory is divided into multiple storage spaces, and the multiple storage spaces are used to store different parameters respectively. One of the storage spaces is the adapter storage space, and the multiple candidate low-rank adapters are stored in pages in the adapter storage space. Furthermore, the present application can update the adapter in the adapter storage space alone. Compared with the related art in which the model weight (Base Model), adapter weight (Adapter), KV cache (KV Cache) and other parameters are mixed and stored in the same space, the way of dividing the space storage in this embodiment is more convenient for the management and hot update of the low-rank adapter.

[0094] Furthermore, in the aforementioned embodiment, swapping out each of the candidate low-rank adapters from the graphics processor memory may specifically include: swapping out each of the candidate low-rank adapters from the graphics processor memory according to the scheduling priority of each of the candidate low-rank adapters, including: determining the candidate low-rank adapter to be swapped out according to the scheduling priority of each of the candidate low-rank adapters; obtaining an address set of the candidate low-rank adapter to be swapped out in the adapter storage space according to the low-rank adapter control block corresponding to the candidate low-rank adapter to be swapped out, and obtaining a target address set; determining the paging of the candidate low-rank adapter to be swapped out in the adapter storage space according to the target address set, and obtaining a target paging; swapping out the candidate low-rank adapter to be swapped out from the target paging in the adapter storage space.

[0095] Multiple candidate low-rank adapters are stored in pages in the adapter storage space. When a candidate low-rank adapter is created, a corresponding low-rank adapter control block is created in the corresponding storage space (such as graphics processor memory, etc.), and the page table information (Page Table) in the low-rank adapter control block is updated. The page table information is a set of addresses of the candidate low-rank adapter in the storage space (such as main memory, graphics processor memory, etc.). The address set in the page table information points to the specific address of the candidate low-rank adapter in the storage space.

[0096] Thus, after determining the candidate low-rank adapters to be replaced according to the scheduling priority of each candidate low-rank adapter (for example, one or more candidate low-rank adapters with the lowest scheduling priority are determined as the candidate low-rank adapters to be replaced), the address set of the candidate low-rank adapters to be replaced in the adapter storage space can be obtained according to the low-rank adapter control blocks corresponding to the candidate low-rank adapters to be replaced, and this address set is used as the target address set to be obtained.

[0097] Furthermore, the paging of the candidate low-rank adapter to be swapped out in the adapter storage space can be determined according to the target address set, and the paging is the target paging obtained. Thus, the candidate low-rank adapter to be swapped out can be accurately swapped out from the target paging in the adapter storage space.

[0098] If necessary, the graphics processing unit (GPU) in this article can be technically replaced by a tensor processing unit (TPU), a neural network processing unit (NPU) or other devices that can be used for model calculation acceleration. Correspondingly, the GPU memory can be technically replaced by the memory of the corresponding device that can be used for model calculation acceleration. In addition, the technical solution described in this article can also be expressed in the form of a computer cluster to express a similar technical solution.

[0099] In order to facilitate better implementation of the low-rank adapter hot update method provided in the embodiment of the present application, the embodiment of the present application also provides a low-rank adapter hot update device based on the above low-rank adapter hot update method. The meaning of the terms is the same as in the above low-rank adapter hot update method, and the specific implementation details can refer to the description in the method embodiment. Figure 7 A block diagram of a low-rank adapter hot update device according to an embodiment of the present application is shown.

[0100] like Figure 7 As shown, the low-rank adapter hot update device 500 may include: a download module 510, used to download a new version of the low-rank adapter corresponding to the old version of the low-rank adapter from the adapter management end; a saving module 520, used to save the new version of the low-rank adapter in the adapter repository in the local storage; a copy module 530, used to copy the low-rank adapter control block that inherits the old version of the low-rank adapter; an update module 540, used to perform state migration processing according to the low-rank adapter control block, so as to use the new version of the low-rank adapter in the adapter repository to hot update the old version of the low-rank adapter.

[0101] In some embodiments of the present application, the update module is used to: remove the old version of the low-rank adapter from multiple adapter storage locations, the multiple adapter storage locations including the local storage, the main memory, and the graphics processor memory; according to the lack of low-rank adapter interrupt / lack of low-rank adapter exception and the low-rank adapter control block of the old version of the low-rank adapter, query and schedule the new version of the low-rank adapter to the graphics processor memory step by step from the graphics processor memory, the main memory to the local storage.

[0102] In some embodiments of the present application, the update module is used to: determine the location corresponding to the state from the local storage, main memory, and graphics processor memory according to the state in the low-rank adapter control block to obtain the target location; remove the old version of the low-rank adapter from the target location; and update the new version of the low-rank adapter in the adapter repository to the target location.

[0103] In some embodiments of the present application, the update module is used to: determine the location corresponding to the state from the local storage, main memory, and graphics processor memory according to the state in the low-rank adapter control block to obtain the target location; change the information in the low-rank adapter control block corresponding to the old version of the low-rank adapter in the target location to the information corresponding to the new version of the low-rank adapter; and remove the old version of the low-rank adapter from the target location.

[0104] In some embodiments of the present application, before copying and inheriting the low-rank adapter control block of the old version of the low-rank adapter, the device also includes a verification module for: verifying the new version of the low-rank adapter to obtain a verification result; and determining whether to copy and inherit the low-rank adapter control block of the old version of the low-rank adapter based on the verification result.

[0105] In some embodiments of the present application, the local includes multiple adapter storage locations, and the multiple adapter storage locations include the local storage, main memory and graphics processor memory. The graphics processor memory stores a set of low-rank adapters for use with large models, and the low-rank adapter set includes multiple candidate low-rank adapters; the device also includes a scheduling module, which is used to: determine whether the access time of each of the candidate low-rank adapters is within the first time range or the second time range, and obtain a target time range, wherein the first time range is farther away from the current time than the second time range; calculate the scheduling priority of each of the candidate low-rank adapters according to the first cut-off weight and the second cut-off weight corresponding to the target time range; if the adapter needs to be swapped out, swap each of the candidate low-rank adapters out of the graphics processor memory according to the scheduling priority of each of the candidate low-rank adapters.

[0106] In some embodiments of the present application, the scheduling module is used to: calculate the first priority of each candidate low-rank adapter according to the interval duration corresponding to each candidate low-rank adapter, wherein the interval duration is the time elapsed between the time of this round of reasoning and the time of the last visit to the candidate low-rank adapter; calculate the second priority of each candidate low-rank adapter according to the number of visits corresponding to each candidate low-rank adapter, wherein the number of visits is the number of times the candidate low-rank adapter is visited before this round of reasoning; calculate the scheduling priority corresponding to each candidate low-rank adapter according to the first priority, the first cut-off weight, the second priority and the second cut-off weight corresponding to each candidate low-rank adapter.

[0107] In some embodiments of the present application, the scheduling module is used to: according to the formula Priority1=e -ax The first priority Priority1 of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1].

[0108] In some embodiments of the present application, the scheduling module is used to: according to the formula Priority2=1-e -by The second priority Priority2 of the candidate low-rank adapter is calculated, where b is the second coefficient, b>0, y is the number of visits corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1].

[0109] In some embodiments of the present application, the scheduling module is used to: calculate the scheduling priority Priority3 corresponding to the candidate low-rank adapter according to the formula Priority3=μ[α(Priority1)+β(Priority2)], wherein α and β are the first trade-off weight and the second trade-off weight, respectively, α, β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

[0110] In some embodiments of the present application, the graphics processor memory is divided into multiple storage spaces, and the multiple storage spaces are used to store different parameters respectively; the multiple storage spaces include an adapter storage space, and the multiple candidate low-rank adapters are stored in pages in the adapter storage space; the scheduling module is used to: determine the candidate low-rank adapter to be swapped out according to the scheduling priority of each of the candidate low-rank adapters; obtain the address set of the candidate low-rank adapter to be swapped out in the adapter storage space according to the low-rank adapter control block corresponding to the candidate low-rank adapter to be swapped out, and obtain a target address set; determine the paging of the candidate low-rank adapter to be swapped out in the adapter storage space according to the target address set, and obtain a target page; swap out the candidate low-rank adapter to be swapped out from the target page in the adapter storage space.

[0111] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0112] In addition, the present application also provides an electronic device, such as Figure 8 As shown, Figure 8 A block diagram of an electronic device according to an embodiment of the present application is shown, specifically:

[0113] The electronic device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will appreciate that Figure 8 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0114] in:

[0115] The processor 601 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, it executes various functions of the computer device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user pages and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.

[0116] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0117] The electronic device also includes a power supply 603 for supplying power to each component. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 603 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0118] The electronic device may further include an input unit 604, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0119] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 601 in the electronic device will load the executable files corresponding to the processes of one or more computer programs into the memory 602 according to the following instructions, and the processor 601 will run the computer programs stored in the memory 602, thereby realizing various functions in the aforementioned embodiments of the present application, such as the processor 601 may perform the following steps:

[0120] Download a new version of the low-rank adapter corresponding to the old version of the low-rank adapter from the adapter management end; save the new version of the low-rank adapter in the adapter repository in the local storage; copy and inherit the low-rank adapter control block of the old version of the low-rank adapter; perform state migration processing according to the low-rank adapter control block to use the new version of the low-rank adapter in the adapter repository to hot update the old version of the low-rank adapter.

[0121] In some embodiments of the present application, the state migration processing is performed according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository, including: removing the old version of the low-rank adapter from multiple adapter storage locations, the multiple adapter storage locations including the local storage, main memory, and graphics processor memory; according to the lack of low-rank adapter interruption / lack of low-rank adapter exception of the old version of the low-rank adapter and the low-rank adapter control block, querying and scheduling the new version of the low-rank adapter to the graphics processor memory step by step from the graphics processor memory, the main memory to the local storage.

[0122] In some embodiments of the present application, the state migration processing is performed according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository, including: according to the state in the low-rank adapter control block, determining the location corresponding to the state from the local storage, main memory, and graphics processor memory to obtain the target location; removing the old version of the low-rank adapter from the target location; and updating the new version of the low-rank adapter in the adapter repository to the target location.

[0123] In some embodiments of the present application, the state migration processing is performed according to the low-rank adapter control block to hot update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository, including: according to the state in the low-rank adapter control block, determining the location corresponding to the state from the local storage, main memory, and graphics processor memory to obtain the target location; changing the information in the low-rank adapter control block corresponding to the old version of the low-rank adapter in the target location to the information corresponding to the new version of the low-rank adapter; and removing the old version of the low-rank adapter from the target location.

[0124] In some embodiments of the present application, before copying and inheriting the low-rank adapter control block of the old version of the low-rank adapter, the method also includes: verifying the new version of the low-rank adapter to obtain a verification result; and determining whether to copy and inherit the low-rank adapter control block of the old version of the low-rank adapter based on the verification result.

[0125] In some embodiments of the present application, the local includes multiple adapter storage locations, and the multiple adapter storage locations include the local storage, main memory and graphics processor memory. The graphics processor memory stores a set of low-rank adapters for use with large models, and the low-rank adapter set includes multiple candidate low-rank adapters; the method also includes: determining that the access time of each of the candidate low-rank adapters is within the first time range or the second time range, and obtaining a target time range, wherein the first time range is farther away from the current time than the second time range; calculating the scheduling priority of each of the candidate low-rank adapters according to the first cut-off weight and the second cut-off weight corresponding to the target time range; if the adapter needs to be swapped out, swapping each of the candidate low-rank adapters out of the graphics processor memory according to the scheduling priority of each of the candidate low-rank adapters.

[0126] In some embodiments of the present application, the scheduling priority of each candidate low-rank adapter is calculated based on the first cut-off weight corresponding to the first adapter set and the second cut-off weight corresponding to the second adapter set, including: calculating the first priority of each candidate low-rank adapter based on the interval duration corresponding to each candidate low-rank adapter, wherein the interval duration is the time elapsed between the time of this round of reasoning and the time of the last visit to the candidate low-rank adapter; calculating the second priority of each candidate low-rank adapter based on the number of visits corresponding to each candidate low-rank adapter, wherein the number of visits is the number of times the candidate low-rank adapter is visited before this round of reasoning; calculating the scheduling priority corresponding to each candidate low-rank adapter based on the first priority, the first cut-off weight, the second priority and the second cut-off weight corresponding to each candidate low-rank adapter.

[0127] In some embodiments of the present application, the first priority of each candidate low-rank adapter is calculated according to the interval duration corresponding to each candidate low-rank adapter, including: according to the formula Priority1=e -ax The first priority Priority1 of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1].

[0128] In some embodiments of the present application, the second priority of each candidate low-rank adapter is calculated according to the number of accesses corresponding to each candidate low-rank adapter, including: according to the formula Priority2=1-e -by The second priority Priority2 of the candidate low-rank adapter is calculated, where b is the second coefficient, b>0, y is the number of visits corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1].

[0129] In some embodiments of the present application, the scheduling priority corresponding to each candidate low-rank adapter is calculated according to the first priority, the first trade-off weight, the second priority and the second trade-off weight corresponding to each candidate low-rank adapter, including: calculating the scheduling priority Priority3 corresponding to the candidate low-rank adapter according to the formula Priority3=μ[α(Priority1)+β(Priority2)], wherein α, β are the first trade-off weight and the second trade-off weight respectively, α, β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

[0130] In some embodiments of the present application, the graphics processor memory is divided into multiple storage spaces, and the multiple storage spaces are used to store different parameters respectively; the multiple storage spaces include an adapter storage space, and the multiple candidate low-rank adapters are stored in pages in the adapter storage space; swapping out each candidate low-rank adapter from the graphics processor memory according to the scheduling priority of each candidate low-rank adapter includes: determining the candidate low-rank adapter to be swapped out according to the scheduling priority of each candidate low-rank adapter; obtaining an address set of the candidate low-rank adapter to be swapped out in the adapter storage space according to the low-rank adapter control block corresponding to the candidate low-rank adapter to be swapped out, and obtaining a target address set; determining the paging of the candidate low-rank adapter to be swapped out in the adapter storage space according to the target address set, and obtaining a target page; swapping out the candidate low-rank adapter to be swapped out from the target page in the adapter storage space.

[0131] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0132] To this end, an embodiment of the present application further provides a storage medium, in which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any method provided in the embodiment of the present application.

[0133] The storage medium may be a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0134] Since the computer program stored in the storage medium can execute the steps in any method provided in the embodiments of the present application, the beneficial effects that can be achieved by the method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0135] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0136] It should be understood that the present application is not limited to the embodiments that have been described above and shown in the accompanying drawings, but various modifications and changes may be made without departing from the scope thereof.

Claims

1. A low-rank adapter hot update method, characterized in that: include: Download the new version of the low-rank adapter corresponding to the old version of the low-rank adapter from the adapter management end; Storing the new version of the low-rank adapter in an adapter repository in local storage; Copying a low-rank adapter control block that inherits the old version of the low-rank adapter; State migration processing is performed according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

2. The method according to claim 1, characterized in that The state migration processing is performed according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository, including: removing the old version low-rank adapter from a plurality of adapter storage locations, the plurality of adapter storage locations comprising the local storage, main memory, and graphics processor memory; According to the low-rank adapter missing interrupt / low-rank adapter missing exception of the old version of the low-rank adapter and the low-rank adapter control block, the new version of the low-rank adapter is queried and scheduled to the graphics processor memory step by step from the graphics processor memory, the main memory to the local storage.

3. The method according to claim 1, characterized in that The state migration processing is performed according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository, including: According to the state in the low-rank adapter control block, determine the position corresponding to the state from the local storage, the main memory, and the graphics processor memory to obtain a target position; removing the old version low-rank adapter from the target location; The new version of the low-rank adapter in the adapter repository is updated to the target location.

4. The method according to claim 1, characterized in that The state migration processing is performed according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository, including: According to the state in the low-rank adapter control block, determine the position corresponding to the state from the local storage, the main memory, and the graphics processor memory to obtain a target position; Change the information in the low-rank adapter control block corresponding to the old version of the low-rank adapter in the target position to the information corresponding to the new version of the low-rank adapter; The old version low-rank adapter is removed from the target location.

5. The method according to claim 1, characterized in that Before the copying inherits the low-rank adapter control block of the old version low-rank adapter, the method further includes: Verifying the new version of the low-rank adapter to obtain a verification result; According to the verification result, it is determined whether to copy the low-rank adapter control block that inherits the old version of the low-rank adapter.

6. The method according to any one of claims 1 to 5, characterized in that: The local includes multiple adapter storage locations, the multiple adapter storage locations include the local storage, the main memory and the graphics processor memory, the graphics processor memory stores a low-rank adapter set for use with a large model, and the low-rank adapter set includes multiple candidate low-rank adapters; The method further comprises: Determine that the access time of each candidate low-rank adapter is within a first time range or a second time range, and obtain a target time range, wherein the first time range is farther from the current time than the second time range; Calculating the scheduling priority of each of the candidate low-rank adapters according to the first cut-off weight and the second cut-off weight corresponding to the target time range; If the adapter needs to be swapped out, each of the candidate low-rank adapters is swapped out from the graphics processor memory according to the scheduling priority of each of the candidate low-rank adapters.

7. The method according to claim 6, characterized in that The calculating the scheduling priority of each of the candidate low-rank adapters according to the first cut-off weight and the second cut-off weight corresponding to the target time range includes: The first priority of each candidate low-rank adapter is calculated according to the interval duration corresponding to each candidate low-rank adapter, wherein the interval duration is the time elapsed from the current round of reasoning time to the last access time of the candidate low-rank adapter; The second priority of each candidate low-rank adapter is calculated according to the number of visits corresponding to each candidate low-rank adapter, wherein the number of visits is the number of times the candidate low-rank adapter is visited before this round of reasoning; The scheduling priority corresponding to each candidate low-rank adapter is calculated based on the first priority, the first cut-off weight, the second priority and the second cut-off weight corresponding to each candidate low-rank adapter.

8. The method according to claim 7, characterized in that The step of calculating the first priority of each candidate low-rank adapter according to the interval duration corresponding to each candidate low-rank adapter includes: According to the formula Priority1=e -ax The first priority Priority1 of the candidate low-rank adapter is calculated, where a is the first coefficient, a>0, x is the interval duration corresponding to the candidate low-rank adapter, and the value range of Priority1 is (0,1].

9. The method according to claim 7, characterized in that: The step of calculating the second priority of each candidate low-rank adapter according to the number of accesses corresponding to each candidate low-rank adapter includes: According to the formula Priority2 = 1-e -b y is used to calculate the second priority Priority2 of the candidate low-rank adapter, where b is the second coefficient, b>0, y is the number of visits corresponding to the candidate low-rank adapter, and the value range of Priority2 is (0,1].

10. The method according to claim 7, characterized in that The calculating, according to the first priority, the first cut-off weight, the second priority and the second cut-off weight corresponding to each candidate low-rank adapter, the scheduling priority corresponding to each candidate low-rank adapter comprises: The scheduling priority Priority3 corresponding to the candidate low-rank adapter is calculated according to the formula Priority3 = μ[α(Priority1) + β(Priority2)], where α and β are the first trade-off weight and the second trade-off weight, respectively, α, β∈[0,1] and α+β=1, Priority1 is the first priority, Priority2 is the second priority, and μ represents the intervention weight.

11. The method according to claim 6, characterized in that The graphics processor memory is divided into a plurality of storage spaces, and the plurality of storage spaces are respectively used to store different parameters; the plurality of storage spaces include an adapter storage space, and the plurality of candidate low-rank adapters are stored in pages in the adapter storage space; The step of swapping out each of the candidate low-rank adapters from the graphics processor memory according to the scheduling priority of each of the candidate low-rank adapters includes: Determining a candidate low-rank adapter to be replaced according to the scheduling priority of each candidate low-rank adapter; According to the low-rank adapter control block corresponding to the candidate low-rank adapter to be replaced, obtain the address set of the candidate low-rank adapter to be replaced in the adapter storage space, and obtain the target address set; Determine the paging of the candidate low-rank adapter to be swapped out in the adapter storage space according to the target address set, and obtain the target paging; The candidate low-rank adapter to be swapped out is swapped out from the target page in the adapter storage space.

12. A low-rank adapter hot update device, characterized in that: include: A download module, used to download a new version of a low-rank adapter corresponding to an old version of a low-rank adapter from an adapter management terminal; A saving module, used to save the new version of the low-rank adapter in an adapter repository in a local storage; A copy module, used for copying a low-rank adapter control block that inherits the old version of the low-rank adapter; An update module is used to perform state migration processing according to the low-rank adapter control block to hot-update the old version of the low-rank adapter using the new version of the low-rank adapter in the adapter repository.

13. A storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

14. An electronic device, characterized in that: include: a memory storing a computer program; A processor reads a computer program stored in a memory to execute the method according to any one of claims 1 to 11.

Images