Distributed LoRA unloading operation method and system

Abstract

The invention relates to a distributed LoRA unloading operation method and system. The method comprises the following steps: initializing a Kademlia DHT and a communication protocol which are respectively used for managing fragmented storage and inter-node communication; fragmenting the weight matrix W of the pre-training model and the LoRA matrixes F and G, storing fragments into corresponding storage managers, and managing fragment information through DHT; generating a calculation task context according to a calculation formula; the calculation manager dynamically queries required storage information according to the context of the calculation task, pulls required data fragments from the storage manager through a communication protocol, and executes the calculation task on the calculation manager; and summarizing the obtained task calculation results through an aggregate communication protocol, and returning the final calculation result to a user or storing the final calculation result in a specified storage position. According to the method, the memory overhead pressure of a single node is unloaded by using remote resources, and PEFT operation of nodes in a resource-constrained environment is supported.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a distributed LoRA offloading operation method and system. Background Art

[0002] Large models generally use pre-training and fine-tuning to support downstream tasks. Unsupervised learning is used in the pre-training stage to predict the next token through an autoregressive model, and parameter training can be performed using public Internet data sets. After completing the pre-training of the model, in order to adapt to downstream tasks, the large model needs to use task-related data sets to continue to optimize parameters to improve task-oriented effects.

[0003] At present, parameter-efficient fine-tuning (PEFT) based on Low-Rank Adaptation (LoRA) is more common, such as Figure 1 As shown in the figure, PEFT fixes the pre-trained model weight matrix W (size is N*M), and adds a low-order matrix decomposition F*G (size is N*d, d*M respectively), and uses the algebraic sum of the two W+ F*G to replace the model weights, and then optimizes the learning of F and G through the task data set. Since d is much smaller than N or M, the computational complexity of model training can be reduced. However, LoRA operations require large-scale matrix multiplication operations. Since the dimensions of N and M are still much higher than d, the memory overhead of LoRA still puts pressure on the accelerator HBM memory of a single node, and it is urgently needed to be unloaded. Summary of the invention

[0004] Based on this, it is necessary to provide a distributed LoRA offloading operation method and system for the above technical problems. This method can use remote resources to offload the memory overhead pressure of a single node and support PEFT operations of nodes in resource-constrained environments.

[0005] A distributed LoRA offloading operation method, the method comprising: Initialize Kademlia DHT and communication protocol for managing shard storage and inter-node communication respectively; add each peer node to DHT to form a distributed network; a peer node is a logical process or physical node with memory resources and computing resources; DHT is used to provide distributed resource management services for storage pools and computing pools, and each peer uses DHT to manage shard storage.

[0006] The pre-trained model weight matrix W, LoRA matrix F and G are sharded, and the obtained shards are stored in the corresponding storage manager, and the shard information is managed through DHT; the storage manager is a Peer node used to store offload tasks.

[0007] Generates a calculation task context based on the calculation formula.

[0008] The computing manager dynamically queries the required storage information according to the computing task context, pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager; the computing manager is a peer node used for computing offload tasks.

[0009] The obtained task calculation results are summarized through the collective communication protocol to obtain the final calculation result.

[0010] The final calculation result is returned to the user or stored in a specified storage location.

[0011] In one embodiment, a collective communication protocol is used for point-to-point or collective communication between nodes, supporting send, receive, broadcast, aggregate and reduce operations.

[0012] Kademlia DHT is used to provide efficient distributed hash table functions and support the insertion, query and deletion operations of shards.

[0013] In one of the embodiments, collective communication refers to supporting one-way or two-way communication between a group of nodes, including: one-to-many communication, many-to-one communication, many-to-one communication and convergence.

[0014] In one embodiment, the pre-trained model weight matrix W, the LoRA matrices F and G are sharded, and the obtained shards are stored in the corresponding storage manager, and the shard information is managed through the DHT, including: Decompose the pre-trained model weight matrix W, LoRA matrix F and G into sub-matrices according to row index, column index or custom method, and take each sub-matrix as a slice; the identifier of each slice corresponds to the concatenated string of the matrix name and index, and the value of the slice is a tensor type, which can be stored independently in the memory of different nodes.

[0015] Each shard is stored in the corresponding storage manager according to its identifier. If fault-tolerant storage is required, it is serialized and stored in the file system of each node through pickle.

[0016] Shard information is managed through DHT, supporting the insertion, query and deletion of shards.

[0017] In one embodiment, generating a computing task context according to a computing formula includes: The computing task context is generated according to the computing formula; the computing task context contains the required shard identifier and the computing formula; the computing task context is used to guide the computing manager to dynamically query the required storage information when executing the computing task, and pull the required data shards through the communication protocol; the calculation formula is:

[0018] in, W is the pre-trained model weight matrix, To calculate the task context, F and G are LoRA matrices.

[0019] In one embodiment, the computing task execution process includes: matrix multiplication and addition operations.

[0020] In one embodiment, the obtained task calculation results are aggregated through a collective communication protocol to obtain a final calculation result, including: The calculation manager sends the obtained task calculation results to the summary node through the collective communication protocol, performs aggregation and reduction operations on the task calculation results, and obtains the final calculation results. In one embodiment, the computing manager dynamically queries required storage information according to the computing task context, pulls required data shards from the storage manager through a communication protocol, and executes the computing task on the computing manager, including: The computing manager dynamically queries the required storage information according to the computing task context and the shard index of the pre-trained model weight matrix W, LoRA matrix F and G, and pulls the required data shards from the storage manager through the communication protocol; the data shard query is managed through DHT, which supports the insertion, query and deletion of shards.

[0021] The compute manager pulls the required data shards from the storage manager through the communication protocol and executes the computing tasks on the compute manager.

[0022] A distributed LoRA offloading computing system comprises: a top-level scheduler, a storage pool and a computing pool, wherein the computing pool comprises a plurality of computing managers, and the storage pool comprises a plurality of storage managers.

[0023] The top-level scheduler is used to initialize the Kademlia DHT and communication protocol used to manage shard storage and inter-node communication respectively; each Peer node is added to the DHT to form a distributed network; it is also used to shard the pre-trained model weight matrix W, LoRA matrices F and G, and store the obtained shards in the corresponding storage manager, and manage the shard information through the DHT; it is also used to generate the computing task context according to the calculation formula, and push the computing task context to the corresponding computing manager; the Peer node is a logical process or physical node with memory resources and computing resources; DHT is used to provide distributed resource management services for storage pools and computing pools, and each Peer uses DHT to manage shard storage.

[0024] The storage manager is a peer node used for storage offloading tasks.

[0025] The compute manager is a peer node used to compute offload tasks. It is used to receive the compute task context pushed by the top-level scheduler, and dynamically query the required storage information based on the compute task context. It pulls the required data shards from the storage manager through the communication protocol, executes the compute tasks, and sends the compute results of the tasks to the summary node through the communication protocol. It performs aggregation and reduction operations on the results to obtain the final compute results, and returns the final compute results to the user or stores them in the specified storage location.

[0026] In one of the embodiments, each node of the storage pool maintains the storage of the shards through a dictionary, and the key-value pairs of the dictionary correspond to the identifier of the shard and the value of the matrix respectively.

[0027] The above-mentioned distributed LoRA offloading operation method and system, the method includes: initializing Kademlia DHT and communication protocol for managing shard storage and inter-node communication respectively; sharding the pre-trained model weight matrix W, LoRA matrix F and G, and storing the shards in the corresponding storage manager, and managing the shard information through DHT; generating a computing task context according to the calculation formula; the computing manager dynamically queries the required storage information according to the computing task context, pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager; summarizes the obtained task calculation results through the collective communication protocol, and returns the final calculation results to the user or stores them in a specified storage location. The method uses remote resources to offload the memory overhead pressure of a single node and supports PEFT operations of nodes in resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the LoRA operation process; Figure 2It is a flowchart of a distributed LoRA offloading operation method in one embodiment; Figure 3 A distributed initialization flow chart in another embodiment; Figure 4 This is a workflow diagram of the LoRA top-level scheduler in another embodiment; Figure 5 A flowchart of shard storage in another embodiment; Figure 6 A flowchart of executing a computing task on a computing manager in another embodiment; Figure 7 This is a working principle diagram of a distributed LoRA offloading computing system in one embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] The distributed LoRA offloading operation method proposed in this application offloads LoRA calculation and storage, that is, the LoRA storage task is only responsible for offloading the storage requirements of a single node. According to the memory and computing resource limitations of a single node, the pre-trained matrix W, LoRA matrix F, G are stored in a group of decentralized nodes through sharding; the LoRA calculation generates the calculation task according to the calculation formula, dynamically queries the required storage information, pulls the required data shards through the communication protocol, and completes the calculation.

[0031] In one embodiment, Figure 2 As shown, a distributed LoRA offloading operation method is provided, the method comprising the following steps: Step 100: Initialize Kademlia DHT and communication protocol for managing shard storage and inter-node communication respectively; add each peer node to DHT to form a distributed network; Peer node is a logical process or physical node with memory resources and computing resources; DHT is used to provide distributed resource management services for storage pool and computing pool, and each peer uses DHT to manage shard storage.

[0032] Specifically, a Peer is a logical process or physical node with certain memory resources and computing resources; it is a node responsible for computing and storing offload tasks.

[0033] Initialize Kademlia DHT and communication protocol to manage shard storage and communication between nodes. The initialization process is as follows Figure 3As shown. First, initialize a DHT server and listen to the specified port; then, define the peer node list and bootstrap each peer node to the DHT network. Each peer node joins the DHT to form a distributed network. Kademlia DHT provides efficient distributed hash table functions and supports the insertion, query and deletion of shards. The collective communication protocol is used for point-to-point and collective communication between nodes, supporting sending, receiving, broadcasting, aggregation and reduction operations. The initialization process is as follows Figure 3 shown.

[0034] Distributed Hash Table (DHT): Provides distributed resource management services for storage pools and computing pools. Each peer uses DHT to manage shard storage, i.e. inserting shards, querying shards, and deleting shards. Storage pool: Each node maintains the storage of shards through a dictionary. The key-value pairs of the dictionary correspond to the identifier of the shard and the value of the matrix respectively. Computing pool: Peer responsible for PEFT, performs distributed LoRA calculations.

[0035] Step 102: Slice the pre-trained model weight matrix W, LoRA matrices F and G, and store the obtained slices in the corresponding storage manager, and manage the slice information through DHT; the storage manager is a Peer node used to store offload tasks.

[0036] Specifically, the pre-trained matrix W, LoRA matrix F, G are decomposed into sub-matrices according to row index, column index or custom method, and each sub-matrix is ​​called a shard. The identifier of each shard corresponds to the concatenated string of the matrix name and index. The value of the shard is a tensor type and can be stored independently in the memory of different nodes.

[0037] Step 104: Generate a computing task context according to the computing formula.

[0038] Specifically, the generation of computing task context can be customized according to specific computing requirements and shard storage conditions.

[0039] Step 106: The computing manager dynamically queries the required storage information according to the computing task context, pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager; the computing manager is a peer node used for computing offload tasks.

[0040] Specifically, the computing tasks are executed on the computing manager. The computing manager receives the computing tasks pushed by the top-level scheduler, executes the computing tasks of the responsible shards, and ensures that the computing tasks of the shards can be completed efficiently. According to the computing task context, the required storage information is dynamically queried, the required data shards are pulled through the communication protocol, and the computing tasks are executed on the computing manager.

[0041] LoRA top-level scheduling: responsible for managing and coordinating the computing tasks of the shards. It pushes tasks to each computing manager, each computing manager performs local calculations, and then the top-level scheduler gathers the results of each computing manager. The top-level scheduler dynamically queries the required storage information based on the computing task context, pulls the required data shards through the communication protocol, and executes the computing tasks on the computing manager, and finally summarizes the computing results through collective communication.

[0042] The LoRA Top Level Scheduler is used for large parameter matrices (such as W, F, G) of distributed computing. It stores large matrices in slices and distributes computing tasks to multiple computing nodes, and finally summarizes the computing results. The scheduler uses distributed hash tables (DHT) and communication protocols to achieve efficient distributed computing.

[0043] (1) Scheduling: First, shard W, F, and G according to the shard size (shard_size) and store them in the DHT; then, generate computing tasks for each shard and push them to the computing manager; wait for all computing tasks to be completed and pull the computing results from the computing manager; finally, aggregate all the results to obtain the final result.

[0044] (2) Push task process: Obtain the corresponding computing node ID through DHT and use the communication module to send the task to the computing node.

[0045] (3) Pull result process: Get the node ID where the result is located through DHT, and use the communication module to receive the calculation result from the node.

[0046] The workflow of the LoRA top-level scheduler is as follows Figure 4 shown.

[0047] Step 108: Summarize the obtained task calculation results through the collective communication protocol to obtain the final calculation result; return the final calculation result to the user or store it in a designated storage location.

[0048] Specifically, the calculation results are summarized through collective communication. The calculation manager sends the calculation results to the summary node through the collective communication protocol to perform result aggregation and reduction operations. The result aggregation process can adopt a variety of collective communication methods, such as broadcast, aggregation and reduction. Through result aggregation, the final calculation result can be obtained and returned to the user or stored in a specified storage location.

[0049] In the above-mentioned distributed LoRA offloading operation method, the method includes: initializing Kademlia DHT and communication protocol for managing shard storage and inter-node communication respectively; sharding the pre-trained model weight matrix W, LoRA matrix F and G, and storing the shards in the corresponding storage manager, and managing the shard information through DHT; generating the computing task context according to the calculation formula; the computing manager dynamically queries the required storage information according to the computing task context, pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager; summarizes the obtained task calculation results through the collective communication protocol, and returns the final calculation results to the user or stores them in the specified storage location. This method uses remote resources to offload the memory overhead pressure of a single node and supports PEFT operations of nodes in resource-constrained environments.

[0050] In one embodiment, a collective communication protocol is used for point-to-point or collective communication between nodes, supporting send, receive, broadcast, aggregate and reduce operations.

[0051] Kademlia DHT is used to provide efficient distributed hash table functions and support the insertion, query and deletion operations of shards.

[0052] Specifically, point-to-point communication supports send / receive synchronous or asynchronous communication.

[0053] In one of the embodiments, collective communication refers to supporting one-way or two-way communication between a group of nodes, including: one-to-many communication, many-to-one communication, many-to-one communication and convergence.

[0054] In one embodiment, step 102 includes: decomposing the pre-trained model weight matrix W, LoRA matrix F and G into sub-matrices according to row index, column index or custom method, and taking each sub-matrix as a shard; wherein the identifier of each shard corresponds to the concatenated string of the matrix name and index, and the value of the shard is a tensor type, which can be independently stored in the memory of different nodes; storing each shard in the corresponding storage manager according to its identifier, and if fault-tolerant storage is required, storing it in the file system of each node through pickle serialization; managing the shard information through DHT, supporting the insertion, query and deletion of shards.

[0055] Specifically, the shard storage process is as follows: Figure 5 As shown. Sharded storage divides the given matrix into multiple smaller blocks (shards) according to the specified shard size, associates these shards with unique identifiers, and then records the location of each shard (corresponding peer_id) in the distributed hash table (DHT). The specific steps include: Step 1: Get the matrix data matrix to be sharded, the name of the matrix matrix_name, the number of rows in each shard (assuming the matrix is ​​sharded by row) shard_size, and the node identifier peer_id responsible for storing the shards. The name of the matrix matrix_name is used to generate a unique identifier for each shard.

[0056] Step 2: Create an empty dictionary shards to store the shard identifier and the corresponding shard data.

[0057] Step 3: Divide the number of rows in the matrix by the shard size to get the required number of shards num_shards.

[0058] Step 4: Loop number: from 0 to num_shards - 1, process each shard one by one, extract the shard: Step 4.1: Extract sub-matrices (slices) of the matrix based on the slice size.

[0059] Step 4.2: Generate Shard Identifier: Generate a unique shard_id using the matrix name and shard index, such as "W_0", "W_1", etc.

[0060] Step 4.3: Update shard mapping: Store the shard data in the shards dictionary, with the key being shard_id and the value being shard.

[0061] Step 4.4: Select DHT storage node peer_id: Select the storage node peer_id of the shard according to the DHT storage rules or user-defined storage mapping rules.

[0062] Step 4.5: Record the correspondence between the shard identifier and the storage node peer_id in the distributed hash table.

[0063] In one embodiment, step 104 includes: generating a computing task context according to a computing formula; the computing task context includes a required shard identifier and a computing formula; the computing task context is used to guide the computing manager to dynamically query the required storage information when executing the computing task, and to pull the required data shards through the communication protocol; the computing formula is: (1) in, W is the pre-trained model weight matrix, To calculate the task context, F and G are LoRA matrices.

[0064] Specifically, the computing task context is generated according to the computing formula (1). The computing task context contains the required shard identifier and the computing formula. The computing task context is used to guide the computing manager to dynamically query the required storage information when executing the computing task, and to pull the required data shards through the communication protocol. The generation of the computing task context can be customized according to the specific computing requirements and shard storage conditions. The generated context data dictionary is: {"W_shard_id": W_shard_id, "F_shard_id": F_shard_id, "G_shard_id": G_shard_id} Among them, W_shard_id is the pre-trained model weight matrix W , F_shard_id is the shard identifier of the LoRA matrix F, and G_shard_id is the shard identifier of the LoRA matrix G.

[0065] In one embodiment, the computing task execution process includes: matrix multiplication and addition operations.

[0066] In one embodiment, step 108 includes: the computing manager sends the obtained task computing results to the aggregation node through the collective communication protocol, performs aggregation and reduction operations on the task computing results, and obtains the final computing results.

[0067] In one of the embodiments, step 106 includes: the computing manager dynamically queries the required storage information according to the computing task context and the shard index of the pre-trained model weight matrix W, the LoRA matrix F and G, and pulls the required data shards from the storage manager through the communication protocol; wherein the data shard query is managed through the DHT, supporting the insertion, query and deletion operations of the shards; the computing manager pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager.

[0068] Specifically, data shard query: According to the shard index of W, F, and G, the computing task context dynamically queries the required storage information and pulls the required data shards through the communication protocol. Data shard query is managed through DHT, which supports the insertion, query, and deletion of shards. Through the communication protocol, the computing manager can pull the required data shards from the storage manager to perform computing tasks locally. The process of data shard query is dynamic and can be adjusted according to the needs of computing tasks.

[0069] The specific steps to obtain the shard include: Select the peer_id corresponding to the shard ID shard_id: peer_id = dht.get(shard_id) Where peer_id is the peer identifier corresponding to the shard returned by dht.get(), shard_id is the shard identifier, and dht.get() is the peer identifier for the corresponding shard pulled from DHT.

[0070] Pull data allocation based on shard_id and peer_id: shard = receive(peer_id, shard_id) Among them, shard is the data shard to be pulled, and receive() is to pull the shard with the identifier dimension shard_id from the distribution node corresponding to peer_id and store it in shard.

[0071] The process of executing computing tasks on the computing manager is as follows Figure 6 The steps to perform the calculation task are as follows: (1) Initialization: DHT reference, compute_task. Compute_task contains a dictionary of compute task information, including shard identifiers such as "W_shard_id", "F_shard_id", and "G_shard_id".

[0072] (2) Query the required shard data Get the corresponding shard data according to the shard identifier in the task; W_shard is obtained through compute_task["W_shard_id"], F_shard is obtained through compute_task["F_shard_id"], and G_shard is obtained through compute_task["G_shard_id"].

[0073] (3) Execute the calculation task. Specifically, first pass in the obtained shard data, execute the specific calculation logic, and then save the calculation result in the variable result.

[0074] (4) Store the calculation results. Specifically, first, generate a unique identifier result_id for the result in the format of "result_{shard index}" to identify the corresponding calculation result; then, use dht.set(result_id, result) to store the calculation result in the DHT; finally, return the calculation result result for subsequent use.

[0075] The calculation manager performs specific calculation operations based on the calculation task context and the pulled data shards. The execution process of the calculation task includes matrix multiplication and addition operations to generate the final calculation result. The calculation manager can dynamically adjust the allocation of computing resources according to the needs of the calculation task to improve computing efficiency and resource utilization. The torch.matmul() function is used to calculate the matrix multiplication of the shards.

[0076] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0077] In one embodiment, a distributed LoRA offloading computing system is provided, the system comprising: a top-level scheduler, a storage pool, and a computing pool, the computing pool comprising a plurality of computing managers, and the storage pool comprising a plurality of storage managers.

[0078] The top-level scheduler is used to initialize the Kademlia DHT and communication protocol used to manage shard storage and inter-node communication respectively; each Peer node is added to the DHT to form a distributed network; it is also used to shard the pre-trained model weight matrix W, LoRA matrices F and G, and store the obtained shards in the corresponding storage manager, and manage the shard information through the DHT; it is also used to generate the computing task context according to the calculation formula, and push the computing task context to the corresponding computing manager; the Peer node is a logical process or physical node with memory resources and computing resources; DHT is used to provide distributed resource management services for storage pools and computing pools, and each Peer uses DHT to manage shard storage.

[0079] The storage manager is a peer node used for storage offloading tasks.

[0080] The compute manager is a peer node used to compute offload tasks. It is used to receive the compute task context pushed by the top-level scheduler, and dynamically query the required storage information based on the compute task context. It pulls the required data shards from the storage manager through the communication protocol, executes the compute tasks, and sends the compute results of the tasks to the summary node through the communication protocol. It performs aggregation and reduction operations on the results to obtain the final compute results, and returns the final compute results to the user or stores them in the specified storage location.

[0081] In one of the embodiments, each node of the storage pool maintains the storage of the shards through a dictionary, and the key-value pairs of the dictionary correspond to the identifier of the shard and the value of the matrix respectively.

[0082] The working principle of the distributed LoRA offloading operation system is as follows Figure 7 shown.

[0083] For the specific limitations of the distributed LoRA unloading operation system, please refer to the limitations of the distributed LoRA unloading operation method above, which will not be repeated here. Each module in the above-mentioned distributed LoRA unloading operation system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0084] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A distributed LoRA offloading operation method, characterized in that: The method comprises: Initialize Kademlia DHT and communication protocol for managing shard storage and inter-node communication respectively; add each peer node to DHT to form a distributed network; the peer node is a logical process or physical node with memory resources and computing resources; the DHT is used to provide distributed resource management services for storage pools and computing pools, and each peer uses DHT to manage shard storage; The pre-trained model weight matrix W, LoRA matrix F and G are sharded, and the obtained shards are stored in the corresponding storage manager, and the shard information is managed through DHT; the storage manager is a Peer node for storing offload tasks; Generate a computing task context based on the computing formula; The computing manager dynamically queries the required storage information according to the computing task context, pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager; the computing manager is a peer node used for computing offload tasks; The obtained task calculation results are aggregated through the collective communication protocol to obtain the final calculation result; The final calculation result is returned to the user or stored in a specified storage location.

2. The distributed LoRA offloading operation method according to claim 1, characterized in that: The collective communication protocol is used for point-to-point or collective communication between nodes, supporting sending, receiving, broadcasting, aggregation and reduction operations; Kademlia DHT is used to provide efficient distributed hash table functions and support the insertion, query and deletion operations of shards.

3. The distributed LoRA offloading operation method according to claim 2, characterized in that: The collective communication refers to supporting one-way or two-way communication between a group of nodes, including: one-to-many communication, many-to-one communication, many-to-one communication and convergence.

4. The distributed LoRA offloading operation method according to claim 1, characterized in that: The pre-trained model weight matrix W, LoRA matrix F and G are sharded, and the obtained shards are stored in the corresponding storage manager. The shard information is managed through DHT, including: Decompose the pre-trained model weight matrix W, LoRA matrix F and G into sub-matrices according to row index, column index or custom method, and take each sub-matrix as a slice; the identifier of each slice corresponds to the concatenated string of the matrix name and index, and the value of the slice is a tensor type, which can be stored independently in the memory of different nodes; Store each shard in the corresponding storage manager according to its identifier. If fault-tolerant storage is required, serialize it through pickle and store it in the file system of each node. Shard information is managed through DHT, supporting the insertion, query and deletion of shards.

5. The distributed LoRA offloading operation method according to claim 1, characterized in that: Generates a computing task context based on the computing formula, including: Generate a computing task context according to the computing formula; the computing task context includes the required shard identifier and the computing formula; the computing task context is used to guide the computing manager to dynamically query the required storage information when executing the computing task, and pull the required data shards through the communication protocol; the computing formula is: in, W is the pre-trained model weight matrix, To calculate the task context, F and G are LoRA matrices.

6. The distributed LoRA offloading operation method according to claim 1, characterized in that: The computational task execution process includes: matrix multiplication and addition operations.

7. The distributed LoRA offloading operation method according to claim 1, characterized in that: The task calculation results are aggregated through the collective communication protocol to obtain the final calculation results, including: The calculation manager sends the obtained task calculation results to the summary node through the collective communication protocol, performs aggregation and reduction operations on the task calculation results, and obtains the final calculation results.

8. The distributed LoRA offloading operation method according to claim 1, characterized in that: The computing manager dynamically queries the required storage information according to the computing task context, pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager, including: The computing manager dynamically queries the required storage information according to the computing task context and the shard index of the pre-trained model weight matrix W, LoRA matrix F and G, and pulls the required data shards from the storage manager through the communication protocol; the data shard query is managed through DHT, which supports the insertion, query and deletion of shards; The computing manager pulls the required data shards from the storage manager through a communication protocol and executes computing tasks on the computing manager.

9. A distributed LoRA offloading computing system, characterized in that: The system includes: a top-level scheduler, a storage pool and a computing pool, wherein the computing pool includes a plurality of computing managers, and the storage pool includes a plurality of storage managers; The top-level scheduler is used to initialize Kademlia DHT and communication protocol for managing shard storage and inter-node communication respectively; add each Peer node to DHT to form a distributed network; and is also used to shard the pre-trained model weight matrix W, LoRA matrix F and G, and store the obtained shards in the corresponding storage manager, and manage the shard information through DHT; and is also used to generate a computing task context according to a calculation formula, and push the computing task context to the corresponding computing manager; the Peer node is a logical process or physical node with memory resources and computing resources; the DHT is used to provide distributed resource management services for storage pools and computing pools, and each Peer uses DHT to manage shard storage; The storage manager is a Peer node used to store offload tasks; The computing manager is a peer node used to compute offload tasks, and is used to receive the computing task context pushed by the top-level scheduler, and dynamically query the required storage information based on the computing task context, pull the required data shards from the storage manager through the communication protocol, and execute the computing tasks, and send the task computing results to the summary node through the communication protocol, perform result aggregation and reduction operations, obtain the final computing results, and return the final computing results to the user or store them in a specified storage location.

10. The distributed LoRA offloading computing system according to claim 9, characterized in that: Each node of the storage pool maintains the storage of the shards through a dictionary, and the key-value pairs of the dictionary correspond to the identifier of the shard and the value of the matrix respectively.

Images