A distributed loRA offloading operation method and system

Abstract

The application relates to a distributed LoRA offloading operation method and system, which comprises the following steps: initializing Kademlia DHT and a communication protocol respectively used for managing fragmented storage and inter-node communication; fragmenting a pre-trained model weight matrix W, a LoRA matrix F and G, and storing the fragments into corresponding storage managers, and managing the fragment information through the DHT; generating a calculation task context according to a calculation formula; a calculation manager dynamically inquires required storage information according to the calculation task context, pulls the required data fragments from the storage managers through the communication protocol, and executes the calculation task on the calculation manager; and the obtained task calculation results are summarized through a set communication protocol, and the final calculation results are returned to a user or stored into a specified storage position. The method offloads the memory opening pressure of a single node by using remote resources, and supports PEFT operation of nodes in a resource-limited environment.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a distributed LoRA offloading operation method and system. Background Art

[0002] Large models generally use pre-training and fine-tuning to support downstream tasks. The pre-training phase employs unsupervised learning, using an autoregressive model to predict the next token. Parameter training can utilize publicly available datasets from the internet. After pre-training, to adapt to downstream tasks, large models need to further optimize parameters using task-specific datasets to improve task-specific performance.

[0003] Currently, parameter-efficient fine-tuning (PEFT) based on Low-Rank Adaptation (LoRA) is more common, such as Figure 1 As shown, PEFT fixes the pre-trained model weight matrix W (size N*M) and adds a low-order matrix factorization F*G (size N*d and d*M, respectively). The algebraic sum of the two, W + F*G, is used to replace the model weights. F and G are then learned through task dataset optimization. Because d is much smaller than N or M, the computational complexity of model training is reduced. However, LoRA operations require large-scale matrix multiplications. Since the dimensions N and M are still much higher than d, LoRA's memory overhead still puts pressure on the HBM memory of the accelerator on a single node, urgently requiring offloading. Summary of the Invention

[0004] Based on this, it is necessary to provide a distributed LoRA offloading method and system to address the above technical issues. This method can use remote resources to offload the memory overhead pressure of a single node and support PEFT operations of nodes in resource-constrained environments.

[0005] A distributed LoRA offloading operation method, the method comprising:

[0006] Initialize the Kademlia DHT and communication protocol for managing shard storage and inter-node communication, respectively. Add each peer node to the DHT to form a distributed network. A peer node is a logical process or physical node with memory and computing resources. DHT is used to provide distributed resource management services for storage and computing pools, and each peer uses DHT to manage shard storage.

[0007] The pre-trained model weight matrix W, LoRA matrix F and G are sharded, and the obtained shards are stored in the corresponding storage manager, and the shard information is managed through DHT; the storage manager is a peer node used to store offload tasks.

[0008] Generates a calculation task context based on the calculation formula.

[0009] The computing manager dynamically queries the required storage information based on the computing task context, pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager; the computing manager is a peer node used to compute offload tasks.

[0010] The obtained task calculation results are aggregated through the collective communication protocol to obtain the final calculation result.

[0011] The final calculation result is returned to the user or stored in a specified storage location.

[0012] In one embodiment, a collective communication protocol is used for point-to-point or collective communication between nodes, supporting send, receive, broadcast, aggregate, and reduce operations.

[0013] Kademlia DHT is used to provide efficient distributed hash table functions, supporting shard insertion, query, and deletion operations.

[0014] In one embodiment, collective communication refers to supporting one-way or two-way communication between a group of nodes, including one-to-many communication, many-to-one communication, many-to-one communication, and convergence.

[0015] In one embodiment, the pre-trained model weight matrix W, LoRA matrices F and G are sharded, and the obtained shards are stored in corresponding storage managers, and the shard information is managed through DHT, including:

[0016] Decompose the pre-trained model weight matrix W and LoRA matrices F and G into sub-matrices according to row index, column index, or custom method, and treat each sub-matrix as a shard. The identifier of each shard corresponds to the concatenation of the matrix name and index. The shard value is a tensor type and can be stored independently in the memory of different nodes.

[0017] Each shard is stored in the corresponding storage manager according to its identifier. If fault-tolerant storage is required, it is serialized and stored in the file system of each node through pickle.

[0018] Shard information is managed through DHT, supporting the insertion, query and deletion of shards.

[0019] In one embodiment, generating a computing task context according to a computing formula includes:

[0020] The computing task context is generated based on the calculation formula. The computing task context contains the required shard identifier and calculation formula. The computing task context is used to guide the computing manager to dynamically query the required storage information when executing the computing task and pull the required data shards through the communication protocol. The calculation formula is:

[0021]

[0022] in, W is the pre-trained model weight matrix, To calculate the task context, F and G are LoRA matrices.

[0023] In one embodiment, the computing task execution process includes: matrix multiplication and addition operations.

[0024] In one embodiment, the obtained task calculation results are aggregated through a collective communication protocol to obtain a final calculation result, including:

[0025] The calculation manager sends the obtained task calculation results to the summary node through the collective communication protocol, performs aggregation and reduction operations on the task calculation results, and obtains the final calculation results.

[0026] In one embodiment, a computing manager dynamically queries required storage information based on the computing task context, pulls required data shards from the storage manager via a communication protocol, and executes the computing task on the computing manager, including:

[0027] The computing manager dynamically queries the required storage information based on the computing task context and the shard index of the pre-trained model weight matrix W, LoRA matrix F and G, and pulls the required data shards from the storage manager through the communication protocol; the data shard query is managed through DHT, which supports shard insertion, query and deletion operations.

[0028] The compute manager pulls the required data shards from the storage manager through a communication protocol and executes the computing tasks on the compute manager.

[0029] A distributed LoRA offloading computing system includes: a top-level scheduler, a storage pool, and a computing pool. The computing pool includes several computing managers, and the storage pool includes several storage managers.

[0030] The top-level scheduler is used to initialize the Kademlia DHT and communication protocol used to manage shard storage and inter-node communication respectively; it adds each peer node to the DHT to form a distributed network; it is also used to shard the pre-trained model weight matrix W, LoRA matrices F and G, and store the resulting shards in the corresponding storage manager, and manage the shard information through the DHT; it is also used to generate the computing task context according to the calculation formula and push the computing task context to the corresponding computing manager; the peer node is a logical process or physical node with memory resources and computing resources; the DHT is used to provide distributed resource management services for the storage pool and computing pool, and each peer uses the DHT to manage shard storage.

[0031] The storage manager is a peer node used for storage offloading tasks.

[0032] The calculation manager is a peer node used to calculate offload tasks. It is used to receive the calculation task context pushed by the top-level scheduler, and dynamically query the required storage information based on the calculation task context. It pulls the required data shards from the storage manager through the communication protocol, executes the calculation task, and sends the task calculation results to the summary node through the communication protocol to perform aggregation and reduction operations on the results to obtain the final calculation results, and returns the final calculation results to the user or stores them in the specified storage location.

[0033] In one embodiment, each node of the storage pool maintains the storage of the shards through a dictionary, and the key-value pairs of the dictionary correspond to the identifier of the shard and the value of the matrix respectively.

[0034] The above-mentioned distributed LoRA offloading calculation method and system includes: initializing the Kademlia DHT and communication protocol for managing shard storage and inter-node communication, respectively; sharding the pre-trained model weight matrix W, LoRA matrices F and G, and storing the shards in the corresponding storage manager, and managing the shard information through the DHT; generating a calculation task context based on a calculation formula; the calculation manager dynamically queries the required storage information based on the calculation task context, pulls the required data shards from the storage manager through the communication protocol, and executes the calculation task on the calculation manager; and aggregating the obtained task calculation results through the collective communication protocol, returning the final calculation results to the user or storing them in a designated storage location. This method utilizes remote resources to offload the memory overhead pressure of a single node, supporting PEFT calculations of nodes in resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the LoRA operation process;

[0036] Figure 2Schematic diagram of a distributed LoRA offloading operation method in one embodiment;

[0037] Figure 3 A distributed initialization flow chart in another embodiment;

[0038] Figure 4 This is a workflow diagram of the LoRA top-level scheduler in another embodiment;

[0039] Figure 5 A flowchart of shard storage in another embodiment;

[0040] Figure 6 A flowchart of executing a computing task on a computing manager in another embodiment;

[0041] Figure 7 Schematic diagram of the working principle of a distributed LoRA offloading operation system in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] The distributed LoRA offloading operation method proposed in this application offloads LoRA calculation and storage, that is, the LoRA storage task is only responsible for offloading the storage requirements of a single node. In view of the memory and computing resource limitations of a single node, the pre-trained matrix W, LoRA matrix F, and G are stored in a group of decentralized nodes through sharding; LoRA calculation generates calculation tasks according to the calculation formula, and dynamically queries the required storage information, pulls the required data shards through the communication protocol, and completes the calculation.

[0044] In one embodiment, Figure 2 As shown, a distributed LoRA offloading operation method is provided, which includes the following steps:

[0045] Step 100: Initialize the Kademlia DHT and communication protocol for managing shard storage and inter-node communication respectively; add each peer node to the DHT to form a distributed network; a peer node is a logical process or physical node with memory resources and computing resources; the DHT is used to provide distributed resource management services for the storage pool and computing pool, and each peer uses the DHT to manage shard storage.

[0046] Specifically, a peer is a logical process or physical node with certain memory resources and computing resources; it is a node responsible for computing and storing offloaded tasks.

[0047] Initialize Kademlia DHT and communication protocol to manage shard storage and inter-node communication. The initialization process is as follows Figure 3 As shown. First, initialize a DHT server and listen to the specified port; then, define the peer node list and guide each peer node (bootstrap) to the DHT network. Each peer node joins the DHT to form a distributed network. Kademlia DHT provides efficient distributed hash table functions and supports the insertion, query and deletion of shards. The collective communication protocol is used for point-to-point and collective communication between nodes, supporting sending, receiving, broadcasting, aggregation and reduction operations. The initialization process is as follows Figure 3 shown.

[0048] Distributed Hash Table (DHT): Provides distributed resource management services for storage and computation pools. Each peer uses the DHT to manage shard storage, including inserting, querying, and deleting shards. In the storage pool, each node maintains shard storage using a dictionary whose key-value pairs correspond to shard identifiers and matrix values. The computation pool is responsible for PEFT, executing distributed LoRA computations.

[0049] Step 102: Slice the pre-trained model weight matrix W, LoRA matrices F and G, and store the obtained shards in the corresponding storage manager, and manage the shard information through DHT; the storage manager is a peer node used to store offload tasks.

[0050] Specifically, the pre-trained matrix W and the LoRA matrices F and G are decomposed into sub-matrices according to row index, column index, or a custom method. Each sub-matrix is ​​called a shard. The identifier of each shard corresponds to the concatenation of the matrix name and index string. The shard value is a tensor type and can be stored independently in the memory of different nodes.

[0051] Step 104: Generate a computing task context according to the computing formula.

[0052] Specifically, the generation of computing task context can be customized according to specific computing requirements and shard storage conditions.

[0053] Step 106: The computing manager dynamically queries the required storage information according to the computing task context, pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager; the computing manager is a peer node used to compute offload tasks.

[0054] Specifically, compute tasks are executed on the compute manager. The compute manager receives tasks pushed by the top-level scheduler and executes the compute tasks for the shards it is responsible for, ensuring that all sharded compute tasks are completed efficiently. Based on the context of the compute task, it dynamically queries the required storage information, pulls the required data shards through the communication protocol, and executes the compute task on the compute manager.

[0055] LoRA top-level scheduling: Responsible for managing and coordinating sharded computing tasks. It pushes tasks to various compute managers, which perform local calculations. The top-level scheduler then aggregates the results from each manager. Based on the context of the compute task, the top-level scheduler dynamically queries the required storage information, pulls the required data shards via communication protocols, executes the compute task on the compute manager, and finally aggregates the results through collective communication.

[0056] The LoRA Top Level Scheduler is used for distributed computing of large parameter matrices (such as W, F, G). It stores large matrices in shards, distributes the computational tasks to multiple computing nodes, and finally aggregates the computational results. The scheduler utilizes a distributed hash table (DHT) and communication protocol to achieve efficient distributed computing.

[0057] (1) Scheduling: First, W, F, and G are sharded according to the shard size (shard_size) and stored in the DHT; then, a computing task is generated for each shard and pushed to the computing manager; after all computing tasks are completed, the computing results are pulled from the computing manager; finally, all the results are aggregated to obtain the final result.

[0058] (2) Push task process: Get the corresponding computing node ID through DHT and use the communication module to send the task to the computing node.

[0059] (3) Pull result process: Get the node ID where the result is located through DHT, and use the communication module to receive the calculation result from the node.

[0060] The workflow of the LoRA top-level scheduler is as follows Figure 4 shown.

[0061] Step 108: Summarize the obtained task calculation results through the collective communication protocol to obtain the final calculation result; return the final calculation result to the user or store it in a designated storage location.

[0062] Specifically, computation results are aggregated through collective communication. The computation manager sends the computation results to the aggregation node via a collective communication protocol, where they are aggregated and reduced. This aggregation process can utilize various collective communication methods, such as broadcast, aggregate, and reduce. Through result aggregation, the final computation result is obtained and returned to the user or stored in a designated storage location.

[0063] The above-mentioned distributed LoRA offloading operation method includes: initializing the Kademlia DHT and communication protocol for managing shard storage and inter-node communication, respectively; sharding the pre-trained model weight matrix W, LoRA matrices F and G, and storing the shards in the corresponding storage manager, and managing the shard information through the DHT; generating a computation task context based on a calculation formula; the computation manager dynamically queries the required storage information based on the computation task context, pulls the required data shards from the storage manager through the communication protocol, and executes the computation task on the computation manager; and aggregating the obtained task computation results through the collective communication protocol, returning the final computation result to the user or storing it in a designated storage location. This method utilizes remote resources to offload the memory overhead pressure of a single node, supporting PEFT operations on nodes in resource-constrained environments.

[0064] In one embodiment, a collective communication protocol is used for point-to-point or collective communication between nodes, supporting send, receive, broadcast, aggregate, and reduce operations.

[0065] Kademlia DHT is used to provide efficient distributed hash table functions, supporting shard insertion, query, and deletion operations.

[0066] Specifically, point-to-point communication supports send / receive synchronous or asynchronous communication.

[0067] In one embodiment, collective communication refers to supporting one-way or two-way communication between a group of nodes, including one-to-many communication, many-to-one communication, many-to-one communication, and convergence.

[0068] In one embodiment, step 102 includes: decomposing the pre-trained model weight matrix W, LoRA matrix F and G into sub-matrices according to row index, column index or custom method, and taking each sub-matrix as a shard; wherein the identifier of each shard corresponds to the concatenation string of the matrix name and index, the value of the shard is a tensor type, and can be independently stored in the memory of different nodes; storing each shard according to its identifier in the corresponding storage manager, and if fault-tolerant storage is required, storing it in the file system of each node through pickle serialization; managing the shard information through DHT, supporting the insertion, query and deletion operations of the shard.

[0069] Specifically, the process of shard storage is as follows Figure 5 As shown in the figure, shard storage divides a given matrix into multiple smaller blocks (shards) according to the specified shard size, associates these shards with unique identifiers, and then records the location of each shard (corresponding peer_id) in a distributed hash table (DHT). The specific steps include:

[0070] Step 1: Obtain the matrix data to be sharded (matrix), the matrix name (matrix_name), the number of rows in each shard (assuming the matrix is ​​sharded by row) (shard_size), and the identifier of the node responsible for storing the shards (peer_id). The matrix name (matrix_name) is used to generate a unique identifier for each shard.

[0071] Step 2: Create an empty dictionary shards to store the shard identifier and the corresponding shard data.

[0072] Step 3: Divide the number of rows in the matrix by the shard size to get the required number of shards (num_shards).

[0073] Step 4: Loop number: from 0 to num_shards - 1, process each shard one by one, and extract the shard:

[0074] Step 4.1: Extract sub-matrices (slices) of the matrix based on the slice size.

[0075] Step 4.2: Generate shard identifier: Generate a unique shard_id using the matrix name and shard index, such as "W_0", "W_1", etc.

[0076] Step 4.3: Update shard mapping: Store the shard data in the shards dictionary, with the key being shard_id and the value being shard.

[0077] Step 4.4: Select DHT storage node peer_id: Select the storage node peer_id of the shard according to the DHT storage rules or user-defined storage mapping rules.

[0078] Step 4.5: Record the correspondence between the shard identifier and the storage node peer_id in the distributed hash table.

[0079] In one embodiment, step 104 includes: generating a computing task context according to a computing formula; the computing task context includes a required shard identifier and a computing formula; the computing task context is used to guide the computing manager to dynamically query required storage information when executing the computing task, and to pull required data shards through a communication protocol; the computing formula is:

[0080] (1)

[0081] in, W is the pre-trained model weight matrix, To calculate the task context, F and G are LoRA matrices.

[0082] Specifically, the computing task context is generated according to the calculation formula (1). The computing task context contains the required shard identifier and the calculation formula. The computing task context is used to guide the computing manager to dynamically query the required storage information when executing the computing task and pull the required data shards through the communication protocol. The generation of the computing task context can be customized according to the specific computing requirements and shard storage conditions. The generated context data dictionary is:

[0083] {"W_shard_id": W_shard_id, "F_shard_id": F_shard_id, "G_shard_id": G_shard_id}

[0084] Among them, W_shard_id is the pre-training model weight matrix W F_shard_id is the shard identifier of the LoRA matrix F, and G_shard_id is the shard identifier of the LoRA matrix G.

[0085] In one embodiment, the computing task execution process includes: matrix multiplication and addition operations.

[0086] In one embodiment, step 108 includes: the calculation manager sends the obtained task calculation results to the summary node through the collective communication protocol, performs aggregation and reduction operations on the task calculation results, and obtains the final calculation results.

[0087] In one embodiment, step 106 includes: the computing manager dynamically queries the required storage information based on the computing task context and the shard index of the pre-trained model weight matrix W, the LoRA matrix F and G, and pulls the required data shards from the storage manager through the communication protocol; wherein the data shard query is managed through the DHT, supporting the insertion, query and deletion operations of the shards; the computing manager pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager.

[0088] Specifically, for data shard queries, the compute task context dynamically queries the required storage information based on the shard indexes of W, F, and G, and retrieves the required data shards via the communication protocol. Data shard queries are managed via the Distributed Hierarchy Transport (DHT), supporting shard insertion, query, and deletion. The compute manager uses the communication protocol to retrieve the required data shards from the storage manager for local computation. The data shard query process is dynamic and can be adjusted based on the needs of the computation task.

[0089] The specific steps to obtain a shard include: Select the peer_id corresponding to the shard ID shard_id:

[0090] peer_id = dht.get(shard_id)

[0091] Where peer_id is the peer identifier corresponding to the shard returned by dht.get(), and shard_id is the shard identifier. dht.get() is the peer identifier used to pull the corresponding shard from the DHT.

[0092] Pull data distribution based on shard_id and peer_id:

[0093] shard = receive(peer_id, shard_id)

[0094] shard is the data shard to be pulled, and receive() pulls the shard with the identifier shard_id from the distribution node corresponding to peer_id and stores it in the shard.

[0095] The process of executing computing tasks on the computing manager is as follows Figure 6 The steps to perform a calculation task are as follows:

[0096] (1) Initialization: DHT reference of the distributed hash table and compute_task. Compute_task contains a dictionary of computing task information, including shard identifiers such as "W_shard_id", "F_shard_id", and "G_shard_id".

[0097] (2) Query the required shard data

[0098] Get the corresponding shard data based on the shard identifier in the task. W_shard is obtained through compute_task["W_shard_id"], F_shard is obtained through compute_task["F_shard_id"], and G_shard is obtained through compute_task["G_shard_id"].

[0099] (3) Execute the calculation task. Specifically, first pass in the obtained shard data, execute the specific calculation logic, and then save the calculation result in the variable result.

[0100] (4) Storing the calculation results. Specifically, first, generate a unique identifier result_id for the result in the format of "result_{shard index}" to identify the corresponding calculation result; then, use dht.set(result_id, result) to store the calculation result in the DHT; finally, return the calculation result result for subsequent use.

[0101] The compute manager performs specific computations based on the task context and the retrieved data shards. The execution of a compute task involves matrix multiplication and addition, generating the final result. The compute manager dynamically adjusts the allocation of computing resources based on the task's requirements to improve computational efficiency and resource utilization. The torch.matmul() function is used to perform matrix multiplication on the shards.

[0102] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0103] In one embodiment, a distributed LoRA offloading computing system is provided, which includes: a top-level scheduler, a storage pool, and a computing pool. The computing pool includes several computing managers, and the storage pool includes several storage managers.

[0104] The top-level scheduler is used to initialize the Kademlia DHT and communication protocol used to manage shard storage and inter-node communication respectively; it adds each peer node to the DHT to form a distributed network; it is also used to shard the pre-trained model weight matrix W, LoRA matrices F and G, and store the resulting shards in the corresponding storage manager, and manage the shard information through the DHT; it is also used to generate the computing task context according to the calculation formula and push the computing task context to the corresponding computing manager; the peer node is a logical process or physical node with memory resources and computing resources; the DHT is used to provide distributed resource management services for the storage pool and computing pool, and each peer uses the DHT to manage shard storage.

[0105] The storage manager is a peer node used for storage offloading tasks.

[0106] The calculation manager is a peer node used to calculate offload tasks. It is used to receive the calculation task context pushed by the top-level scheduler, and dynamically query the required storage information based on the calculation task context. It pulls the required data shards from the storage manager through the communication protocol, executes the calculation task, and sends the task calculation results to the summary node through the communication protocol to perform aggregation and reduction operations on the results to obtain the final calculation results, and returns the final calculation results to the user or stores them in the specified storage location.

[0107] In one embodiment, each node of the storage pool maintains the storage of the shards through a dictionary, and the key-value pairs of the dictionary correspond to the identifier of the shard and the value of the matrix respectively.

[0108] The working principle of the distributed LoRA offloading operation system is as follows Figure 7 shown.

[0109] For the specific definition of the distributed LoRA offloading operation system, please refer to the definition of the distributed LoRA offloading operation method above, which will not be repeated here. Each module in the above-mentioned distributed LoRA offloading operation system can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0110] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A distributed LoRA offloading operation method, characterized in that: The method comprises: Initialize the Kademlia DHT and communication protocol for managing shard storage and inter-node communication, respectively; add each peer node to the DHT to form a distributed network; the peer node is a logical process or physical node with memory and computing resources; the DHT is used to provide distributed resource management services for the storage pool and computing pool, and each peer uses the DHT to manage shard storage; The pre-trained model weight matrix W, LoRA matrix F and G are sharded, and the obtained shards are stored in the corresponding storage manager, and the shard information is managed through DHT; the storage manager is a peer node for storing offload tasks; Generate calculation task context based on calculation formula; The computing manager dynamically queries the required storage information according to the computing task context, pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager; the computing manager is a peer node used for computing offload tasks; The obtained task calculation results are aggregated through the collective communication protocol to obtain the final calculation result; The final calculation result is returned to the user or stored in a designated storage location.

2. The distributed LoRA offloading operation method according to claim 1, characterized in that: The collective communication protocol is used for point-to-point or collective communication between nodes, supporting send, receive, broadcast, aggregate and reduce operations; Kademlia DHT is used to provide efficient distributed hash table functions, supporting shard insertion, query, and deletion operations.

3. The distributed LoRA offloading operation method according to claim 2, characterized in that: The collective communication refers to supporting one-way or two-way communication between a group of nodes, including one-to-many communication, many-to-one communication, many-to-one communication and convergence.

4. The distributed LoRA offloading operation method according to claim 1, characterized in that: The pre-trained model weight matrix W, LoRA matrix F and G are sharded and stored in the corresponding storage manager. The shard information is managed through DHT, including: Decompose the pre-trained model weight matrix W and the LoRA matrices F and G into sub-matrices according to row index, column index, or custom method, and treat each sub-matrix as a slice. The identifier of each slice corresponds to the concatenation of the matrix name and index. The slice value is a tensor type and can be stored independently in the memory of different nodes. Store each shard in the corresponding storage manager according to its identifier. If fault-tolerant storage is required, serialize it through pickle and store it in the file system of each node. Shard information is managed through DHT, supporting the insertion, query and deletion of shards.

5. The distributed LoRA offloading operation method according to claim 1, characterized in that: Generates a calculation task context based on the calculation formula, including: A computing task context is generated based on the computing formula; the computing task context includes the required shard identifier and the computing formula; the computing task context is used to guide the computing manager to dynamically query the required storage information when executing the computing task and to pull the required data shards through the communication protocol; the computing formula is: in, W is the pre-trained model weight matrix, To calculate the task context, F and G are LoRA matrices.

6. The distributed LoRA offloading operation method according to claim 1, characterized in that: The computational task execution process includes: matrix multiplication and addition operations.

7. The distributed LoRA offloading operation method according to claim 1, characterized in that: The obtained task calculation results are aggregated through the collective communication protocol to obtain the final calculation results, including: The calculation manager sends the obtained task calculation results to the summary node through the collective communication protocol, performs aggregation and reduction operations on the task calculation results, and obtains the final calculation results.

8. The distributed LoRA offloading operation method according to claim 1, characterized in that: The computing manager dynamically queries the required storage information according to the computing task context, pulls the required data shards from the storage manager through the communication protocol, and executes the computing task on the computing manager, including: The computation manager dynamically queries the required storage information based on the computation task context and the shard indexes of the pre-trained model weight matrix W, LoRA matrix F, and G, and pulls the required data shards from the storage manager through the communication protocol. Data shard queries are managed through the DHT, which supports shard insertion, query, and deletion operations. The computing manager pulls the required data shards from the storage manager through a communication protocol and executes computing tasks on the computing manager.

9. A distributed LoRA offloading computing system, characterized in that: The system includes: a top-level scheduler, a storage pool, and a computing pool, wherein the computing pool includes a plurality of computing managers, and the storage pool includes a plurality of storage managers; The top-level scheduler is used to initialize the Kademlia DHT and communication protocol for managing shard storage and inter-node communication, respectively; add each peer node to the DHT to form a distributed network; and is also used to shard the pre-trained model weight matrix W and LoRA matrices F and G, and store the resulting shards in the corresponding storage manager, managing the shard information through the DHT; and is also used to generate a computing task context based on the calculation formula and push the computing task context to the corresponding computing manager. The peer node is a logical process or physical node with memory resources and computing resources. The DHT is used to provide distributed resource management services for the storage pool and computing pool, and each peer uses the DHT to manage shard storage. The storage manager is a peer node for storing offload tasks; The computing manager is a peer node used to compute offload tasks, and is used to receive the computing task context pushed by the top-level scheduler, and dynamically query the required storage information based on the computing task context, pull the required data shards from the storage manager through the communication protocol, and execute the computing task, and send the task computing results to the summary node through the communication protocol, perform aggregation and reduction operations on the results, obtain the final computing results, and return the final computing results to the user or store them in a designated storage location.

10. The distributed LoRA offloading operation system according to claim 9, characterized in that: Each node in the storage pool maintains the storage of the shards through a dictionary, and the key-value pairs of the dictionary correspond to the identifier of the shard and the value of the matrix respectively.

Images