Multi-node distribution method and device for large model

Abstract

The embodiment of the specification provides a multi-node distribution method and device for a large model. The large model corresponds to a plurality of model files. A target model file is pre-divided into a plurality of file blocks, and each file block is pre-divided into a plurality of shards. The method comprises the following steps: a target node sends a download request to a scheduler, wherein the target model file is indicated; the scheduler selects a plurality of candidate nodes to join a candidate parent node set from candidate nodes in a cluster based on at least index values of state indicators of the candidate nodes and global distribution information of the shards on the candidate nodes, the candidate nodes having downloaded at least one shard, and the candidate nodes and the target node at least satisfying topology connectivity and not forming a loop; the scheduler determines a target file block to be preferentially downloaded by the target node from the plurality of file blocks; the scheduler sends the candidate parent node set and a block identifier of the target file block to the target node; and the target node preferentially downloads shards in the target file block from nodes in the candidate parent node set.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computers, and more particularly to a method and apparatus for multi-node distribution of large models. Background Technology

[0002] Currently, large-scale models are deployed extensively in various practical applications. To obtain high-quality model output results, large models often have tens of billions of parameters, and the overall model data volume can reach hundreds of gigabytes. At the same time, the actual load changes of inference services that rely on large models are highly dynamic. To ensure service quality, it is usually necessary to distribute and deploy model instances across multiple nodes in a cluster.

[0003] In existing technologies, when a model instance on a node is initialized, it typically needs to load all model parameters from a remote storage system. This process often takes tens of minutes or even longer, making it difficult for the node to accept service requests within a short period. The resulting high latency during cold starts severely impacts the cluster's responsiveness and has become a key bottleneck for the service quality of large models.

[0004] Therefore, building an efficient and low-latency model distribution mechanism is of great significance for alleviating model loading latency and improving the overall performance of the cluster. Summary of the Invention

[0005] This specification describes one or more embodiments of a multi-node distribution method and apparatus for large models, which can achieve efficient and low-latency model distribution.

[0006] Firstly, a multi-node distribution method for large models is provided. The method is executed by a scheduler, where the large model corresponds to several model files. The target model file is pre-splitted into multiple file blocks, and each file block is pre-splitted into multiple fragments. The method includes:

[0007] Receive a download request sent by the target node, which indicates the target model file;

[0008] Based at least on the status index values ​​of each candidate node in the cluster and the global distribution information of each shard on each candidate node, several candidate nodes are selected from each candidate node to be added to the candidate parent node set. Each candidate node has downloaded at least one shard and is at least topologically connected to the target node without forming a loop.

[0009] From the plurality of file blocks, determine the target file block that the target node will prioritize downloading;

[0010] The candidate parent node set and the block identifier of the target file block are sent to the target node, so that the target node downloads the fragments in the target file block from the nodes in the candidate parent node set first.

[0011] In one possible implementation, several candidate nodes are selected from the candidate nodes to add to the candidate parent node set, including:

[0012] Each candidate node is scored in multiple dimensions, including an indicator value score based on the indicator value and a shard coverage score based on the global distribution information; wherein, the shard coverage score is calculated based on the number of intersections between the shards already held by the candidate node and the remaining undownloaded shards of the target node.

[0013] The top K candidate nodes are selected in descending order of their scores and added to the candidate parent node set.

[0014] Furthermore, the index value score includes at least one of the following:

[0015] Load quality score is used to characterize whether candidate nodes are available, whether they are saturated for a long time, or whether concurrency exceeds the limit;

[0016] Internet Data Center (IDC) affinity score is used to characterize whether candidate nodes and target nodes belong to the same IDC;

[0017] Geographical affinity score, used to characterize the geographical distance between candidate nodes and target nodes;

[0018] Node type score is used to characterize whether a candidate node belongs to the seed node category.

[0019] In one possible implementation, determining the target file block that the target node prioritizes for download includes:

[0020] Obtain the number of nodes corresponding to each of the multiple file blocks as the target file blocks to be downloaded first;

[0021] Select the file block with the fewest corresponding nodes from the multiple file blocks, and determine it as the target file block that the target node will prioritize downloading.

[0022] In one possible implementation, the method further includes:

[0023] Receive a rescheduling request sent by the target node, the rescheduling request being used to request the reallocation of the candidate parent node set;

[0024] Based at least on the update index values ​​of the status indicators of each candidate node in the cluster, the updated global distribution information of each shard on each candidate node, and the target file block, several candidate nodes are reselected from each candidate node as the updated candidate parent node set.

[0025] The updated set of candidate parent nodes is sent to the target node.

[0026] In one possible implementation, the method further includes:

[0027] Receive information on downloaded shards reported by each node in the cluster and dynamically update the global shard distribution information;

[0028] The updated global shard distribution information is sent to each node in the cluster so that each node can select the shard to download based on the global shard distribution information.

[0029] Secondly, a multi-node distribution method for large models is provided. The method is executed by the target node in the cluster. The large model corresponds to several model files. The target model files are pre-splitted into multiple file blocks, and each file block is pre-splitted into multiple fragments. The method includes:

[0030] Send a download request to the scheduler of the cluster, indicating the target model file;

[0031] The system receives a set of candidate parent nodes and the block identifier of the target file block that the target node prioritizes for download, sent by the scheduler. The set of candidate parent nodes is determined at least based on the index values ​​of the status indicators of each candidate node in the cluster and the global distribution information of each shard on each candidate node. Each candidate node has downloaded at least one shard and is at least topologically connected to the target node without forming a loop.

[0032] The nodes in the candidate parent node set are used to download fragments from the target file block first.

[0033] In one possible implementation, the step of preferentially downloading fragments from the target file block from nodes in the candidate parent node set includes:

[0034] Metadata for each shard on each node in the candidate parent node set is collected, and the metadata includes the storage location.

[0035] For the metadata of any collected fragment, if the fragment belongs to the target file block, its metadata is distributed to the high-priority queue; otherwise, it is distributed to the low-priority queue.

[0036] Prioritize selection from the high-priority queue to determine several target fragments to be downloaded;

[0037] For any target fragment, select its corresponding target parent node from the set of candidate parent nodes;

[0038] Based on the metadata, download the corresponding target fragments from each target parent node.

[0039] Furthermore, a number of target fragments to be downloaded are selected from the high-priority queue, including:

[0040] The high-priority queue is selected as the target queue first; when the high-priority queue is empty, the low-priority queue is selected as the target queue.

[0041] A predetermined number of fragments that can be downloaded in parallel are selected from the target queue as the target fragments.

[0042] Furthermore, several target fragments to be downloaded were identified, including:

[0043] Receive global fragmentation distribution information periodically sent by the scheduler;

[0044] Based on the latest received global sharding distribution information, several shards with higher scarcity are selected as the target shards. The scarcity is determined by the number of nodes in the cluster that hold the shard; the fewer the nodes, the higher the scarcity.

[0045] Further, the step of selecting the corresponding target parent node from the candidate parent node set for any target fragment includes:

[0046] The weights are calculated based on the current state information of each node in the candidate parent node set; wherein the current state information includes at least one of the following: the node's current idle bandwidth, the most recently smoothed bandwidth obtained by exponential moving average processing, and the load saturation determined based on the ratio of the current number of connections to the maximum number of connections;

[0047] Based on the weights, a target parent node is selected for the target shard from the set of candidate parent nodes.

[0048] Furthermore, the load saturation is determined as follows: when the ratio of the current number of connections to the maximum number of connections of a node is less than 1, the load saturation is 1; when the ratio is greater than or equal to 1, the load saturation is the reciprocal of the ratio.

[0049] Further, the step of collecting metadata for each shard from each node included in the candidate parent node set includes:

[0050] When collecting metadata from each shard, wait for a preset time window to aggregate metadata from multiple nodes for the same shard.

[0051] Furthermore, the method also includes:

[0052] The downloaded shard information is reported to the scheduler so that the scheduler updates the global shard distribution information.

[0053] In one possible implementation, the method further includes:

[0054] When it is detected that the candidate parent node set cannot complete the download of the remaining fragments, a rescheduling request is sent to the scheduler. The rescheduling request is used to request the reallocation of the candidate parent node set.

[0055] Thirdly, a multi-node distribution device for large models is provided. The device is located in a scheduler, wherein the large model corresponds to several model files, the target model file is pre-segmented into multiple file blocks, and each file block is pre-segmented into multiple fragments. The device includes:

[0056] The receiving unit is used to receive download requests sent by the target node, which indicate the target model file;

[0057] The selection unit is used to select several candidate nodes from the candidate nodes and add them to the candidate parent node set, based at least on the indicator values ​​of the status indicators of each candidate node in the cluster and the global distribution information of each shard on each candidate node. The candidate nodes have downloaded at least one shard and are at least topologically connected to the target node and will not form a loop.

[0058] The determining unit is configured to determine, from the plurality of file blocks, the target file block that the target node will prioritize downloading;

[0059] The sending unit is used to send the candidate parent node set obtained by the selection unit and the block identifier of the target file block obtained by the determination unit to the target node, so that the target node downloads the fragments in the target file block from the nodes in the candidate parent node set first.

[0060] Fourthly, a multi-node distribution device for large models is provided. The device is set on the target node in a cluster. The large model corresponds to several model files. The target model files are pre-splitted into multiple file blocks, and each file block is pre-splitted into multiple fragments. The device includes:

[0061] A sending unit is used to send a download request to the scheduler of the cluster, wherein the target model file is indicated;

[0062] The receiving unit is used to receive the set of candidate parent nodes and the block identifier of the target file block that the target node prioritizes to download, sent by the scheduler; the set of candidate parent nodes is determined at least based on the index values ​​of the status indicators of each candidate node in the cluster and the global distribution information of each shard on each candidate node, and the candidate node has downloaded at least one shard and is at least topologically connected to the target node and will not form a loop.

[0063] The download unit is used to preferentially download fragments from the target file block from the set of candidate parent nodes received by the receiving unit.

[0064] Fifthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first or second aspect.

[0065] In a sixth aspect, a computing device is provided, including a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, it implements the method of the first aspect or the second aspect.

[0066] The methods and apparatus provided in the embodiments of this specification allow for the large model to correspond to several model files. The target model file is pre-segmented into multiple file blocks, and each file block is pre-segmented into multiple fragments. These fragments serve as independently transmittable data units, enabling more flexible data scheduling, bandwidth utilization, and task parallelization. When the scheduler selects a set of candidate parent nodes for the target node, it considers not only the status indicators of each candidate node but also the global distribution information of each fragment across the candidate nodes, improving the effectiveness of nodes in the candidate parent node set from the scheduling source. Furthermore, the scheduler determines the target file blocks that the target node prioritizes for download. By rationally allocating target file blocks to different nodes, fragments can be distributed to different nodes more quickly, allowing subsequent nodes to download fragments in parallel from different sources. In summary, this achieves efficient and low-latency model distribution. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification;

[0069] Figure 2 This is a schematic diagram of the system architecture of one embodiment disclosed in this specification;

[0070] Figure 3 This is a schematic diagram of the node topology of one embodiment disclosed in this specification;

[0071] Figure 4 This diagram illustrates the interaction of a multi-node distribution method for a large model according to one embodiment.

[0072] Figure 5 This diagram illustrates a scheduling node allocating priority download target file blocks to peer nodes according to one embodiment;

[0073] Figure 6 This diagram illustrates the interaction of a multi-node distribution method for a large model according to another embodiment.

[0074] Figure 7 A schematic block diagram of a multi-node distribution apparatus for a large model is shown according to one embodiment;

[0075] Figure 8 A schematic block diagram of a multi-node distribution apparatus for a large model according to another embodiment is shown. Detailed Implementation

[0076] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0077] Figure 1This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification. This implementation scenario involves multi-node distribution of large models. It can be understood that "large model" is a general term in the field of artificial intelligence, encompassing models with a large parameter scale and generalized capabilities, covering multiple directions such as computer vision, natural language processing, multimodal computing, and scientific computing. "Large Language Model" (LLM) is a specialized subset of large models in the field of natural language processing, specializing in text understanding, generation, and reasoning tasks. Therefore, all LLMs belong to the category of large models, but large models are not limited to large language models. In this embodiment, only LLMs are used as examples for illustration. A large language model is a deep learning model pre-trained on a large-scale corpus, capable of understanding and generating natural language, and performing excellently in tasks such as text generation, dialogue, translation, and code generation. Large language models are typically built based on stacked multi-layer Transformer layers, relying on a very large parameter scale to capture complex language patterns. In this embodiment, model inference can be performed after model distribution. Model inference refers to the process of using the trained model to compute input data and generate output. The inference phase no longer updates parameters; instead, it uses learned features for prediction. For LLM, inference includes input encoding, context modeling, and text generation based on probability distributions. Inference speed, resource consumption, and scalability are key metrics in practical deployments. Model distribution is the process of transferring the trained model from the central storage node and deploying it to multiple computing nodes or devices. Distribution efficiency directly affects system startup speed, resource utilization, and the overall performance of distributed inference and multi-node collaboration; therefore, an efficient distribution strategy needs to be designed considering model size, network bandwidth, and node capabilities.

[0078] In practice, large models are typically stored as model files, which contain model parameter data. For example, safetensors are a common file format specifically used to store the weights of deep learning models. Safetensors files use JSON format to store metadata in the header, including the name, data type (dtype), and shape of each tensor, while the body stores the tensor data that constitutes the model weights. A single model file generally has a size limit; therefore, large models with a large number of parameters can be stored using multiple safetensors files. For example, a large model with 70B parameters might require 10-15 safetensors files to ensure that the size of a single file does not exceed, for example, 5GB. Assuming model A is divided into 15 safetensors files, these 15 files can be represented as: modelA-0xx-of-015.safetensors, where xx can be 01 to 15.

[0079] Model distribution is essentially the process of distributing model files. Typically, to improve model processing capabilities through parallel computing or for disaster recovery, large model files need to be distributed to multiple computing nodes or devices. In related technologies, a model file can be divided into multiple pieces, where each piece serves as the basic unit of transmission during file download connections between computing nodes. A computing node is considered to have completed downloading a model file only when it has downloaded all of its pieces.

[0080] The inventors discovered that the scheme of downloading model files by fragment by computing nodes still has insufficient efficiency. In practice, by default, each computing node downloads fragments sequentially from the central node according to the fragment order (e.g., determined by fragment number or name). This easily leads to computing nodes simultaneously downloading the same fragments that are ranked higher in the order. Consequently, these higher-ranking fragments are distributed across many nodes in the network, while lower-ranking fragments are rarely accessed, resulting in uneven data distribution and affecting the overall download speed. For example, assuming a safetensors file is divided into 12 fragments, P1-P12, computing nodes that need to download this safetensors file often start downloading from P1 and P2, causing these two fragments to be stored across multiple computing nodes, while the lower-ranking fragments P10-P12 are almost never downloaded. This method is not conducive to the rapid distribution of model files among the computing nodes that need to download them.

[0081] To improve the efficiency of model distribution, the embodiments in this specification propose dividing the model file into multiple file blocks, each containing multiple fragments. During the simultaneous download of the model file by multiple computing nodes, by considering the distribution of each fragment across the nodes and assigning priority download file blocks to each computing node, the fragments are distributed more dispersedly and quickly across multiple computing nodes.

[0082] Reference Figure 1 A large model can be stored as several model files. The target model file is pre-splitted into multiple file blocks, and each file block is pre-splitted into multiple fragments. Specifically, model files are used to store model parameters, and the target model file is any one of the several model files. A file block (Chunk) refers to an independent substructure after a model file is split. A fragment (Piece) refers to the smallest independently transferable data unit in a model file. Each fragment contains a segment of parameter or structural data of the model and can be independently scheduled, transferred, and stored between nodes. A file block contains multiple fragments. By splitting the model into file blocks and then dividing the file blocks into fragments, the system can more flexibly perform data scheduling, bandwidth utilization, and task parallelization.

[0083] It should be noted that, Figure 1 The quantitative relationships between model files and file blocks, and between file blocks and fragments, are only illustrative. The method of dividing file blocks and the method and number of fragments can be based on various preset strategies. The embodiments in this specification do not limit the above preset strategies.

[0084] Figure 2 This is a schematic diagram of the system architecture of one embodiment disclosed in this specification. (Refer to...) Figure 2Multi-node distribution of large models is executed based on a model distribution system. This system comprises a scheduler and multiple peer nodes. The scheduler, the core coordination component, allocates a set of candidate parent nodes to peer nodes based on the status indicators of each node in the cluster and the global distribution information of each shard across the nodes. Simultaneously, the scheduler also allocates priority download target file blocks to peer nodes. Peer nodes are the data transmission execution nodes in the model distribution system, capable of both downloading and uploading. As an uploader, it acts as a parent node to other peer nodes, synchronizing its own status indicator values ​​to the scheduler and providing shard upload services to other peer nodes. As a downloader, it acts as a child node to other peer nodes, sending download requests to the scheduler, receiving the candidate parent node set and the block identifier of the target file block to be downloaded by the target node, and downloading shards from the target file block prioritized by nodes in its candidate parent node set.

[0085] In the embodiments described in this specification, a peer node refers to a node participating in model distribution. Each peer node can either act as a requester (i.e., a child node) to obtain shards from other nodes, or act as a provider (i.e., a parent node) to provide the shards it already holds. Since there is no fixed master-slave relationship between peer nodes, the system can maintain good fault tolerance and horizontal scalability when network conditions change, nodes join, or load changes.

[0086] A parent node is an upstream node in the model distribution chain that provides shards for a particular peer node. A peer node typically connects to multiple parent nodes simultaneously to download different shards in parallel, thereby improving overall transmission bandwidth and download efficiency. Therefore, selecting the most suitable parent node based on information such as bandwidth, load, and scarcity is a crucial factor affecting model distribution speed, system stability, and overall performance.

[0087] A seed node is a stable parent node that is pre-deployed in the model distribution system, always online, and fully holds the task data. Its core responsibility is to provide a reliable data "seed" for the cluster in the initial stage of the task. When ordinary nodes have not yet formed sufficient shard coverage, the seed node acts as a highly available, high-bandwidth upstream source to provide shards to other nodes, thereby quickly guiding the establishment of the model distribution system topology, accelerating the diffusion of the first batch of shards, and avoiding the concentration of back-to-origin pressure on the repository or origin server.

[0088] Figure 3 This is a schematic diagram of the node topology of one embodiment disclosed in this specification. (Refer to...) Figure 3In this embodiment, the scheduler constructs a Directed Acyclic Graph (DAG) for each peer node in the same model download task (corresponding to downloading the same model file). A DAG is a graph structure with directional edges and no directed cycles. Each node in the graph represents a peer node participating in the download, and directed edges point from a parent node to a child node, indicating that the child node can download shards from that parent node. The direction of edges between nodes is determined by the propagation of nodes that have more data or completed their downloads earlier to nodes that are still downloading. The system avoids mutual waiting or circular dependencies by prohibiting the formation of cycles, such as node A depending on node B, and node B depending on node A, thereby ensuring that shards can spread layer by layer along the topology, the download process progresses unidirectionally, and eventually converges.

[0089] It's important to note that a model file is divided into multiple file blocks, and each file block is further divided into multiple fragments. A complete large model typically consists of multiple model files. A single node can initiate download tasks for multiple model files simultaneously to improve overall download efficiency. However, it's crucial to understand that each model file is independently divided into file blocks and fragments, forming its own independent DAG transmission topology. In other words, the DAG transmission topologies of different model files are independent of each other.

[0090] Figure 4 This diagram illustrates an interaction of a multi-node distribution method for a large model according to one embodiment, which can be based on... Figure 1 The implementation scenarios shown Figure 2 The system architecture shown and Figure 3 The node topology shown primarily involves the interaction between the scheduler and the target node. A large model corresponds to several model files, and the target model file is pre-splitted into multiple file blocks, each of which is further pre-splitted into multiple fragments. For example... Figure 4 As shown, the multi-node distribution method for large models in this embodiment includes the following steps: Step 41, the target node sends a download request to the scheduler, indicating the target model file; Step 42, the scheduler selects several candidate nodes from among the candidate nodes to add to the candidate parent node set, based at least on the indicator values ​​of the status indicators of each candidate node in the cluster and the global distribution information of each shard among the candidate nodes. The candidate nodes have downloaded at least one shard and are at least topologically connected to the target node without forming a loop; Step 43, the scheduler determines the target file block that the target node will prioritize downloading from multiple file blocks; Step 44, the scheduler sends the candidate parent node set and the block identifier of the target file block to the target node; Step 45, the target node prioritizes downloading the shards in the target file block from the nodes in the candidate parent node set. The specific execution method of each of the above steps is described below.

[0091] First, in step 41, the target node sends a download request to the scheduler, specifying the target model file. It is understood that the target node and the scheduler belong to the same cluster or the same model distribution system. The model parameters of a large model exist in the form of model files, and a large model typically corresponds to several model files. The target model file can be any one of these model files.

[0092] In the embodiments described in this specification, the target node can be... Figure 2 In the system shown, any peer node is triggered by a user to send a download request from the target node to the scheduler. For example, a user initiates a download request for a target model file to the target node. The target node generates a unique task ID, which can correspond to the filename of the target model file. The target node can first check if the target content to be downloaded exists in its local cache. If the cache is hit, it directly returns the target content; otherwise, it sends a download request to the scheduler, reporting task metadata to the scheduler for scheduling. The download request sent by the target node to the scheduler can include the aforementioned task ID or the filename of the target model file.

[0093] As mentioned earlier, a large model can be stored as multiple model files. When a peer node needs to download two or more model files, in a typical implementation, the peer node generates and sends a separate download request for each model file it wishes to download. In another implementation, the peer node can also specify multiple model files it wishes to download in a single download request. The download request in step 41 can be any of the download requests described above, indicating the target model files to be downloaded.

[0094] In a specific example, the large model is stored as model file 1 and model file 2, and the target model file is model file 1. Model files can be in various formats such as safetensors files and bin files; no specific format is required here. The target model file (e.g., model file 1) has been pre-divided into multiple file blocks, for example, file blocks A, B, and C, and each file block is further divided into multiple fragments. The file blocks and fragments mentioned below refer to those within the target model file.

[0095] Then, in step 42, the scheduler selects several candidate nodes to add to the candidate parent node set, based at least on the status metric values ​​of each candidate node in the cluster and the global distribution information of each shard across the candidate nodes. Each candidate node must have downloaded at least one shard and must be topologically connected to the target node without forming a loop. It is understandable that a parent node of the target node needs to meet certain conditions; candidate nodes are initially selected based on these conditions, while the nodes in the candidate parent node set are further selected from the candidate nodes.

[0096] In one example, several candidate nodes are selected from the candidate nodes and added to the candidate parent node set, including:

[0097] Each candidate node is scored in multiple dimensions, including an indicator value score based on the indicator value and a shard coverage score based on the global distribution information; wherein, the shard coverage score is calculated based on the number of intersections between the shards already held by the candidate node and the remaining undownloaded shards of the target node.

[0098] The top K candidate nodes are selected in descending order of their scores and added to the candidate parent node set.

[0099] In this example, candidate nodes with better node status can be selected based on the indicator score, and candidate nodes that provide more available shards can be selected based on the shard coverage score, thus improving the effectiveness of the selection of candidate parent node set.

[0100] Furthermore, the index value score includes at least one of the following:

[0101] Load quality score is used to characterize whether candidate nodes are available, whether they are saturated for a long time, or whether concurrency exceeds the limit;

[0102] Internet Data Center (IDC) affinity score is used to characterize whether candidate nodes and target nodes belong to the same IDC;

[0103] Geographical affinity score, used to characterize the geographical distance between candidate nodes and target nodes;

[0104] Node type score is used to characterize whether a candidate node belongs to the seed node category.

[0105] In this example, based on the load quality score, performance bottleneck nodes can be dynamically avoided, ensuring the stability of the download link and bandwidth utilization; based on the IDC affinity score, cross-data center traffic costs can be minimized, and the advantages of intranet transmission within the same data center can be strengthened; based on the geographical location affinity score, network path fine-grained optimization can be achieved in cross-IDC scenarios; based on the node type score, the core supply role of seed nodes can be strengthened, and the topology cold start of peer-to-peer networks can be accelerated.

[0106] Next, in step 43, the scheduler determines the target file block that the target node will prioritize downloading from multiple file blocks. This means that when multiple fragments are available for download by the target node, the target node first selects the fragment belonging to the target file block for download, and then selects fragments belonging to other file blocks for download.

[0107] In one example, determining the target file block that the target node will prioritize downloading includes:

[0108] Obtain the number of nodes corresponding to each of the multiple file blocks as the target file blocks to be downloaded first;

[0109] Select the file block with the fewest corresponding nodes from the multiple file blocks, and determine it as the target file block that the target node will prioritize downloading.

[0110] In this example, the scheduler allocates target file blocks to target nodes for priority download, ensuring that the number of nodes prioritizing the download of the same file block is as balanced as possible. This avoids imbalanced node distribution that could negatively impact the download efficiency of subsequent nodes. Such a grouping strategy enables more even distribution of fragments among nodes, thereby improving the overall system throughput efficiency.

[0111] Figure 5 This diagram illustrates how a scheduling node allocates priority download target file blocks to peer nodes according to one embodiment. Continuing the previous example, it is assumed that the target model file to be distributed has been pre-segmented into three file blocks (i.e., file block A, file block B, and file block C). File block A contains fragments P1-P4, file block B contains fragments P5-P8, and file block C contains fragments P9-P12.

[0112] The current system has five peer nodes. The scheduling node has completed the initial allocation for some peer nodes: peer node 1 and peer node 4 have priority download target file block A, peer node 2 has priority download target file block B, and peer node 3 has priority download target file block C. When it is necessary to allocate a priority download target file block for peer node 5, the scheduling node counts the number of nodes currently designated as priority download target file blocks for each file block: file block A corresponds to 2 nodes, and file blocks B and C each correspond to 1 node. Based on the principle of balanced distribution, the scheduling node determines the file block with the fewest nodes (i.e., file block B or file block C) as the priority download target file block for peer node 5. This allocation strategy ensures that each file block has sufficient node support, providing resource guarantees for the parallel download of subsequent nodes.

[0113] In step 44, the scheduler sends the set of candidate parent nodes and the block identifier of the target file block to the target node. It can be understood that the target node can determine the set of candidate parent nodes and the target file block based on the information received from the scheduler, so that it can preferentially download fragments from the target file block from the nodes in the candidate parent node set.

[0114] In the embodiments of this specification, the candidate parent node set and the target file block are determined by the scheduler. The candidate parent node set includes multiple parent nodes, each of which has several fragments available for download. The subsequent target node can specifically select the target fragment to download and from which parent node to download the target fragment.

[0115] Finally, in step 45, the target node prioritizes downloading fragments from the target file block from the candidate parent node set. It can be understood that the target node can download fragments from the target file block, or fragments from other file blocks.

[0116] Continuing the previous example, if the target node is assigned file block C, the target node will preferentially download fragments P8-P12 from file block C from the candidate parent node. Since each file block is assigned as the priority download file block for different nodes in a relatively balanced manner, the fragments P1-P12 in the target model file can be more dispersed and evenly distributed among the peer nodes, avoiding the situation where a large number of nodes download a small number of fragments at the same time.

[0117] In one example, the step of preferentially downloading fragments from the target file block from nodes in the candidate parent node set includes:

[0118] Metadata for each shard on each node in the candidate parent node set is collected, and the metadata includes the storage location.

[0119] For the metadata of any collected fragment, if the fragment belongs to the target file block, its metadata is distributed to the high-priority queue; otherwise, it is distributed to the low-priority queue.

[0120] Prioritize selection from the high-priority queue to determine several target fragments to be downloaded;

[0121] For any target fragment, select its corresponding target parent node from the set of candidate parent nodes;

[0122] Based on the metadata, download the corresponding target fragments from each target parent node.

[0123] In this example, metadata-driven dual-queue priority scheduling prioritizes the download of target file block fragments by placing them in a high-priority queue. This allows nodes to trigger inference pipeline assembly immediately after downloading the target file block, precisely accelerating inference pipeline construction, significantly reducing service cold start latency, and achieving the core goal of loading and inference simultaneously. The high-priority-first, low-priority-last-resort queue strategy avoids download downtime, ensuring continuous bandwidth saturation and eliminating bandwidth resource waste.

[0124] Furthermore, a number of target fragments to be downloaded are selected from the high-priority queue, including:

[0125] The high-priority queue is selected as the target queue first; when the high-priority queue is empty, the low-priority queue is selected as the target queue.

[0126] A predetermined number of fragments that can be downloaded in parallel are selected from the target queue as the target fragments.

[0127] In this example, the target node can select a fragment from the fragments that meet the download conditions to trigger the download based on the priority mechanism. If there is a downloadable fragment in the high priority queue, it will be selected first. If the high priority queue is empty, it will fall back to select a fragment in the low priority queue.

[0128] Furthermore, several target fragments to be downloaded were identified, including:

[0129] Receive global fragmentation distribution information periodically sent by the scheduler;

[0130] Based on the latest received global sharding distribution information, several shards with higher scarcity are selected as the target shards. The scarcity is determined by the number of nodes in the cluster that hold the shard; the fewer the nodes, the higher the scarcity.

[0131] In this example, the target node synchronizes with the scarcity priority scheduling strategy based on global shard distribution information. It can identify the scarce shards held by the fewest nodes (such as those held by only 1 or 2 nodes) from a real-time global perspective and download them first, so that they can be quickly spread to more node caches, improving shard spread efficiency. It accelerates the equalization of shard distribution across the entire network, reduces the continuous dependence on seed nodes or origin servers, and enhances the robustness of the system when nodes dynamically join or leave. It coordinates pipeline parallelism and global efficiency. Under the dual-queue framework, scarcity priority is implemented within the queues to ensure the accelerated download of critical file blocks, while avoiding scarce shards from becoming a bottleneck in the pipeline stage readiness.

[0132] Further, the step of selecting the corresponding target parent node from the candidate parent node set for any target fragment includes:

[0133] The weights are calculated based on the current state information of each node in the candidate parent node set; wherein the current state information includes at least one of the following: the node's current idle bandwidth, the most recently smoothed bandwidth obtained by exponential moving average processing, and the load saturation determined based on the ratio of the current number of connections to the maximum number of connections;

[0134] Based on the weights, a target parent node is selected for the target shard from the set of candidate parent nodes.

[0135] In this example, before the target node initiates the download of each shard, it will reselect each node in the candidate parent node set based on the current state information of each node in the candidate parent node set. It will randomly sample based on the probability distribution constructed by the weights to ensure that each shard can be bound to the current better parent node.

[0136] In the embodiments described in this specification, during the download process of the target node, real-time status information, such as network bandwidth information and upload load information, can be periodically synchronized from each node in the candidate parent node set. The target node calculates weights based on the current status information of each node, and these weights are used to guide the selection and dynamic scheduling of subsequent target parent nodes.

[0137] Furthermore, the load saturation is determined as follows: when the ratio of the current number of connections to the maximum number of connections of a node is less than 1, the load saturation is 1; when the ratio is greater than or equal to 1, the load saturation is the reciprocal of the ratio.

[0138] In this example, nodes nearing overload are identified by calculating the proportion of the current load to the total capacity.

[0139] Further, the step of collecting metadata for each shard from each node included in the candidate parent node set includes:

[0140] When collecting metadata from each shard, wait for a preset time window to aggregate metadata from multiple nodes for the same shard.

[0141] In this example, the target node collects metadata for each shard from the candidate parent node set. Each parent node returns a list of shards it currently owns. This information is aggregated, and a short period of time (e.g., 5ms by default) is waited to collect as much metadata as possible from multiple candidate parent nodes for a single shard. Subsequently, a parent node can be selected from these multiple candidate parent nodes as the download source for that shard, which may include, but is not limited to, random selection.

[0142] Furthermore, the method also includes:

[0143] The target node reports the downloaded shard information to the scheduler so that the scheduler updates the global shard distribution information; the scheduler receives the downloaded shard information reported by each node in the cluster and dynamically updates the global shard distribution information; the updated global shard distribution information is sent to each node in the cluster so that each node selects the shard to download based on the global shard distribution information.

[0144] In this example, during the download process, the target node can immediately report to the scheduler after each fragment is downloaded. This information is used to update the global fragment distribution information, as well as the topology and dynamic scheduling strategy of the peer network. The scheduler can periodically synchronize the global fragment distribution information to the target node so that the target node can prioritize the selection of scarce fragments in subsequent downloads.

[0145] In one example, the method further includes:

[0146] When the target node detects that the set of candidate parent nodes cannot complete the download of the remaining fragments, it sends a rescheduling request to the scheduler. The rescheduling request is used to request the reallocation of the set of candidate parent nodes.

[0147] The scheduler selects several candidate nodes from the candidate nodes as an updated set of candidate parent nodes, based at least on the updated status index values ​​of each candidate node in the cluster, the updated global distribution information of each shard on each candidate node, and the target file block.

[0148] The updated set of candidate parent nodes is sent to the target node.

[0149] In this example, the target node sends a rescheduling request to the scheduler when its parent nodes in the current round cannot complete all remaining shards, some parent nodes fail, or their performance degrades. The scheduler returns a new set of candidate parent nodes based on the latest task status and peer network topology, thus entering the next round of scheduling, until all shards are downloaded or the maximum number of scheduling attempts is reached. The new set of candidate parent nodes may no longer include the previous parent nodes. For example, if the target node downloaded shards from parent node 1 and parent node 2 in the first round, then after rescheduling, the new set of candidate parent nodes will no longer include parent node 1 and parent node 2.

[0150] The method provided in the embodiments of this specification allows for the distribution of large models into several model files. The target model file is pre-segmented into multiple file blocks, and each file block is pre-segmented into multiple fragments. When the scheduler selects a set of candidate parent nodes for the target node, it considers not only the status indicators of each candidate node but also the global distribution information of each fragment across the candidate nodes, thus improving the effectiveness of nodes in the candidate parent node set from the scheduling source. Furthermore, the scheduler determines the target file blocks that the target node prioritizes for download. By rationally allocating target file blocks to different nodes, fragments can be distributed to different nodes more quickly, allowing subsequent nodes to download fragments in parallel from different sources. In summary, this achieves efficient and low-latency model distribution.

[0151] Figure 6 This diagram illustrates the interaction of a multi-node distribution method for a large model according to another embodiment. (Refer to...) Figure 6 This method is executed by a model distribution system, which involves a scheduler, target nodes, and a set of candidate parent nodes allocated by the scheduler to the target nodes. The model file is pre-divided into several file blocks according to a fixed size or optimization strategy, with each file block containing multiple fragments. First, the user initiates a download request for the target model file to the target node; then, the target node generates a unique task identifier for the task and reports the task metadata to the scheduler; next, the scheduler allocates a set of candidate parent nodes to the target node and specifies the target file block to download first; the target node starts a fragment collection unit to collect metadata of the fragments owned by each node in the candidate parent node set, and sends the collected metadata to a fragment selection unit to select the fragment to download; the target node starts a parent node selection unit to collect the status information of each node in the candidate parent node set and select a parent node for the fragment to be downloaded; the target content, i.e., the fragments, is downloaded asynchronously through multi-threading.

[0152] The following is a specific example of how the scheduler assigns a set of candidate parent nodes to the target node.

[0153] First, the scheduler checks the status of the target node. Upon receiving a scheduling request from a target node, the scheduler checks whether the node's status allows it to enter the scheduling process. For example, only target nodes in the ReceivedNormal (just initiated a normal download request) or Running (downloading, may need rescheduling) states will enter the scheduling process; target nodes in BackToSource, Failed / Succeeded, or Pending states (corresponding to back to the source, failure / completion, and pending, respectively) will not enter the scheduling process. The scheduler will directly return an "unschedulable" message or return to the source branch to avoid invalid scheduling.

[0154] Then, the nodes in the cluster are coarsely screened to obtain various candidate nodes. After the status is valid, the scheduler randomly samples a batch of possible parent nodes and then applies filtering conditions to each of them to obtain a list of candidate nodes that can "enter the scoring stage". For example, the filtering rules include, but are not limited to: same hostname filtering, removing nodes with the same hostname as the target node to avoid loops or self-downloading; DAG existence filtering, requiring that the node must already exist in the DAG and have an in-degree that can be found; edge feasibility filtering, requiring that the parent node must satisfy the DAG condition, i.e., no loops. The candidate nodes obtained after this filtering step form the minimum available set, ensuring that the topology is connectable, the status is normal, and no bad links are formed.

[0155] Finally, each candidate node is scored, and a set of candidate parent nodes is obtained based on the scores. The candidate nodes after the initial screening are then comprehensively scored and sorted in descending order of score. The score can be composed of a weighted average of 5 sub-scores:

[0156] The Load Quality Score is determined by combining the parent node's current available bandwidth (reflecting "whether it is currently usable"), continuous upload load (reflecting "whether it has been saturated for a long time recently"), and concurrency health (reflecting "whether the current node's concurrency has exceeded the limit").

[0157] IDC Affinity Score: 1 for the same IDC, 0 for different IDCs;

[0158] Location Affinity Score: Higher scores are given to locations with closer geographical proximity and shorter network paths.

[0159] The Host Type Score assigns 0.5 points to a normal peer, 1 point to a seed node in normal condition, and 0 points to any other case.

[0160] The Piece Coverage Score, which prefers parent nodes that provide more available shards, is calculated as the reciprocal of the following formula: , where M is the remaining undownloaded shard set of the target node, and P is the completed shard set of the parent node.

[0161] After the scores are sorted, the scheduler, based on the limit on the number of nodes in the candidate parent node set, prunes from high score to low score, and selects the top K (Top-K) nodes to form the candidate parent node set.

[0162] The following is a specific example of a target node selecting the shard it wants to download.

[0163] This embodiment introduces a priority-based fragment scheduling mechanism. During the download process, nodes will prioritize acquiring fragments within the target file block allocated to them, but they are not restricted to downloading only high-priority fragments. Instead, a "priority-but-not-exclusive" scheduling strategy is adopted: if a high-priority fragment is available, it will be scheduled first; if a high-priority fragment cannot be downloaded at present, it will automatically fall back to downloading low-priority fragments to avoid bandwidth idleness and maintain the continuity of the overall download process.

[0164] To implement this mechanism, the fragment collection unit distributes all fragment metadata that meets the download criteria to two independent queues: a high-priority queue for storing fragments belonging to the target file block that the node is responsible for; and a low-priority queue for storing other downloadable fragments. When a node has available download concurrency, the fragment scheduling process is triggered. The scheduling strategy is to always prioritize selecting fragments from the high-priority queue; only when the high-priority queue is empty are fragments selected from the low-priority queue. This strategy ensures that critical fragments are downloaded first while avoiding bandwidth waste caused by overly strict prioritization, thereby maximizing download efficiency.

[0165] After determining the selected queue, when deciding which fragment to download from the queue, the system further adopts a Rarest-First strategy. The node will periodically obtain global fragment distribution information from the scheduler and prioritize the fragment with the fewest fragments in the entire network from the current queue for download.

[0166] The following is a specific example of a target node selecting a parent node for the shard it is about to download.

[0167] To further adapt to dynamic changes in cluster bandwidth and load, a parent node selection can be performed before the formal download is initiated. The core idea of ​​this selection mechanism is to calculate weights based on the real-time bandwidth and load characteristics of candidate parent nodes, and then select the parent node accordingly. The key features used to calculate the weights and the parent node selection method in this embodiment will be described in detail below:

[0168] Real-time idle bandwidth characteristics Using the periodically reported state snapshots from candidate parent nodes, calculate the theoretically maximum remaining capacity they can provide. Let candidate parent node i ( The total upload capacity is denoted as The current instantaneous upload load is denoted as .

[0169] This feature reflects the node's current instantaneous capability.

[0170] Bandwidth smoothing feature (EMA) To avoid drastic weight fluctuations due to network jitter, the historical bandwidth performance of candidate parent nodes is treated as a time series. An exponential moving average (EMA) is used to analyze this data. Perform smoothing. Assume... This is a smoothing factor (e.g., 0.3).

[0171] This feature reflects the node's stable upload capability over a period of time.

[0172] Load saturation characteristics ( ), calculate the current load as a percentage of total capacity to identify nodes nearing overload. Record the size of each shard as . ,Will The maximum number of connections is denoted as The current number of connections is denoted as .

[0173] ,when When, it indicates that the concurrency of the candidate parent node has not yet reached the upper limit, so it is set to 1. At this point, the candidate parent node concurrency is saturated; the more concurrency, the lower the weight.

[0174] final, Selected weights The combined weights are derived from the above features:

[0175] ;

[0176] in, ;

[0177] Target node based on The distribution is randomly sampled to ensure that each shard can be bound to a parent node that is in a better overall state and has a balanced load.

[0178] At the parent node selection level, the target node side constructs a multi-feature weight model based on real-time idle bandwidth, historical smoothed bandwidth, and load saturation. It then uses probabilistic sampling for pre-download reselection and dynamic scheduling, enabling the system to adapt to bandwidth fluctuations and node load changes, improve bandwidth utilization, and avoid single-point overheating. The synergy of these three elements makes the distribution process more efficient and robust, and significantly accelerates the distributed loading of large models and the startup of inference services.

[0179] According to another embodiment, a multi-node distribution device for a large model is also provided. The device is disposed in a scheduler and is used to execute the actions of the scheduler in the method provided in the embodiments of this specification. The large model corresponds to several model files, and the target model file is pre-segmented into multiple file blocks, and each file block is pre-segmented into multiple fragments. Figure 7 A schematic block diagram of a multi-node distribution apparatus for a large model according to one embodiment is shown. Figure 7 As shown, the device 700 includes:

[0180] The receiving unit 71 is used to receive a download request sent by the target node, which indicates the target model file;

[0181] Selection unit 72 is used to select several candidate nodes from the candidate nodes and add them to the candidate parent node set, based at least on the index values ​​of the status indicators of each candidate node in the cluster and the global distribution information of each shard on each candidate node. The candidate nodes have downloaded at least one shard and at least satisfy the topological connectivity with the target node and will not form a loop.

[0182] The determining unit 73 is used to determine the target file block that the target node will prioritize downloading from the plurality of file blocks;

[0183] The sending unit 74 is used to send the candidate parent node set obtained by the selection unit 72 and the block identifier of the target file block obtained by the determination unit 63 to the target node, so that the target node downloads the fragments in the target file block from the nodes in the candidate parent node set first.

[0184] Optionally, as an embodiment, the selection unit 72 includes:

[0185] The scoring subunit is used to score each candidate node in multiple dimensions. The multi-dimensional scoring includes an indicator value score based on the indicator value and a shard coverage score based on the global distribution information. The shard coverage score is calculated based on the number of intersections between the shards already held by the candidate node and the remaining undownloaded shards of the target node.

[0186] The first selection subunit is used to select the top K candidate nodes in descending order of the scores obtained from the scoring subunit and add them to the candidate parent node set.

[0187] Furthermore, the index value score includes at least one of the following:

[0188] Load quality score is used to characterize whether candidate nodes are available, whether they are saturated for a long time, or whether concurrency exceeds the limit;

[0189] Internet Data Center (IDC) affinity score is used to characterize whether candidate nodes and target nodes belong to the same IDC;

[0190] Geographical affinity score, used to characterize the geographical distance between candidate nodes and target nodes;

[0191] Node type score is used to characterize whether a candidate node belongs to the seed node category.

[0192] Optionally, as an embodiment, the determining unit 73 includes:

[0193] The acquisition subunit is used to acquire the number of nodes corresponding to the multiple file blocks as target file blocks to be downloaded first;

[0194] The second selection subunit is used to select the file block with the fewest nodes obtained by the corresponding acquisition subunit from the plurality of file blocks, and determine it as the target file block that the target node prioritizes for downloading.

[0195] Optionally, as an embodiment, the receiving unit 71 is further configured to receive a rescheduling request sent by the target node, the rescheduling request being used to request the reallocation of the candidate parent node set;

[0196] The selection unit 72 is further configured to reselect several candidate nodes from the candidate nodes as an updated set of candidate parent nodes, based at least on the update index values ​​of the status indicators of each candidate node in the cluster, the updated global distribution information of each shard on each candidate node, and the target file block.

[0197] The sending unit 74 is further configured to send the updated set of candidate parent nodes to the target node.

[0198] Optionally, as an embodiment, the receiving unit 71 is further configured to receive information on downloaded shards reported by each node in the cluster and dynamically update the global shard distribution information;

[0199] The sending unit 74 is further configured to send the updated global shard distribution information to each node in the cluster, so that each node selects the shard to download based on the global shard distribution information.

[0200] According to another embodiment, a multi-node distribution device for a large model is also provided. The device is set on a target node in a cluster and is used to execute the actions of the target node in the method provided in the embodiments of this specification. The large model corresponds to several model files, and the target model files are pre-segmented into multiple file blocks, and each file block is pre-segmented into multiple fragments. Figure 8 A schematic block diagram of a multi-node distribution apparatus for a large model according to another embodiment is shown. Figure 8 As shown, the device 800 includes:

[0201] Sending unit 81 is used to send a download request to the scheduler of the cluster, wherein the target model file is indicated;

[0202] The receiving unit 82 is used to receive the candidate parent node set and the block identifier of the target file block that the target node prioritizes to download, sent by the scheduler; the candidate parent node set is determined at least based on the index value of the status index of each candidate node in the cluster and the global distribution information of each shard on each candidate node, and the candidate node has downloaded at least one shard and is at least topologically connected to the target node and will not form a loop.

[0203] Download unit 83 is used to preferentially download fragments from the target file block from the set of candidate parent nodes received by receiving unit 82.

[0204] Optionally, as one embodiment, the download unit 83 includes:

[0205] A collection subunit is used to collect metadata of each shard on each node in the candidate parent node set, the metadata including storage location;

[0206] The distribution subunit is used to distribute the metadata of any fragment collected by the collection subunit to a high-priority queue if the fragment belongs to the target file block; otherwise, it is distributed to a low-priority queue.

[0207] The first selection subunit is used to select from the high-priority queue obtained by the distribution subunit to determine several target fragments to be downloaded;

[0208] The second selection subunit is used to select the corresponding target parent node from the set of candidate parent nodes for any target fragment obtained by the first selection subunit.

[0209] The download subunit is used to download the corresponding target fragments from each target parent node based on the metadata.

[0210] Furthermore, the first selection subunit is specifically used to preferentially select the high-priority queue as the target queue, and when the high-priority queue is empty, select the low-priority queue as the target queue; and select a predetermined number of fragments that can be downloaded in parallel from the target queue as the plurality of target fragments.

[0211] Furthermore, the first selection subunit is specifically used to receive global fragment distribution information periodically sent by the scheduler; based on the latest received global fragment distribution information, several fragments with higher scarcity are selected as the several target fragments, wherein the scarcity is determined by the number of nodes in the cluster holding the fragment, and the fewer the number of nodes, the higher the scarcity.

[0212] Further, the second selection subunit is specifically used to calculate weights based on the current state information of each node in the candidate parent node set; wherein, the current state information includes at least one of the node's current idle bandwidth, the most recently smoothed bandwidth obtained by exponential moving average processing, and the load saturation determined based on the ratio of the current number of connections to the maximum number of connections; and selects a target parent node for the target shard from the candidate parent node set according to the weights.

[0213] Furthermore, the load saturation is determined as follows: when the ratio of the current number of connections to the maximum number of connections of a node is less than 1, the load saturation is 1; when the ratio is greater than or equal to 1, the load saturation is the reciprocal of the ratio.

[0214] Furthermore, the collection subunit is specifically used to wait for a preset time window to aggregate metadata from multiple nodes for the same shard when collecting metadata for each shard.

[0215] Furthermore, the sending unit 81 is also used to report the downloaded fragment information to the scheduler so that the scheduler updates the global fragment distribution information.

[0216] Optionally, as an embodiment, the sending unit 81 is further configured to send a rescheduling request to the scheduler when it is detected that the candidate parent node set cannot complete the download of the remaining fragments. The rescheduling request is used to request the reallocation of the candidate parent node set.

[0217] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 4 The method described.

[0218] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 4 The method described.

[0219] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0220] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-node distribution method for large models, the method being executed by a scheduler, wherein the large model corresponds to several model files, the target model file is pre-splitted into multiple file blocks, and each file block is pre-splitted into multiple fragments, the method comprising: Receive a download request sent by the target node, which indicates the target model file; Based at least on the status index values ​​of each candidate node in the cluster and the global distribution information of each shard on each candidate node, several candidate nodes are selected from each candidate node to be added to the candidate parent node set. Each candidate node has downloaded at least one shard and is at least topologically connected to the target node without forming a loop. From the plurality of file blocks, determine the target file block that the target node will prioritize downloading; The candidate parent node set and the block identifier of the target file block are sent to the target node, so that the target node downloads the fragments in the target file block from the nodes in the candidate parent node set first.

2. The method as described in claim 1, wherein, Several candidate nodes are selected from the candidate nodes and added to the candidate parent node set, including: Each candidate node is scored in multiple dimensions, including an indicator value score based on the indicator value and a shard coverage score based on the global distribution information; wherein, the shard coverage score is calculated based on the number of intersections between the shards already held by the candidate node and the remaining undownloaded shards of the target node. The top K candidate nodes are selected in descending order of their scores and added to the candidate parent node set.

3. The method as described in claim 2, wherein, The index score includes at least one of the following: Load quality score is used to characterize whether candidate nodes are available, whether they are saturated for a long time, or whether concurrency exceeds the limit; Internet Data Center (IDC) affinity score is used to characterize whether candidate nodes and target nodes belong to the same IDC; Geographical affinity score, used to characterize the geographical distance between candidate nodes and target nodes; Node type score is used to characterize whether a candidate node belongs to the seed node category.

4. The method of claim 1, wherein, The step of determining the target file block that the target node will prioritize downloading includes: Obtain the number of nodes corresponding to each of the multiple file blocks as the target file blocks to be downloaded first; Select the file block with the fewest corresponding nodes from the multiple file blocks, and determine it as the target file block that the target node will prioritize downloading.

5. The method of claim 1, wherein, The method further includes: Receive a rescheduling request sent by the target node, the rescheduling request being used to request the reallocation of the candidate parent node set; Based at least on the update index values ​​of the status indicators of each candidate node in the cluster, the updated global distribution information of each shard on each candidate node, and the target file block, several candidate nodes are reselected from each candidate node as the updated candidate parent node set. The updated set of candidate parent nodes is sent to the target node.

6. The method of claim 1, wherein, The method further includes: Receive information on downloaded shards reported by each node in the cluster and dynamically update the global shard distribution information; The updated global shard distribution information is sent to each node in the cluster so that each node can select the shard to download based on the global shard distribution information.

7. A multi-node distribution method for a large model, the method being executed by a target node in a cluster, the large model corresponding to several model files, the target model files being pre-splitted into multiple file blocks, each file block being pre-splitted into multiple fragments, the method comprising: Send a download request to the scheduler of the cluster, indicating the target model file; The system receives a set of candidate parent nodes and the block identifier of the target file block that the target node prioritizes for download, sent by the scheduler. The set of candidate parent nodes is determined at least based on the index values ​​of the status indicators of each candidate node in the cluster and the global distribution information of each shard on each candidate node. Each candidate node has downloaded at least one shard and is at least topologically connected to the target node without forming a loop. The nodes in the candidate parent node set are used to download fragments from the target file block first.

8. The method of claim 7, wherein, The step of preferentially downloading fragments from the target file block from nodes in the candidate parent node set includes: Metadata for each shard on each node in the candidate parent node set is collected, and the metadata includes the storage location. For the metadata of any collected fragment, if the fragment belongs to the target file block, its metadata is distributed to the high-priority queue; otherwise, it is distributed to the low-priority queue. Prioritize selection from the high-priority queue to determine several target fragments to be downloaded; For any target fragment, select its corresponding target parent node from the set of candidate parent nodes; Based on the metadata, download the corresponding target fragments from each target parent node.

9. The method of claim 8, wherein, Prioritize selection from the high-priority queue to determine several target fragments to be downloaded, including: The high-priority queue is selected as the target queue first; when the high-priority queue is empty, the low-priority queue is selected as the target queue. A predetermined number of fragments that can be downloaded in parallel are selected from the target queue as the target fragments.

10. The method of claim 8, wherein, Several target fragments to be downloaded have been identified, including: Receive global fragmentation distribution information periodically sent by the scheduler; Based on the latest received global sharding distribution information, several shards with higher scarcity are selected as the target shards. The scarcity is determined by the number of nodes in the cluster that hold the shard; the fewer the nodes, the higher the scarcity.

11. The method of claim 8, wherein, The step of selecting the corresponding target parent node from the set of candidate parent nodes for any target fragment includes: The weights are calculated based on the current state information of each node in the candidate parent node set; wherein the current state information includes at least one of the following: the node's current idle bandwidth, the most recently smoothed bandwidth obtained by exponential moving average processing, and the load saturation determined based on the ratio of the current number of connections to the maximum number of connections; Based on the weights, a target parent node is selected for the target shard from the set of candidate parent nodes.

12. The method of claim 11, wherein, The load saturation is determined as follows: when the ratio of the current number of connections to the maximum number of connections of a node is less than 1, the load saturation is 1; when the ratio is greater than or equal to 1, the load saturation is the reciprocal of the ratio.

13. The method of claim 8, wherein, The step of collecting metadata for each shard from each node in the candidate parent node set includes: When collecting metadata from each shard, wait for a preset time window to aggregate metadata from multiple nodes for the same shard.

14. The method of claim 10, wherein, The method further includes: The downloaded shard information is reported to the scheduler so that the scheduler updates the global shard distribution information.

15. The method of claim 7, wherein, The method further includes: When it is detected that the candidate parent node set cannot complete the download of the remaining fragments, a rescheduling request is sent to the scheduler. The rescheduling request is used to request the reallocation of the candidate parent node set.

16. A multi-node distribution device for a large model, the device being disposed in a scheduler, the large model corresponding to several model files, the target model file being pre-segmented into multiple file blocks, each file block being pre-segmented into multiple fragments, the device comprising: The receiving unit is used to receive download requests sent by the target node, which indicate the target model file; The selection unit is used to select several candidate nodes from the candidate nodes and add them to the candidate parent node set, based at least on the indicator values ​​of the status indicators of each candidate node in the cluster and the global distribution information of each shard on each candidate node. The candidate nodes have downloaded at least one shard and are at least topologically connected to the target node and will not form a loop. The determining unit is configured to determine, from the plurality of file blocks, the target file block that the target node will prioritize downloading; The sending unit is used to send the candidate parent node set obtained by the selection unit and the block identifier of the target file block obtained by the determination unit to the target node, so that the target node downloads the fragments in the target file block from the nodes in the candidate parent node set first.

17. A multi-node distribution device for a large model, the device being disposed on a target node in a cluster, the large model corresponding to several model files, the target model files being pre-splitted into multiple file blocks, each file block being pre-splitted into multiple fragments, the device comprising: A sending unit is used to send a download request to the scheduler of the cluster, wherein the target model file is indicated; The receiving unit is used to receive the set of candidate parent nodes and the block identifier of the target file block that the target node prioritizes to download, sent by the scheduler; the set of candidate parent nodes is determined at least based on the index values ​​of the status indicators of each candidate node in the cluster and the global distribution information of each shard on each candidate node, and the candidate node has downloaded at least one shard and is at least topologically connected to the target node and will not form a loop. The download unit is used to preferentially download fragments from the target file block from the set of candidate parent nodes received by the receiving unit.

18. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-15.

19. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-15.

Images