Large model pre-training system based on distributed parallel processing

Abstract

The invention relates to the technical field of distributed learning, in particular to a large model pre-training system based on distributed parallel processing. In the system, a data distribution layer collects a node resource state through a fragmentation module and generates a dynamic scheduling strategy; a computing resource layer configuration model initialization module and a strategy switching module support flexible switching of multiple modes such as tensor parallelism, data parallelism and assembly line parallelism; the network communication management layer is combined with topology perception and gradient compression technologies, so that the communication efficiency is improved; and the model aggregation layer realizes global parameter updating and training tuning under privacy protection through a security aggregation and optimization control mechanism. All layers of the system operate cooperatively, the calculation efficiency, the communication performance and the data security of large model pre-training can be effectively improved, and the method is suitable for model development and deployment in a large-scale heterogeneous calculation environment.

Description

Technical Field

[0001] The present invention relates to the field of distributed learning, and in particular to a large model pre-training system based on distributed parallel processing. Background Art

[0002] With the widespread application of large models in natural language processing, computer vision, and multimodal tasks, their parameter size and computational complexity are growing exponentially, and the computing resources, data size, and training time required for model pre-training are constantly increasing. In this context, distributed parallel training technology is widely used in large model training tasks, improving computational efficiency and scalability through multi-node collaboration.

[0003] To adapt to the ever-increasing scale of models and data processing requirements, existing technologies employ methods such as tensor parallelism, data parallelism, and pipeline parallelism to accelerate training, and deploy flexible task scheduling frameworks on a variety of heterogeneous computing platforms. Simultaneously, with the increasing demand for data security and privacy protection, training methods based on technologies such as homomorphic encryption and differential privacy are also gaining attention. However, in the multi-node collaboration process, training systems often face coordination challenges across multiple dimensions, including heterogeneous resource management, parallel strategy selection, communication topology optimization, privacy protection, and performance tuning. For example, in an environment with dynamically changing resources, how to rationally divide data and schedule tasks based on node status, how to flexibly select the optimal parallel mode based on task characteristics, and how to ensure the efficiency and security of gradient communication are all important areas of ongoing research attention.

[0004] To this end, a large model pre-training system based on distributed parallel processing is proposed. Summary of the Invention

[0005] The present invention provides a large-model pre-training system based on distributed parallel processing, including a data distribution layer, a computing resource layer, a network communication management layer and a model aggregation layer. It supports intelligent data scheduling, dynamic switching of parallel strategies, secure and efficient communication and privacy-protected model updates, and realizes efficient pre-training and optimization of large-scale models.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A large model pre-training system based on distributed parallel processing, including:

[0008] The data distribution layer includes a sharding module, which generates sharding and scheduling strategies for training data based on the resource usage of computing nodes.

[0009] The computing resource layer includes multiple heterogeneous nodes, each of which is configured with a model initialization module and a strategy switching module. After each node receives sharded data, the model initialization module loads the shared model structure and parameters. The strategy switching module selects tensor parallelism, data parallelism, or pipeline parallelism based on the task structure and load status.

[0010] The network communication management layer includes a topology awareness module and a gradient compression module. The topology awareness module monitors the communication status in real time based on SDN technology and dynamically builds the optimal transmission topology. The gradient compression module uses sparsification and residual compression to generate structured gradient packets to reduce communication overhead.

[0011] The model aggregation layer includes a secure aggregation module and an optimization control module. The secure aggregation module implements privacy aggregation of compressed gradients based on threshold homomorphic encryption and zero-knowledge proof. The optimization control module predicts loss trends through LSTM and dynamically adjusts the learning rate, quantization strategy, and early stopping conditions to achieve coordinated optimization of training convergence and resource utilization.

[0012] Furthermore, the sharding module performs the following steps:

[0013] Periodically collect the memory remaining, GPU utilization, and network throughput of each computing node to build a resource status matrix;

[0014] Input the resource state matrix into the pre-trained deep Q network and output the scheduling result;

[0015] The scheduling results are bound to heterogeneous nodes through the Kubernetes scheduler to achieve dynamic collaboration between data and computing.

[0016] Furthermore, the model initialization module performs the following steps:

[0017] Obtain the preset model structure definition file and initialization weight snapshot by accessing the distributed storage system;

[0018] Map the model structure into a computational graph and divide it into subgraphs, which are then allocated to heterogeneous devices based on node resource capabilities.

[0019] A broadcast mechanism is used to synchronize initial parameters to all participating nodes, ensuring that multiple nodes start training in a unified initial state.

[0020] Furthermore, the strategy switching module performs the following steps:

[0021] Collect GPU utilization, video memory occupancy, and historical task execution records of local computing nodes;

[0022] Analyze the model structure characteristics and data scale of the current task, and evaluate the resource matching degree of different parallel strategies;

[0023] Based on the evaluation results, a heuristic scheduling algorithm is used to select tensor parallelism, data parallelism or pipeline parallelism mode.

[0024] Furthermore, the topology awareness module performs the following steps:

[0025] Build a programmable communication structure that decouples the network control plane and forwarding plane, and collects inter-node link latency, bandwidth utilization, and data congestion indicators in real time;

[0026] Based on the above monitoring data and the node's historical communication records, a time series prediction model is trained to estimate potential communication bottlenecks in the next cycle;

[0027] The network topology is modeled as a weighted graph, the minimum spanning tree algorithm is used to generate the communication topology, and the SDN controller is used to dynamically reconstruct the communication topology to optimize the gradient transmission efficiency.

[0028] Furthermore, the gradient compression module performs the following steps:

[0029] Perform singular value decomposition on the local gradient matrix and extract the first k principal components to construct a low-rank gradient representation;

[0030] The difference between the original gradient and the low-rank gradient representation is defined as a residual term, and the residual is quantized and entropy coded to form a reversible compressed package;

[0031] The low-rank representation is combined with the compressed residual to generate a structured gradient packet, which is distributedly transmitted via the optimal path selected by the topology-aware module.

[0032] Furthermore, the security aggregation module performs the following steps:

[0033] The validity of the gradients of each participating node is verified through zero-knowledge proof, and Gaussian noise is injected into the aggregation process to enhance differential privacy protection;

[0034] A threshold homomorphic encryption scheme is constructed based on Shamir secret sharing, and at least t nodes participate in decryption aggregation.

[0035] Furthermore, the optimization control module performs the following steps:

[0036] The LSTM neural network predicts the downward trend of the loss function and automatically reduces the learning rate when the predicted decline rate falls below a preset threshold.

[0037] If the accuracy stagnates for several consecutive rounds, the early stopping strategy will be automatically triggered, terminating the current round of pre-training.

[0038] The beneficial effects of the present invention are:

[0039] 1. By introducing a sharding scheduling strategy based on deep reinforcement learning, the data distribution layer can perceive the real-time status of compute node resources such as memory and bandwidth, dynamically generate the optimal data scheduling solution, and improve the efficiency of data and computing collaboration. This mechanism effectively avoids computing bottlenecks caused by uneven resource distribution, enhances system flexibility and robustness, and is suitable for complex task scheduling scenarios in large-scale heterogeneous clusters.

[0040] 2. The computing resource layer combines a model initialization module with a strategy switching module to enable unified loading and flexible deployment of pre-trained model structures and parameters across heterogeneous nodes. Based on the task load and model structure, the strategy switching module intelligently selects the most suitable parallel computing strategy, enabling each node to fully utilize its resource advantages, thereby improving forward and backward computation efficiency and reducing overall training latency.

[0041] 3. The network communication management layer integrates topology awareness and gradient compression mechanisms to predict bottlenecks and reconstruct efficient topologies before communication. It also performs low-rank decomposition and compression on gradient data, reducing communication traffic and latency. This design significantly improves network utilization and transmission stability during large model training, and is particularly suitable for model parallel tasks in scenarios with high communication costs or uneven bandwidth.

[0042] 4. The model aggregation layer combines threshold homomorphic encryption, zero-knowledge proofs, and differential privacy mechanisms to achieve trusted gradient aggregation across multiple nodes while ensuring parameter security. Furthermore, the optimization control module, integrated with LSTM prediction, adaptively adjusts the learning rate and early stopping strategy based on loss trends, enhancing the convergence and resource efficiency of the training process and effectively supporting multi-task dynamic optimization requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0044] Figure 1 This is a schematic diagram of the structure of a large model pre-training system based on distributed parallel processing provided by the present invention;

[0045] Figure 2 It is a flowchart of the execution steps of the sharding module provided by the present invention;

[0046] Figure 3 It is a flowchart of the execution steps of the topology perception module provided by the present invention;

[0047] Figure 4 This is a flow chart of the execution steps of the gradient compression module provided by the present invention. DETAILED DESCRIPTION

[0048] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0049] Example 1

[0050] A large model pre-training system based on distributed parallel processing, such as Figure 1 Shown, including:

[0051] The data distribution layer includes a sharding module, which generates sharding and scheduling strategies for training data based on the resource usage of computing nodes.

[0052] Furthermore, the sharding module performs the following steps: Figure 2 Shown, including:

[0053] Periodically collect the memory remaining, GPU utilization, and network throughput of each computing node to build a resource status matrix;

[0054] Input the resource state matrix into the pre-trained deep Q network and output the scheduling result;

[0055] The scheduling results are bound to heterogeneous nodes through the Kubernetes scheduler to achieve dynamic collaboration between data and computing.

[0056] Specifically, the system periodically collects key resource usage metrics from all participating nodes, including but not limited to: memory availability (in MB), GPU core utilization (percentage), current network bandwidth usage, and throughput (in MB / s). This collected data forms a node-wise resource status matrix, which represents the current overall system load. This resource status matrix is fed into a pre-trained deep Q network, which has been trained for policy optimization using historical scheduling efficiency as a reward function. The network outputs include the optimal data slice ratio for each node, the scheduling priority of each data batch, and the optimal inter-node transmission path (e.g., cross-GPU vs. cross-server). This output forms a complete scheduling action combination that guides data flow distribution. This scheduling action combination is registered and bound to the system's underlying resource manager interface. Specifically, the Kubernetes scheduler mounts the training data shards to the corresponding compute nodes based on the node allocation decisions output by the DQN. If the node resource status changes, the DQN re-evaluates the next round of scheduling actions, implementing a dynamic scheduling mechanism characterized by "resource awareness + rolling optimization."

[0057] Through a data sharding and scheduling mechanism driven by resource status, dynamic and balanced distribution of training data among heterogeneous computing nodes can be achieved. This not only improves the overall resource utilization and training throughput of the system, but also has good scalability and adaptability. Especially in actual deployment scenarios such as multiple nodes, uneven loads or network fluctuations, it can significantly reduce node idleness and congestion, thereby effectively improving the efficiency and stability of the large model pre-training process.

[0058] The computing resource layer includes multiple heterogeneous nodes, each of which is configured with a model initialization module and a strategy switching module. After each node receives sharded data, the model initialization module loads the shared model structure and parameters. The strategy switching module selects tensor parallelism, data parallelism, or pipeline parallelism based on the task structure and load status.

[0059] Furthermore, the model initialization module performs the following steps:

[0060] Obtain the preset model structure definition file and initialization weight snapshot by accessing the distributed storage system;

[0061] Map the model structure into a computational graph and divide it into subgraphs, which are then allocated to heterogeneous devices based on node resource capabilities.

[0062] A broadcast mechanism is used to synchronize initial parameters to all participating nodes, ensuring that multiple nodes start training in a unified initial state.

[0063] Specifically, the system first accesses a cluster-shared distributed storage system (such as Ceph, HDFS, or Amazon S3) to retrieve a preset model definition file and initial weight snapshot. The model definition file, in a standardized format (such as ONNX, PyTorch Script, or TensorFlow SavedModel), describes the network structure and layer connectivity. The initial weight file is a snapshot of parameters for the predefined structure, derived from a pretrained model or a cold start strategy. The system then parses the model structure into a static or dynamic computational graph, where nodes represent operators and edges represent tensor dependencies. Based on the computational graph topology and each node's resource information (such as memory size and computing power rating), a graph partitioning algorithm (such as Metis or GreedyPartition) is used to partition the entire model into several subgraphs, each adapted to one or more target compute devices. Once partitioned, the system distributes the initial weights of the corresponding subgraphs to the target nodes and synchronizes the globally shared parameters to all compute nodes through a parameter broadcast mechanism (such as NCCLbroadcast or Horovod Allgather), ensuring that all replicas are in a consistent initialization state. After the parameter broadcast is complete, each node performs structure loading and parameter loading on the local subgraph and returns a successful initialization signal. After the control node summarizes all responses, it enters the training phase.

[0064] The model initialization module enables structural partitioning and parameter consistency initialization of large-scale models in heterogeneous computing environments, significantly improving the collaborative efficiency and stability of distributed systems during the training startup phase. By combining computational graph segmentation with resource-aware allocation strategies, it not only achieves efficient matching of computing tasks and node capabilities, but also avoids convergence anomalies caused by inconsistent parameters. Furthermore, with the help of parameter broadcasting and verification mechanisms, all computing nodes are ensured to enter the training process at the same starting point, effectively reducing the risk of initialization delays and inter-node coordination errors.

[0065] Furthermore, the strategy switching module performs the following steps:

[0066] Collect GPU utilization, video memory occupancy, and historical task execution records of local computing nodes;

[0067] Analyze the model structure characteristics and data scale of the current task, and evaluate the resource matching degree of different parallel strategies;

[0068] Based on the evaluation results, a heuristic scheduling algorithm is used to select tensor parallelism, data parallelism or pipeline parallelism mode.

[0069] Specifically, the resource monitor continuously collects the GPU utilization, video memory occupancy and recent task execution records of the local node, builds a resource status snapshot and records task operation performance indicators, such as training throughput, memory overflow times and communication waiting time. Next, the model configuration parser parses the structural characteristics of the current training task (such as parameter scale, number of layers, forward and backward calculation ratio, etc.) and the batch size of the input data to generate a task representation vector. Subsequently, the policy switching module inputs the node resource snapshot and task representation vector into a preset heuristic evaluation model (such as a lightweight decision tree). The model combines predefined policy priorities and resource adaptability scores to evaluate the feasibility and rank the three strategies of tensor parallelism, data parallelism and pipeline parallelism. Finally, the optimal parallel strategy is selected based on the evaluation score, and the strategy is injected into the training scheduling process through a unified execution interface to achieve dynamic parallel switching at the node level.

[0070] The policy switching module implements a dynamic parallel strategy adaptation mechanism that integrates node-level resource awareness with task characteristics. This allows the system to autonomously select the most suitable parallel execution method under diverse computing loads and model structures. Compared to fixed policy configurations, this module can effectively alleviate resource bottlenecks, improve computing utilization, and reduce communication latency, thereby improving overall training efficiency and system throughput.

[0071] The network communication management layer includes a topology awareness module and a gradient compression module. The topology awareness module monitors the communication status in real time based on SDN technology and dynamically builds the optimal transmission topology. The gradient compression module uses sparsification and residual compression to generate structured gradient packets to reduce communication overhead.

[0072] Furthermore, the topology perception module performs the following steps: Figure 3 Shown, including:

[0073] Build a programmable communication structure that decouples the network control plane and forwarding plane, and collects inter-node link latency, bandwidth utilization, and data congestion indicators in real time;

[0074] Based on the above monitoring data and the node's historical communication records, a time series prediction model is trained to estimate potential communication bottlenecks in the next cycle;

[0075] The network topology is modeled as a weighted graph, the minimum spanning tree algorithm is used to generate the communication topology, and the SDN controller is used to dynamically reconstruct the communication topology to optimize the gradient transmission efficiency.

[0076] Specifically, the team first constructed a programmable communication structure. The network adopts an SDN architecture, decoupling the control and forwarding planes. The controller communicates with each node switch or virtual network interface card (VNIC) via the OpenFlow protocol, collecting link status data in real time. Collected metrics include one-way or two-way link latency between node pairs, instantaneous bandwidth utilization, packet loss rate, and queue length. Secondly, predictive modeling of historical communication data is performed. The topology awareness module maintains historical node communication records using a sliding window mechanism and uses a time series prediction model to predict trends in latency and congestion risk for key links. Finally, a communication topology optimization scheme is developed. All compute nodes in the training task are treated as nodes in a weighted graph, with edge weights measured by the predicted bandwidth-delay product. A minimum spanning tree algorithm is used to generate a low-load, high-efficiency communication topology. Once generated, the SDN controller issues path reconstruction commands in real time to achieve gradient synchronization and parameter transmission path reconfiguration and switching.

[0077] By introducing an SDN architecture and a communication bottleneck prediction mechanism, we can fully understand and predict the communication status of the training network. Combined with graph optimization algorithms, we can dynamically generate a low-latency, low-congestion communication topology. Compared to static path configuration, this module significantly improves gradient synchronization efficiency and reduces communication latency during training.

[0078] Furthermore, the gradient compression module performs the following steps: Figure 4 Shown, including:

[0079] Perform singular value decomposition on the local gradient matrix and extract the first k principal components to construct a low-rank gradient representation;

[0080] The difference between the original gradient and the low-rank gradient representation is defined as a residual term, and the residual is quantized and entropy coded to form a reversible compressed package;

[0081] The low-rank representation is combined with the compressed residual to generate a structured gradient packet, which is distributedly transmitted via the optimal path selected by the topology-aware module.

[0082] Specifically, after each round of training, each node generates a local gradient matrix. The system performs an SVD operation on this matrix, extracting the first k principal components and constructing a low-rank approximate representation. The value of k can be dynamically adjusted based on the compression rate target or reconstruction error threshold. The low-rank representation retains the key information of the gradient matrix, significantly reducing the representation dimensionality. The difference between the original gradient and the low-rank approximation is then used as the residual term. The residual is discretized using non-uniform quantization and then converted into a compact bitstream using Huffman coding, forming a reversible compressed packet. This process ensures that the original gradient information can be restored when needed to meet training accuracy requirements. The low-rank principal components and compressed residuals are then packaged into structured gradient data according to a predefined format. The structured gradient packet contains metadata such as index identifiers, principal component matrices, residual compression stream, and decoding instructions, facilitating rapid decompression and reconstruction at the receiving node. Finally, the system's topology-aware module selects a transmission path for the structured gradient packet, automatically avoiding network bottlenecks, thereby ensuring efficient transmission and latency performance of the gradient update.

[0083] By introducing mechanisms such as singular value decomposition, residual compression, and structured encapsulation, the amount of gradient communication during distributed training is significantly reduced while maintaining minimal degradation in model accuracy. Combined with topology-aware path scheduling, this approach reduces inter-node communication latency and the risk of network congestion, improving overall training throughput. This approach is particularly suitable for training large models with numerous parameters and frequent communication.

[0084] The model aggregation layer includes a secure aggregation module and an optimization control module. The secure aggregation module implements privacy aggregation of compressed gradients based on threshold homomorphic encryption and zero-knowledge proof. The optimization control module predicts loss trends through LSTM and dynamically adjusts the learning rate, quantization strategy, and early stopping conditions to achieve coordinated optimization of training convergence and resource utilization.

[0085] Furthermore, the security aggregation module performs the following steps:

[0086] The validity of the gradients of each participating node is verified through zero-knowledge proof, and Gaussian noise is injected into the aggregation process to enhance differential privacy protection;

[0087] A threshold homomorphic encryption scheme is constructed based on Shamir secret sharing, and at least t nodes participate in decryption aggregation.

[0088] Specifically, before aggregation, the system executes a zero-knowledge proof protocol on the gradient vectors uploaded by each participating node, using non-interactive proof mechanisms such as Bulletproofs or zk-SNARKs. This verifies that the uploaded gradients conform to computational specifications (including gradient norm range and source model consistency) without revealing the specific gradient values. This process effectively prevents attacks involving the upload of forged or malicious gradients, improving system robustness. To further enhance the privacy protection of training data, before gradient aggregation, the system injects Gaussian noise that complies with differential privacy constraints into each gradient vector. The intensity of this noise is controlled according to a preset privacy budget to ensure that the aggregated output meets the (ε, δ)-differential privacy standard. Finally, during the encrypted aggregation phase, the system constructs a threshold homomorphic encryption protocol based on the Shamir secret sharing mechanism. Each node's gradient is encrypted and sharded using a t-out-of-n threshold, which is then transmitted to at least t authorized nodes. Decryption and aggregation of the encrypted gradients are completed only when at least t legitimate nodes participate in the aggregation process.

[0089] By integrating zero-knowledge proofs, Gaussian differential privacy, and threshold homomorphic encryption into the gradient aggregation process, we achieve both validation of uploaded data and strong privacy protection during the aggregation process during distributed training. This system effectively prevents attacks such as gradient contamination and membership inference, improving the security, reliability, and compliance of large-scale model training.

[0090] Furthermore, the optimization control module performs the following steps:

[0091] The LSTM neural network predicts the downward trend of the loss function and automatically reduces the learning rate when the predicted decline rate falls below a preset threshold.

[0092] If the accuracy stagnates for several consecutive rounds, the early stopping strategy will be automatically triggered, terminating the current round of pre-training.

[0093] Specifically, a dynamic prediction model is constructed based on LSTM. The LSTM network takes the loss function value, gradient change rate, accuracy evaluation index, etc. of each round in the historical training process as input, and outputs the predicted trend of the loss value of several rounds in the future. When the loss decrease rate predicted by the LSTM model is lower than the preset threshold for multiple consecutive cycles (such as the loss decrease rate is less than 0.01 for five consecutive cycles), the system automatically reduces the learning rate to avoid oscillating convergence or falling into a local optimum. The learning rate adjustment strategy can be flexibly implemented by means of piecewise attenuation, cosine annealing or adaptive optimizer parameter update, which is not limited in this embodiment. When the system detects that the model accuracy (such as the verification set accuracy, F1-score, etc.) has not been significantly improved in several consecutive training rounds (such as the verification set accuracy is less than 0.01 for five consecutive training rounds), it means that the model has converged, and the system will automatically trigger the early stopping strategy to terminate the training process of the current round.

[0094] By introducing the LSTM prediction mechanism and early stopping strategy, the optimization control module can dynamically perceive and intelligently regulate the convergence behavior of the training process, significantly reducing invalid training rounds and improving training efficiency; at the same time, it avoids overfitting and resource waste, and enhances the generalization performance and stability of the model in actual deployment.

[0095] Example 2

[0096] In a large medical institution, a multi-center collaborative large-scale medical image diagnosis model was built. The goal was to use distributed multi-node computing resources to jointly pre-train multimodal data containing millions of CT and MRI images to improve the accuracy and robustness of the model in rare disease identification and early lesion localization. In this application scenario, the system based on the present invention is implemented as follows:

[0097] Each medical center anonymizes the medical images it collects locally, then uses a sharding module to divide them into image blocks. These blocks are then dynamically distributed based on the available memory and network bandwidth of each computing node. The scheduling results generated by the Deep Q Network ensure load balancing and transmission efficiency between nodes.

[0098] The system accesses a unified repository of model definitions and weight snapshots. Each node's model initialization module loads the pathology image recognition model (ResNet-152 + UNet) and synchronizes initial parameters. Subsequently, a strategy switching module automatically selects a strategy combining tensor parallelism and pipeline parallelism based on factors such as image size and number of feature channels, executing training on a multi-GPU heterogeneous cluster.

[0099] The topology awareness module detects link latency differences between hospital nodes in real time and dynamically adjusts communication paths through the SDN controller. The gradient compression module uses singular value decomposition and entropy coding to compress each round of gradient updates, reducing the burden of wide-area transmission between hospitals.

[0100] After each round of training, the secure aggregation module uses threshold homomorphic encryption to merge private gradients at a central node and verifies the validity of each node's gradients through a zero-knowledge proof mechanism to prevent malicious uploads or data bias. The optimization control module uses LSTM to analyze training loss trends, dynamically adjust the learning rate, and terminate training early when performance saturation occurs.

[0101] To verify the performance advantages of the system of the present invention in medical imaging pre-training tasks, a comparative experiment was designed, as shown in Table 1. As can be seen from the table, the system of the present invention exhibits significant advantages over traditional full-data parallel solutions and mainstream federated learning frameworks in medical imaging pre-training scenarios: First, by introducing dynamic scheduling of deep Q networks and SDN-controlled communication topology reconstruction, the training time is shortened from the traditional 26 hours to 16.2 hours, and the training efficiency is improved by more than 37%; second, by combining gradient compression and optimal path transmission, the network bandwidth occupancy is effectively reduced to less than half of the original system; third, by leveraging efficient resource scheduling and optimization control mechanisms, the model's AUC in the lung nodule recognition task is improved from 0.914 to 0.943, significantly enhancing prediction performance; finally, the system integrates privacy mechanisms such as zero-knowledge verification and threshold homomorphic encryption to ensure that the entire process meets data security requirements in high-privacy scenarios, making it particularly suitable for cross-node collaborative training in data-sensitive fields such as medical care.

[0102] Table 1 Data comparison table under different training schemes

[0103]

[0104]

[0105] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A large model pre-training system based on distributed parallel processing, characterized in that: include: The data distribution layer includes a sharding module that generates sharding and scheduling strategies for training data based on the resource usage of computing nodes. The computing resource layer includes multiple heterogeneous nodes, each of which is configured with a model initialization module and a strategy switching module; After each node receives the sharded data, the model initialization module loads the shared model structure and parameters; the strategy switching module selects tensor parallelism, data parallelism, or pipeline parallelism based on the task structure and load status; The network communication management layer includes a topology awareness module and a gradient compression module. The topology awareness module monitors the communication status in real time based on SDN technology and dynamically builds the optimal transmission topology. The gradient compression module uses sparsification and residual compression to generate structured gradient packets to reduce communication overhead. Model aggregation layer, including security aggregation module and optimization control module; The secure aggregation module implements privacy aggregation of compressed gradients based on threshold homomorphic encryption and zero-knowledge proof. The optimization control module predicts the loss trend through LSTM, dynamically adjusts the learning rate, quantization strategy and early stopping conditions, and achieves coordinated optimization of training convergence and resource utilization.

2. A large model pre-training system based on distributed parallel processing according to claim 1, characterized in that: The sharding module performs the following steps: Periodically collect the memory remaining, GPU utilization, and network throughput of each computing node to build a resource status matrix; Input the resource state matrix into the pre-trained deep Q network and output the scheduling result; The scheduling results are bound to heterogeneous nodes through the Kubernetes scheduler to achieve dynamic collaboration between data and computing.

3. A large model pre-training system based on distributed parallel processing according to claim 1, characterized in that: The model initialization module performs the following steps: Obtain the preset model structure definition file and initialization weight snapshot by accessing the distributed storage system; Map the model structure into a computational graph and divide it into subgraphs, which are then allocated to heterogeneous devices based on node resource capabilities. A broadcast mechanism is used to synchronize initial parameters to all participating nodes, ensuring that multiple nodes start training in a unified initial state.

4. A large model pre-training system based on distributed parallel processing according to claim 1, characterized in that: The strategy switching module performs the following steps: Collect GPU utilization, video memory occupancy, and historical task execution records of local computing nodes; Analyze the model structure characteristics and data scale of the current task, and evaluate the resource matching degree of different parallel strategies; Based on the evaluation results, a heuristic scheduling algorithm is used to select tensor parallelism, data parallelism or pipeline parallelism mode.

5. The large model pre-training system based on distributed parallel processing according to claim 1 is characterized in that: The topology awareness module performs the following steps: Build a programmable communication structure that decouples the network control plane and forwarding plane, and collects inter-node link latency, bandwidth utilization, and data congestion indicators in real time; Based on the above monitoring data and the node's historical communication records, a time series prediction model is trained to estimate potential communication bottlenecks in the next cycle; The network topology is modeled as a weighted graph, the minimum spanning tree algorithm is used to generate the communication topology, and the SDN controller is used to dynamically reconstruct the communication topology to optimize the gradient transmission efficiency.

6. A large model pre-training system based on distributed parallel processing according to claim 1, characterized in that: The gradient compression module performs the following steps: Perform singular value decomposition on the local gradient matrix and extract the first k principal components to construct a low-rank gradient representation; The difference between the original gradient and the low-rank gradient representation is defined as a residual term, and the residual is quantized and entropy coded to form a reversible compressed package; The low-rank representation is combined with the compressed residual to generate a structured gradient packet, which is distributedly transmitted via the optimal path selected by the topology-aware module.

7. The large model pre-training system based on distributed parallel processing according to claim 1, characterized in that: The security aggregation module performs the following steps: The validity of the gradients of each participating node is verified through zero-knowledge proof, and Gaussian noise is injected into the aggregation process to enhance differential privacy protection; A threshold homomorphic encryption scheme is constructed based on Shamir secret sharing, and at least t nodes participate in decryption aggregation.

8. The large model pre-training system based on distributed parallel processing according to claim 1, characterized in that: The optimization control module performs the following steps: The LSTM neural network predicts the downward trend of the loss function and automatically reduces the learning rate when the predicted decline rate falls below a preset threshold. If the accuracy stagnates for several consecutive rounds, the early stopping strategy will be automatically triggered, terminating the current round of pre-training.

Images