Big model cooperative training-oriented federal splitting learning method and device

Abstract

The invention provides a federal splitting learning method and device for large model cooperative training, and the method comprises the steps: splitting a large model into a plurality of sub-models through employing a calculation graph segmentation algorithm based on a calculation graph of a to-be-trained large model; respectively distributing the plurality of sub-models to corresponding client devices; receiving model parameters uploaded by a plurality of client devices and obtained based on local data training; the received model parameters of the plurality of client devices are aggregated to generate a global model, which is a large model that integrates training knowledge of all client devices. According to the method, the problems of equipment resource mismatching, high frequent packet communication overhead and data privacy protection when a large model is trained in a heterogeneous distributed environment in the prior art can be solved, cooperative training of the large model is efficiently and safely carried out in a resource limited environment, and meanwhile, model performance and user privacy are guaranteed.

Description

Technical Field

[0001] This invention relates to the field of federated decomposition learning technology, and in particular to a federated decomposition learning method and apparatus for collaborative training of large models. Background Technology

[0002] With the rapid development of deep learning technology, large language models have been successfully applied in fields such as healthcare, finance, and the Internet of Things, demonstrating powerful performance and playing a vital role in natural language processing and computer vision. However, traditional model training methods, such as federated learning and split learning, face limitations related to data privacy, communication overhead, and computational resources. Both federated learning (FL) and split learning (SL), as distributed machine learning paradigms, still have certain limitations.

[0003] Federated learning effectively protects user privacy by allowing clients to train using local data, but it still has significant shortcomings in practical applications. Federated learning requires all participating client devices to have sufficient computing resources to train a complete model, which is infeasible for resource-constrained client devices. Simultaneously, frequent model parameter transfers incur significant communication overhead, and model training is limited by bandwidth resources. Split learning reduces the demand on client computing power by dividing large-scale language models into client-side and server-side components using different splitting methods. Furthermore, existing split learning architectures typically use a centralized server to handle all client computing requests. This design relies on the central server's computing resources; if the server fails, model training cannot proceed.

[0004] Federated Split Learning (FSL) combines the advantages of federated learning and split learning, offering a new approach to solving the aforementioned problems. Within this framework, the client only needs to deploy and compute a subset of the split models, then send the intermediate results to the server for further processing, which helps alleviate the computational resource pressure on individual devices. Furthermore, since it does not require direct sharing of local raw data and complete model parameters, this approach offers advantages in privacy protection compared to either federated learning or split learning alone. However, when training large-scale pre-trained models in heterogeneous distributed environments, this approach still faces the following challenges: mismatched device resources lead to difficulties in deploying sub-models; frequent transmission of intermediate results incurs significant overhead due to communication bandwidth limitations and latency; and existing privacy protection mechanisms still need further enhancement to address potential data leakage risks. Summary of the Invention

[0005] This invention provides a federated split learning method and apparatus for collaborative training of large models, addressing the problems of device resource mismatch, high overhead from frequent small packet communication, and data privacy protection faced by existing technologies when training large models in heterogeneous distributed environments. It enables efficient and secure collaborative training of large models in resource-constrained environments while ensuring model performance and user privacy. The technical solution proposed by this invention is as follows: In a first aspect, the present invention provides a federated split learning method for collaborative training of large models, comprising: Based on the computation graph of the large model to be trained, the large model is split into multiple sub-models using a computation graph segmentation algorithm. The multiple sub-models are distributed to their respective client devices; Receive model parameters uploaded by the multiple client devices, which are trained based on local data; The model parameters received from multiple client devices are aggregated to generate a global model, which is a large model that integrates the training knowledge of all client devices.

[0006] Optionally, the computational graph based on the large model to be trained is divided into multiple sub-models using a computational graph segmentation algorithm, including: By symbolic tracing or model structure parsing, a directed acyclic graph corresponding to the large model is generated; Analyze the topology of the directed acyclic graph and estimate the computational and storage overhead of each node in the graph; Based on the computation and storage overhead of each node and the resource status of the client device, a partitioning strategy is selected; wherein, the partitioning strategy includes a memory-constrained partitioning strategy, a communication-optimized partitioning strategy, or a hybrid partitioning strategy, and the resource status of the client device includes one or more of the following: computing power, memory capacity, and network bandwidth. Based on the selected segmentation strategy, segmentation points are determined and segmentation is performed on the directed acyclic graph to generate the multiple sub-models; When segmenting the directed acyclic graph, the segmentation operation is preferentially performed at the boundary of the Transformer module.

[0007] Optionally, when the large model is transformed into a directed acyclic graph, the large model is divided into multiple predefined components at the module level, including an embedding layer, an encoding layer, a pooling layer, and a classification layer.

[0008] Optionally, the method further includes: Configure a communication protocol for the client device, the communication protocol supporting a multi-channel parallel transmission mechanism, the multi-channel including: The first channel is configured to transmit activation values ​​and control commands generated by the forward propagation of the transmission model. The second channel is configured to transmit gradient data generated by the backpropagation of the model. The third channel is configured to transmit logs and monitoring data during system operation; Among them, the data transmission priority of the first channel is higher than that of the second channel, and the data transmission priority of the second channel is higher than that of the third channel.

[0009] Optionally, the communication protocol employs a packet-splitting strategy for data transmission. This strategy is executed based on a packet-splitting threshold and a time window parameter, and is implemented through the following mechanism: When the accumulated amount of data to be sent reaches the packetization threshold, the accumulated data is immediately packaged and sent. If the data merging threshold is not reached, the accumulated data will be packaged and sent when the data merging time window expires.

[0010] Optionally, the aggregation of model parameters from multiple client devices generates a global model, including: According to the preset privacy protection policy, perform the following operations: If the strategy is configured to add noise to the gradient, then noise is injected into the gradients received from each client device, and the gradients after the noise injection are aggregated and calculated to generate the global model. If the strategy is configured to add noise to the aggregation process, then the updated model parameters received from each client device are aggregated to obtain the parameter aggregation result, and noise is injected into the parameter aggregation result to generate the global model.

[0011] Secondly, the present invention also provides a federated split learning device for collaborative training of large models, comprising the following modules: The model partitioning management module is used to split the large model to be trained into multiple sub-models based on the computation graph of the large model, using a computation graph partitioning algorithm; and to distribute the multiple sub-models to the corresponding client devices. The model aggregation and update module is used to receive model parameters uploaded by the multiple client devices and trained based on local data; and to aggregate the received model parameters from the multiple client devices to generate a global model, which is a large model that integrates the training knowledge of all client devices.

[0012] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the federated split learning method for large model collaborative training as described in the first aspect above.

[0013] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the federated split learning method for large model collaborative training as described in the first aspect above.

[0014] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the federated split learning method for collaborative training of large models as described in the first aspect above.

[0015] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows: This invention provides a federated splitting learning method and apparatus for collaborative training of large models. Based on the computation graph of the large model to be trained, it uses a computation graph splitting algorithm to split the large model into multiple sub-models, and then distributes the sub-models to corresponding client devices. This adaptive splitting can flexibly split according to the resource conditions of different client devices, so that each client device only needs to process the sub-model suitable for its own resource capabilities, avoiding the problem of difficult sub-model deployment due to mismatched device resources, and realizing effective collaborative training of large models in resource-constrained environments. In the prior art, frequent transmission of intermediate results will cause significant overhead due to communication bandwidth limitations and latency. However, this invention receives model parameters trained based on local data uploaded by multiple client devices, instead of frequently transmitting intermediate results. By aggregating these model parameters to generate a global model, the frequency and amount of data transmission are reduced, thereby reducing the overhead caused by communication bandwidth limitations and latency, making it more suitable for large-scale pre-trained model training in heterogeneous distributed environments. In this invention, the client devices upload model parameters trained based on local data, rather than the original local data and complete model parameters. This approach is more advantageous in terms of privacy protection than single federated learning and split learning because it reduces the risk of privacy leaks caused by direct sharing of raw data, protects user privacy, and enables collaborative training of large models while ensuring model performance.

[0016] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the architecture of the federated split learning method for collaborative training of large models provided by this invention.

[0020] Figure 2 This is a flowchart illustrating the federated split learning method for collaborative training of large models provided by this invention.

[0021] Figure 3 This is a schematic diagram of the calculation graph provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the computational graph segmentation algorithm provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the frame header structure provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of the message body provided by the present invention.

[0025] Figure 7 This is a schematic diagram of the front-end performance display interface provided by the present invention.

[0026] Figure 8 This is a schematic diagram of the structure of the federated split learning device for collaborative training of large models provided by the present invention.

[0027] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The purpose of this invention is to provide a new path that balances performance and security for efficient training of large models in resource-constrained environments. This path uses a federated split learning method to divide a large model into multiple sub-models and distribute them to clients, thereby significantly reducing the computational and communication overhead of a single device during model training. At the same time, since the client device only processes the part of the model that interacts with local data, the original data does not need to leave the local device, thus ensuring the privacy and security of user data from the source.

[0030] To achieve federated decomposition learning for collaborative training of large models, this invention provides a federated decomposition learning system for collaborative training of large models. (Refer to...) Figure 1As shown, the system is divided into two main parts: the front-end and the back-end, which interact through interfaces. The front-end, serving as the user interface, provides a visual interface. Users can intuitively configure training tasks, monitor training progress in real time, and view model evaluation results through this interface. The front-end initiates requests by calling the application programming interfaces (APIs) provided by the back-end and presents the execution results returned by the back-end to the user in the form of charts, thus ensuring a good user experience and ease of operation. The back-end is the core computing and coordination hub of the system, deployed in a server environment. It fully utilizes the abundant computing resources and high operating efficiency, significantly improving the system's processing power and interactive performance. Based on functional roles, the back-end is generally divided into two parts: a central server, local servers (server1…servern), and multiple client devices (client1, client2, client3…clientn). The local servers perform local management functions, receiving instructions from the central server and executing model splitting and task distribution within their jurisdiction. The central server undertakes the core functions of global coordination and model aggregation. Specifically, the central server first performs model splitting and distribution operations, splitting large-scale pre-trained models (hereinafter referred to as large models) such as Llama and BERT, running a computational graph partitioning algorithm, and distributing the split sub-models to various client devices. During training, the central server is responsible for performing model aggregation and parameter update operations, collecting model parameters or intermediate results uploaded by each client device, and splicing and aggregating models or intermediate results from different client devices, then updating global parameters to generate an updated global model. Simultaneously, each client device, as a terminal carrying distributed computing, performs local model training. Each client device receives a sub-model from the central server and trains it using its local data. During training, each client device calculates intermediate activation values ​​and intermediate gradients based on its local data, communicates intermediate data or gradients with other client devices through a secure communication mechanism based on Transmission Control Protocol / Internet Protocol (TCP / IP), and returns these intermediate results to the local server. The local server then sends the intermediate results to the central server.

[0031] The central server concatenates and aggregates models or intermediate results from different client devices, then updates global parameters to complete one round of federated learning or distributed training. Client devices are also responsible for periodically reporting training status and resource usage to the central server. This architecture, combining centralized coordination and aggregation by the central server with distributed training by the clients, effectively supports distributed training and centralized aggregation of models in federated learning scenarios. It achieves collaborative model optimization while protecting data privacy, completing each round of federated learning or distributed training. The central server will be referred to simply as the server below.

[0032] Reference Figure 2 As shown, this federated split learning method for collaborative training of large models includes the following: S110. Based on the computation graph of the large model to be trained, the large model is split into multiple sub-models using a computation graph segmentation algorithm.

[0033] The term "large model" refers to a large-scale pre-trained deep learning model with a parameter count exceeding one billion and a complex structure. This includes not only large language models, such as BERT, the GPT series, and Llama, but also other modalities of massive models, such as large-scale multimodal models, large-scale speech models, and large-scale scientific computing models.

[0034] The server loads the large model to be trained and obtains its computational graph. The computational graph of a large model is a graphical representation of the various computational nodes (such as convolutional layers, fully connected layers, activation functions, etc.) in the model and the data flow relationships between them. Each node corresponds to an operation or component, and each edge represents the data dependency between nodes. It shows the entire computational process of the model from input data to output results; each node represents a specific computational operation, and the edges between nodes indicate the direction of data flow. For example, in a large image classification model, the computational graph might contain nodes such as the input layer receiving image data, multiple convolutional and pooling layers performing feature extraction, and then fully connected layers performing classification prediction, along with corresponding data connections. Figure 3 A basic example of a computation graph structure is given. For instance, in the computation graph corresponding to the expression y=(x+z)*(z+1), node b represents x + z, and node c represents z. 1. Node a represents the final multiplication operation.

[0035] The server runs a computation graph segmentation algorithm. This algorithm analyzes the topology of the computation graph and assesses the computational and memory requirements of different parts of the model. Simultaneously, the algorithm considers the resource status of each client device participating in the training. Based on this analysis, the computation graph segmentation algorithm automatically determines the optimal splitting points, dividing the complete large model into several logically coherent sub-models. This process overcomes the inefficiency and inaccuracy of manual segmentation, achieving model segmentation and matching with the resources of heterogeneous devices.

[0036] S120. Distribute the multiple sub-models to the corresponding client devices respectively.

[0037] After model splitting is complete, the server, according to a predetermined allocation strategy, distributes the sub-models generated in step S110 to pre-specified client devices via the network. Each client device typically receives and maintains only one sub-model, thus distributing the massive model computational load across multiple devices. Specifically, the server obtains resource information for each client device, such as processor model, memory capacity, available storage space, network connection type, and bandwidth. Then, based on this information and the resource requirements of the sub-models, precise matching is performed. For example, sub-models with high memory requirements are prioritized for distribution to devices with sufficient memory; sub-models with high network bandwidth requirements are selected for distribution to devices with stable network connections and high bandwidth. The distribution process can be implemented using secure network communication protocols to ensure that the sub-models are not tampered with or leaked during transmission. For example, an encrypted transmission channel is used to encrypt the sub-model data before transmission, and only authorized client devices can decrypt and receive the corresponding sub-models.

[0038] S130: Receive model parameters uploaded by the multiple client devices, which are trained based on local data.

[0039] Each client device trains its assigned sub-model using local data. This local data is typically a task-specific dataset; for example, in a medical image classification task, the client device possesses collected medical image data. During training, the client device performs forward and backward propagation calculations, continuously adjusting the sub-model parameters to minimize the loss function and improve the model's performance on local data. For instance, through multiple iterative calculations, the weight parameters of each layer in the sub-model are updated, making the model's predictions on local data more accurate. After training is complete, the client device uploads the training results, i.e., the model parameters, to the server via a secure network channel. These model parameters embody the knowledge gained from the local data of each client device.

[0040] Specifically, upon receiving the training instruction from the server, the client device immediately initiates the local training process, strictly adhering to the preset epochs to perform complete forward propagation and backward gradient propagation. During training, the client device records and caches the training loss value, validation accuracy, and the actual number of samples participating in training for each batch in real time, proactively submitting these training metrics to the server at configured time intervals. Once the server completes global model aggregation, the client device receives the latest global model parameter update package through a secure channel, first verifying the parameter version number and digital signature, and then loading it into the local sub-model. The client device employs a hot update mechanism to ensure that parameter replacement does not affect the ongoing training task, sending a confirmation receipt to the server immediately after the parameter update is complete. Throughout the training cycle, the client device continuously monitors local resource usage, automatically pausing training and requesting parameter storage services from the server when memory is insufficient or the computational load is too high. Once resources are restored, the training task resumes from the point of interruption. Regarding local data management, training or test data (supporting PyTorch Dataset format) is loaded from a specified directory; if the data does not exist, simulated data is generated, ensuring that the original data never leaves the client device. Noise can be added to the server during gradient calculation to prevent data leakage. Real-time metric tracking is performed on client devices, recording metrics such as training time, communication latency, and sample processing volume. Model accuracy and loss are calculated on the test set, and the results are encrypted and uploaded to the server. Memory usage is also reported in real time.

[0041] S140. Aggregate the model parameters received from multiple client devices to generate a global model, which is a large model that integrates the training knowledge of all client devices.

[0042] After receiving model parameters uploaded from all client devices, the server performs parameter aggregation. The purpose of aggregation is to merge the local knowledge scattered across various client devices. The server uses an aggregation algorithm to comprehensively process all received model parameters, ultimately generating a unified and more powerful global model. This global model integrates the training results from all client devices and is the final output of the method of this invention.

[0043] Specifically, parameter aggregation methods can employ weighted averaging, federated averaging, and other algorithms. Taking weighted averaging as an example, the server assigns a weight to the model parameters of each client device based on factors such as the amount of data and device performance. Then, the parameters of all client devices are weighted and averaged to obtain the parameters of the global model. For instance, if a client device has a large amount of training data and good performance, it may be assigned a larger weight during aggregation, making its trained parameters have a greater impact on the global model. Through parameter aggregation, the server integrates the sub-model parameters trained by each client device into a complete global model parameter set. This global model integrates the training knowledge of all participating client devices and is the final output of the federated decomposition learning process. Then, a global model is constructed based on these global model parameters. This global model integrates the knowledge gained by all client devices trained on different local data, resulting in broader applicability and better performance.

[0044] The federated splitting learning method for collaborative training of large models provided by this invention uses a computation graph splitting algorithm to divide the large model into multiple sub-models based on the computation graph of the model to be trained, and then distributes the sub-models to corresponding client devices. This adaptive splitting can flexibly split the model according to the resource conditions of different client devices, so that each client device only needs to process the sub-model suitable for its own resource capabilities, avoiding the problem of difficult sub-model deployment due to mismatched device resources, and realizing effective collaborative training of large models in resource-constrained environments.

[0045] In existing technologies, frequent transmission of intermediate results incurs significant overhead due to communication bandwidth limitations and latency. This invention, however, receives model parameters trained on local data uploaded by multiple client devices, rather than frequently transmitting intermediate results. By aggregating these model parameters to generate a global model, the frequency and amount of data transmission are reduced, thereby mitigating the overhead caused by communication bandwidth limitations and latency. This makes it more suitable for large-scale pre-trained model training in heterogeneous distributed environments.

[0046] Existing privacy protection mechanisms still need further enhancement to address potential data leakage risks. In this invention, the client device uploads model parameters trained on local data, rather than the original local data and complete model parameters. This approach is more advantageous in terms of privacy protection than single federated learning and split learning because it reduces the privacy leakage risk caused by direct sharing of original data, protects user privacy, and also enables collaborative training of large models while ensuring model performance.

[0047] In an optional embodiment, addressing the problem of complex computational graph structures and low efficiency of traditional manual partitioning strategies during large model training and inference, this invention combines computational graph analysis with an adaptive computational graph segmentation algorithm to achieve efficient and structured model decomposition. This strategy not only identifies key computational nodes at a fine-grained level but also maintains the integrity of the model structure at the module level, thereby achieving balanced distribution of computational load in multi-device heterogeneous computing environments. The adaptive computational graph segmentation algorithm designed in this invention is as follows: Figure 4 As shown. The algorithm first parses the topology of the model computation graph, then determines candidate segmentation points according to the selected segmentation strategy, and verifies the feasibility of the segmentation. Subsequently, based on the computing power and network bandwidth conditions of the client device, sub-models are generated and deployed to the client device. Specifically, the computation graph of the large model to be trained described in S110 above uses a computation graph segmentation algorithm to split the large model into multiple sub-models, including: S1101. Generate the directed acyclic graph corresponding to the large model through symbolic tracing or model structure parsing.

[0048] The large model is analyzed using symbolic tracing tools (e.g., torch.fx) provided by deep learning frameworks (such as PyTorch), transforming its dynamic computation process into a static directed acyclic graph (DAG) consisting of nodes and edges. For complex model structures that cannot be automatically traced, the graph is manually constructed by directly parsing its module definitions and connections. In this graph, each node represents a complete predefined component (such as an embedding layer or encoder layer), and edges represent data dependencies between components. For example, if the output of node A is used as the input of node B, a directed edge from A to B is established in the computation graph. By analyzing the topological relationships of the DAG, this invention can identify critical paths (such as the longest dependency path), high-load nodes (modules with high memory consumption or computational requirements), and further estimate the storage requirements and computational overhead of each node, providing a quantitative basis for subsequent segmentation.

[0049] S1102. Analyze the topology of the directed acyclic graph and estimate the computation and storage overhead of each node in the graph.

[0050] This paper analyzes the topology of the directed acyclic graph (DAG) to identify critical paths and high-load nodes. Simultaneously, it quantitatively estimates the computational and storage costs of each node in the graph, providing accurate data for subsequent segmentation. The critical path is the path with the longest total computation time among all possible execution paths from input to output in the entire DAG. This path determines the theoretical minimum time for the model to perform one round of forward propagation and is the bottleneck of system performance. Identifying the critical path helps understand the model's speed limit and considers how to shorten the total time of this path through parallelization during segmentation. For example, in a serially connected Transformer model, the critical path is usually the path that runs through all coding layers. High-load nodes refer to individual nodes (i.e., predefined components) in the graph that have particularly high system resource requirements. High-load nodes include both computationally and storage-intensive nodes. Computationally intensive nodes are those requiring intensive mathematical operations (such as large matrix multiplications and convolution calculations), characterized by huge computational overhead. High-load nodes refer to nodes that require caching large amounts of intermediate results (activation values) during forward or backward propagation for use in backpropagation, or nodes with a large number of parameters, characterized by huge storage overhead. For example, in the BERT model, the feedforward network module typically computes high-load nodes, while the embedding layer, due to its large vocabulary parameters, often stores high-load nodes. Identifying high-load nodes is beneficial for distributing the load evenly across resource-constrained devices.

[0051] Computational overhead is measured in gigaflops (GFLOPs). It is estimated by analyzing the types of operations performed by the nodes and the shapes of the input and output tensors. For fully connected layers, the computational overhead is approximately 2 × B × I × O, where B is the batch size, I is the input dimension, and O is the output dimension. For self-attention mechanisms, the computational overhead mainly comes from the projection of the query, key, and value matrices and the calculation of attention weights, and its complexity is proportional to the square of the sequence length. The total GFLOPs of a node are estimated by summing the FLOPs of all major operations within the node. Storage overhead is measured in megabytes (MB). It mainly consists of parameter storage overhead and activation storage overhead. Parameter storage overhead refers to the space occupied by all trainable parameters of the node itself. At FP32 precision, each parameter occupies 4 bytes. For example, the parameter storage of a linear layer is (input dimension I × output dimension O). The output dimension is O) × 4 bytes, then converted to MB. Activation storage overhead refers to the space occupied by the intermediate result (activation value) calculated by this node during forward propagation, which needs to be cached for backpropagation. The parameter storage overhead and activation storage overhead are added together to obtain an estimate of the node's total storage overhead.

[0052] S1103. Select a partitioning strategy based on the computation and storage overhead of each node and the resource status of the client device.

[0053] An adaptive partitioning strategy is selected by comprehensively considering information from two dimensions: first, the computational and storage overhead of each node within the model obtained in the previous step; and second, the resource status from the client device, including one or more of computing power, memory capacity, and network bandwidth. Based on this information, the most suitable partitioning strategy is selected from a predefined strategy library. The partitioning strategies include memory-constrained partitioning strategies, communication-optimized partitioning strategies, or hybrid partitioning strategies. The core objective of memory-constrained partitioning strategies is to ensure that each sub-model can run within the limited memory of the client device, using algorithms such as greedy algorithms for module allocation. The core objective of communication-optimized partitioning strategies is to minimize the amount of intermediate data that needs to be transferred between sub-models, prioritizing partitioning between sparsely connected layers in the computational graph. Hybrid partitioning strategies refer to a comprehensive balance between memory and communication overhead, first generating an initial scheme, and then dynamically adjusting parts that do not meet memory constraints. The partitioning strategy selection is achieved through the following steps: First, data integration and constraint determination are performed. The computational and storage overhead data of each node are matched with the resource status table maintained by the client device management module. This resource status table records the computational power, available memory, and network bandwidth information of each client device. During hard constraint checks, all client devices are traversed to check if any node's storage overhead exceeds its available memory. This check is implemented through a loop of numerical comparisons: for each node, its storage overhead is compared with the available memory of each client device. If any node is found to be unacceptable to any client device, a memory constraint partitioning strategy is triggered first.

[0054] The memory-constrained partitioning strategy is triggered when a hard constraint check finds a node that cannot be accommodated. The execution process employs a greedy allocation algorithm, starting from the input layer of the computation graph and traversing nodes in topological order, maintaining a pointer to the current client device and a memory usage counter. Nodes are allocated to the current client device, and memory usage is accumulated. When the accumulated memory usage exceeds a preset percentage (e.g., 90%) of the available memory on the client device, a split point is set at the current node, and allocation continues to the next client device. This process ensures that the memory usage of each partition remains within a safe range.

[0055] The communication optimization partitioning strategy is triggered when hard constraint checks are passed and the average network bandwidth is below a preset bandwidth, such as 100Mbps. The execution process first constructs a communication cost matrix, recording the size of the intermediate tensors that need to be transmitted along each edge of the computation graph. Then, a graph partitioning algorithm is used to divide the computation graph into a specified number of partitions, aiming to minimize the total communication volume across partition edges. The size of the intermediate tensor refers to the amount of activation data that needs to be transferred from one sub-model to another during the model's forward propagation, when the computation graph is partitioned by a partition point.

[0056] The hybrid partitioning strategy is triggered when, despite passing hard constraint checks, some client devices have less than a preset memory margin, such as 150%. This strategy employs a two-stage optimization: The first stage calls a communication optimization partitioning strategy to generate an initial partitioning scheme aimed at minimizing communication overhead. The second stage verifies the memory feasibility of this initial partitioning scheme. Specifically, if a partition in the initial scheme exceeds memory limits, while maintaining the overall communication optimization framework, the position within the excess partition with the minimum communication volume with subsequent nodes is selected as the new partitioning point. Through iterative adjustments, the computing nodes exceeding memory limits are rationally allocated to adjacent partitions. This inter-stage linkage mechanism ensures that scheme defects are dynamically corrected while maintaining communication efficiency. Through multiple iterations, a complete partitioning scheme that simultaneously satisfies memory constraints and communication optimization requirements is ultimately generated. Finally, a detailed partitioning scheme file is output, explicitly listing the positions of each partitioning point and the mapping relationship between sub-models and client devices.

[0057] S1104. Based on the selected segmentation strategy, determine the segmentation points on the directed acyclic graph and perform segmentation to generate the multiple sub-models.

[0058] Based on the selected segmentation strategy, specific segmentation points are determined on the directed acyclic graph (DAG). When segmenting the DAG, segmentation is preferentially performed at the boundaries of Transformer modules. This is because Transformer modules are the core building blocks of most large language models; segmenting at their boundaries, such as after an encoder layer, ensures the functional integrity of the resulting sub-models while minimizing the tensors required for cross-partition communication, thereby reducing communication overhead and simplifying subsequent concatenation and aggregation logic. Based on the determined segmentation points, the complete DAG is divided into multiple subgraphs, each of which is compiled and encapsulated into an independent, deployable sub-model.

[0059] This invention overcomes the shortcomings of traditional manual partitioning strategies, which are inefficient and difficult to adapt to heterogeneous environments. Through computational graph analysis and multi-strategy selection, it achieves efficient and structured partitioning of complex models, ensuring optimal matching between the partitioning scheme and heterogeneous device resources. By quantitatively estimating node overhead and combining it with device resource status for decision-making, it can accurately allocate computational load to appropriate client devices, avoiding resource bottlenecks on individual devices and achieving a balanced distribution of computational load, thereby improving the overall utilization of cluster resources and training efficiency. By prioritizing partitioning at the boundaries of Transformer modules, it maximizes the structural and functional integrity of sub-models. This not only avoids performance degradation caused by improper partitioning but also, due to the interruption at natural boundaries, makes the interfaces between sub-models clearer and reduces intermediate data transmission, greatly ensuring the effectiveness and communication efficiency of subsequent cross-device collaborative training.

[0060] In an optional embodiment, when the large model is transformed into a directed acyclic graph, the large model is divided into multiple predefined components at the module level.

[0061] Structural analysis of large models (such as BERT) no longer treats them as a series of isolated mathematical operations, such as addition or matrix multiplication. Instead, based on their model architecture, they are divided into multiple predefined functional components at the module level. These components are relatively independent substructures within the model that perform specific functions. These predefined components include embedding layers, encoding layers, pooling layers, and classification layers. The embedding layer is responsible for mapping discrete input symbols, such as words, into dense vector representations. The encoding layer, as the core of the model, consists of multiple stacked Transformer modules with identical structures and is responsible for feature extraction and transformation. The pooling layer is used to summarize the feature sequence, for example, extracting the vector corresponding to the [CLS] label as the representation of the entire sequence. The classification layer is one or more fully connected layers responsible for the final classification or regression prediction based on the extracted features.

[0062] When generating the directed acyclic graph (DAG), these predefined components are used as the basic nodes of the graph. For example, a complete Transformer encoder layer is treated as a single node, rather than being broken down into dozens of operator nodes. The edges between nodes represent the data flow between these macromodules; for example, the output vector of the embedding layer is used as the input to the first encoding layer. This coarse-grained representation ensures that each sub-model can contain one or more fully functional modules during subsequent segmentation. For example, a sub-model may contain all components from the embedding layer to the Nth encoding layer, guaranteeing that the sub-model can independently perform a series of coherent computations from the input to the intermediate feature representations.

[0063] Unlike traditional splitting methods that rely solely on operator-level rules, this invention employs a computational graph topology analysis and multi-strategy fusion approach when handling complex large models. Specifically, this invention performs structural modeling, dividing the model into several predefined components, including embedding layers, encoding layers, pooling layers, and classification layers, and representing these as complete modules as nodes in the computational graph. Unlike fine-grained operator-level nodes, this strategy ensures that each partition maintains functional integrity and structural rationality, avoiding performance degradation or computational graph resolution failures caused by excessive splitting. Based on this, reasonable splitting points are selected according to the number of target partitions and resource constraints, prioritizing splitting along Transformer module boundaries, and then adjusting based on memory and bandwidth conditions, thereby achieving a splitting effect that preserves both model structural integrity and resource adaptability. This invention fundamentally avoids the problem of traditional operator-level splitting potentially fragmenting a functional module. It ensures that each sub-model is a self-contained unit with clear semantics and complete functionality, effectively preventing significant performance degradation caused by unreasonable splitting. Because the number of nodes is drastically reduced compared to fine-grained operator graphs, the graph's topology is clearer, and dependencies are easier to resolve. This reduces the risk of segmentation algorithm failures and significantly reduces the number of nodes that need to be analyzed and estimated, making the segmentation decision process faster and more stable. Since segmentation occurs at module boundaries, especially between Transformer encoding layers, the interaction interfaces between sub-models are clear, and the data tensors are regularized. This greatly simplifies the design of cross-device communication, reduces the overhead and latency caused by complex interfaces and frequent small-scale communications, and improves the overall efficiency of collaborative training.

[0064] The server plays a core global coordination role, and its core functions include model partitioning and sub-model management, client device coordination and training control, model aggregation and updates, and privacy protection and secure communication.

[0065] Model partitioning and sub-model management refers to the server's structural splitting of a large model based on a computation graph partitioning algorithm, using a predefined partitioning strategy. This process comprehensively considers the topological dependencies of the computation graph, tensor shape matching, and the computing capabilities of heterogeneous devices, dividing the complete model into several logically independent sub-model modules. After splitting, the system intelligently allocates these sub-models according to the hardware configuration and network conditions of the client devices, ensuring balanced computational load across all client devices. During model distribution, the system fully records key metadata of the partitioning scheme, including the location of the split points, interface specifications, and resource configuration requirements of each module, and persistently stores this metadata. To adapt to dynamic training environments, the server implements an on-demand sub-model loading mechanism: when a client device cannot continue running due to insufficient memory or link interruption, the system can immediately unload non-critical sub-models and quickly reload them when conditions recover, ensuring the continuity and consistency of parameter states.

[0066] Client device coordination and training control refer to the server maintaining the real-time status of all participating nodes through a client device connection manager, monitoring link health using a heartbeat detection mechanism, and automatically marking a client device as offline if it does not respond within 60 seconds. The system employs an exclusive allocation strategy, meaning each sub-model partition is bound to only one active client device at any given time, while maintaining a partition-client device mapping table. For training process control, the server provides a centralized scheduler supporting both synchronous and asynchronous modes: synchronous mode requires all client devices to strictly align to training rounds, while asynchronous mode allows nodes to progress at different speeds. During training, the server tracks the training progress of each client device through a global counter and collects training status data in real time. The system's built-in aggregation trigger automatically executes model parameter aggregation and calls the performance evaluation module to verify the global model effect when preset conditions are met (such as reaching a specified number of rounds or a specific proportion of client devices completing training). The evaluation results are fed back to the scheduler to guide subsequent training strategies. All status updates and coordination instructions are transmitted through a reliable message queue to ensure the stability of the training process even when client devices dynamically join or leave.

[0067] Privacy protection and secure communication refer to the system's use of SSL / TLS encryption mechanisms to prevent man-in-the-middle attacks and ensure communication and data security. Client devices must verify their identity via a handshake protocol before accessing the system, and only authorized devices can participate in training. During the global parameter aggregation phase, the server injects Gaussian noise into the gradients or parameter aggregation results to effectively prevent the client device's original data from being inferred from the parameters.

[0068] Model aggregation and updating refers to the server's support for two parameter aggregation calculation modes: weighted average and equal-weighted average. In weighted average mode, the server automatically calculates aggregation weights based on the amount of training data submitted by each client device. Client devices with larger data volumes receive higher weights during aggregation. These weights can be determined using a logarithmic function or a linear proportion of the sample size. In equal-weighted average mode, the update gradients of all client devices are averaged using equal weights. The server verifies that the parameter version numbers of all active client devices are aligned before allowing the next round of training to begin, preventing model divergence caused by network latency or desynchronization. For temporarily offline client devices, the system caches the latest global parameters and prioritizes re-sending them upon reconnection, ensuring that all participants always train based on a unified model version.

[0069] The client device is responsible for collaborating with the server to complete distributed model training, while simultaneously enabling cross-device communication and protecting local data privacy during the communication process. Existing mainstream distributed training frameworks (such as the NVIDIA Collective Communication Library (NCCL) upon which PyTorch relies) heavily depend on high-speed interconnects and dedicated hardware for cross-device communication, resulting in insufficient system interoperability and portability. To address this issue, this invention designs a large-scale distributed communication mechanism based on an improved TCP / IP protocol, supporting multi-channel parallel transmission, data packet merging, and priority scheduling. It can efficiently transmit intermediate activation values ​​and gradient information generated by Transformer blocks during forward and backward propagation. This mechanism ensures data transmission reliability while introducing Secure Sockets Layer / Transport Layer Security (SSL / TLS) encryption and breakpoint resumption, improving communication security and robustness across public networks and heterogeneous device environments. Compared to traditional communication schemes, the communication module of this invention significantly enhances the compatibility and autonomous collaboration capabilities of client devices, lowers the deployment threshold, and provides greater flexibility and scalability for large-scale distributed training. Specifically, after the model is split, the client device establishes corresponding communication channels based on the dependencies between the sub-models in the computation graph. When establishing a TCP connection, the communication management module introduces an SSL / TLS encryption mechanism and maintains the stability of the link through heartbeat detection and automatic reconnection mechanisms.

[0070] In an optional embodiment, to improve transmission efficiency, the present invention defines a multi-channel communication mechanism at the application layer. The method further includes: Configure a communication protocol for the client device, the communication protocol supporting a multi-channel parallel transmission mechanism, the multi-channel including: The first channel is configured to transmit activation values ​​and control commands generated by the forward propagation of the transmission model. The second channel is configured to transmit gradient data generated by the backpropagation of the model. The third channel is configured to transmit logs and monitoring data during system operation; Among them, the data transmission priority of the first channel is higher than that of the second channel, and the data transmission priority of the second channel is higher than that of the third channel.

[0071] In practice, each channel is equipped with a dedicated sending thread, sending buffer, and TCP socket to avoid interference between different types of data and achieve physical isolation. The data frame format adopts a unified binary protocol structure, including three parts: frame header, message body, and checksum. The frame header and message body are as follows: Figure 5 and Figure 6As shown. The StartFlag identifies the start of the frame, ensuring accurate synchronization at the receiving end; the Frame Length records the total number of bytes in the entire data frame; the Frame Sequence Number assigns a unique number to each data frame for out-of-order recovery and retransmission control; and the Reserved field is a reserved field.

[0072] The message type (MsgType) identifies the message category, such as gradients, activation values, logs, or control commands. The channel ID (ChannelID) indicates the logical channel to which the message belongs, supporting multi-channel concurrent communication mechanisms. Priority defines the service priority of the message, ensuring that critical business operations are transmitted preferentially in congestion situations. The edge node ID (EdgeID) uniquely identifies the edge node from which the message originates, corresponding to the topological dependency of the directed acyclic graph. The payload length (PayloadLength) is a 4-byte field used to explicitly indicate the exact length of the subsequent payload portion. The payload refers to the actual data object being transmitted; it is the binary data stream obtained after serialization of gradients, activation values, parameters, and control commands. When MsgType is gradient, the payload refers to the serialized model gradient tensor. This is the partial derivative calculated during backpropagation and used to update the model parameters. When MsgType is activation value, the payload refers to the serialized intermediate layer output activation value tensor. This is the intermediate result generated at the sub-model split point during model forward propagation and needs to be passed to the next computation node. When MsgType is model parameters, the payload refers to the serialized sub-model parameters. This is the complete or updated model weights that the client device is ready to upload to the server after local training. When MsgType is control instructions, the payload refers to control signaling that encodes a specific command. For example, this could be a structured data packet indicating the start of training, synchronization barrier, iteration termination, or heartbeat detection.

[0073] The first channel, a high-priority channel, is dedicated to transmitting activation values ​​generated during model forward propagation, training start / stop commands, synchronization signals, and other critical control instructions. This channel employs an immediate transmission strategy, attempting to send data as soon as it is generated. A preemptive refresh mechanism is designed for urgent control instructions, allowing interruption of data being assembled during transmission to ensure minimal latency. The second channel, a medium-priority channel, is dedicated to transmitting gradient data generated during model backpropagation. The third channel, a low-priority channel, is dedicated to transmitting non-real-time data such as logs, performance monitoring data, and model checkpoint files. This channel uses a background asynchronous transmission mode to ensure its transmission does not block the core training data stream. Internally, the protocol stack implements a priority scheduler based on channel IDs, strictly adhering to the processing order of first channel over second channel, and second channel over third channel. When network bandwidth is limited, this scheduling mechanism ensures that activation values ​​and control instructions receive priority transmission resources, thereby guaranteeing a smooth training process and real-time coordination capabilities.

[0074] This invention, by establishing high-priority channels for activation values ​​and control commands and employing an instantaneous transmission strategy, ensures that the model's forward propagation pipeline is not blocked by communication latency in distributed training, significantly reducing the total time consumption of a single training iteration. By using a packet-merging strategy for gradient data, which is large in volume but less time-sensitive, a large number of small data packets are combined into a small number of large data packets for transmission, effectively reducing network protocol header overhead and the number of send / receive system calls. This achieves higher effective bandwidth and throughput while ensuring transmission reliability. Separating auxiliary data such as logs and monitoring from the core training data stream prevents these large but non-critical data from crowding out valuable network resources, ensuring the stability of the training process. Simultaneously, independent channels also make system status monitoring and data collection smoother, without affecting the performance of the core training task.

[0075] In an optional embodiment, the communication protocol employs a packet-splitting strategy for data transmission. The packet-splitting strategy is executed based on a packet-splitting threshold and a time window parameter. The packet-splitting strategy is executed through the following mechanism: when the accumulated amount of data to be sent reaches the packet-splitting threshold, the accumulated data is immediately packaged and sent; if the packet-splitting threshold is not reached, the accumulated data is packaged and sent when the packet-splitting time window expires.

[0076] When processing messages, the data transmission module first checks the message type and priority, and then dynamically selects whether to send immediately or delay and merge packets based on the aforementioned mechanism. For intermediate tensors that need to be transmitted across devices, the system automatically performs serialization and injects the corresponding sub-model into the deserialized data on the target client device to continue computation. To further improve efficiency, the system supports mechanisms such as tensor compression, breakpoint resumption, and encrypted transmission. The server node is responsible for the synchronous coordination of global training, while the client device focuses on local computation and communication operations. The two maintain efficient interaction through this improved protocol, ensuring the consistency of gradients and parameters during distributed training.

[0077] The packet merging strategy is a core data transmission mechanism designed to balance transmission latency and bandwidth utilization. Its implementation relies on two key parameters: the packet merging threshold and the packet merging time window. These parameters can be set according to actual needs; for example, the packet merging threshold can be 64KB, and the packet merging time window can be 10ms. The packet merging strategy is executed in the data transmission module according to the following logic: When data to be sent enters the sending buffer, the communication management module starts two parallel monitoring mechanisms: one continuously checks the amount of data accumulated in the buffer, and the other starts a countdown timer with a duration set to the packet merging time window.

[0078] During the countdown of the time window, as soon as the amount of data to be sent accumulated in the buffer reaches or exceeds the preset packet merging threshold, the waiting will be immediately interrupted, triggering a packetizing operation. This involves packaging all the data accumulated in the current buffer into a single data frame and immediately sending it out via a TCP socket. This is an efficiency-first triggering mode.

[0079] If the accumulated data volume fails to reach the merging threshold throughout the entire merging time window, the system will still trigger the merging and sending operation at the moment the time window expires. At this time, regardless of the size of the data in the current buffer, it will be merged and sent. This is the delay control triggering mode, designed to prevent a very small amount of data from remaining in the buffer for an extended period.

[0080] This invention utilizes a threshold-triggered mechanism to merge multiple scattered small data packets generated by the application into one or a small number of large data packets at the application layer before transmission. This effectively reduces the protocol header overhead, system call counts, and network interface operations required for each data packet during network transmission, thereby significantly reducing the overall system overhead of network transmission and significantly increasing the proportion of effective data payload in the total transmission volume. A timeout-triggered mechanism sets a maximum waiting time limit for small-volume data, preventing indefinite delays caused by data volume failing to reach the threshold. This ensures that even small updates or instructions have end-to-end latency limited to a controllable and predictable time range (i.e., not exceeding the packet merging time window), guaranteeing the real-time requirements of distributed training collaboration. This strategy cleverly adjusts its behavior dynamically based on actual data traffic. When data traffic is high, it prioritizes batch processing to maximize throughput; when data traffic is low, it prioritizes timeliness to control latency. This adaptability allows the communication protocol to maintain high transmission performance under different network conditions and training stages. This packet merging strategy solves the communication overhead problem caused by frequent small packet transmissions, providing stable and efficient data transmission guarantees for distributed training of large models.

[0081] In an optional embodiment, the aggregation of model parameters received from multiple client devices to generate a global model includes: According to the preset privacy protection policy, perform the following operations: If the strategy is configured to add noise to the gradient, then noise is injected into the gradients received from each client device, and the gradients after the noise injection are aggregated and calculated to generate the global model. If the strategy is configured to add noise to the aggregation process, then the updated model parameters received from each client device are aggregated to obtain the parameter aggregation result, and noise is injected into the parameter aggregation result to generate the global model.

[0082] Specifically, before starting the training task, the administrator configures a privacy protection policy on the server, specifying the noise addition method to be used in this training. At the start of each aggregation round, the server determines which path to execute based on this configuration. When the privacy protection policy is configured to add noise to the gradients, the server receives the raw gradients calculated locally by each client device and verifies the validity of their format and signature. The server injects Gaussian noise into the gradients received from each client device. Specifically, for the gradient tensor uploaded by a client device... The noise addition process is ,in, Indicates a normal distribution. It is the noise standard deviation that controls the strength of privacy protection, where I is the identity matrix. This is the gradient after adding noise. The server aggregates the gradients from all client devices using the noisy gradients. The aggregation calculation can employ a weighted average method, resulting in the aggregated gradient. ,in, It is the weight of the i-th client device. This is the gradient after noise is injected into the i-th client device, where n is the number of client devices. The weights of the client devices can be set according to actual needs, such as their data volume. The server aggregates the gradients. The global model from the previous round is applied, and the parameters are updated by an optimizer (such as SGD or Adam) to generate a new global model.

[0083] When the privacy protection policy is configured to add noise to the aggregation process, the server receives updated model parameters (i.e., the model state after gradient application has been completed locally on each client device) uploaded from each client device and verifies their consistency. The server then processes these updated model parameters... Perform aggregation calculations to obtain the initial global model parameters, i.e., the parameter aggregation results. For example, using a weighted average: The server aggregates the parameters obtained from the aggregation. Injecting Gaussian noise, parameters after noise addition The parameters after adding noise The values ​​are directly assigned to the model to generate the final global model.

[0084] Both strategies of this invention effectively obfuscate the exact contribution of individual client devices to the global model by injecting precisely calculated Gaussian noise. This prevents attackers from reliably inferring the original training data of any specific client device by analyzing the final global model or intermediate gradients, thus theoretically providing a strict and quantifiable privacy protection boundary. Through preset strategies, system administrators can flexibly choose the timing of noise addition based on the different privacy and accuracy requirements of the task. Adding noise to gradients protects information earlier but may significantly impact model convergence; adding noise to the aggregation results provides a relatively accurate intermediate model state first, followed by noise protection, which is generally more performance-friendly for the same noise level. This flexibility allows the solution to adapt to application scenarios with varying privacy sensitivities, such as healthcare and finance. Deeply integrating the privacy protection mechanism into the core process of model aggregation and standardizing and streamlining it fundamentally improves the security baseline of the entire distributed training system. By merging privacy protection capabilities with efficient model aggregation, it ensures that high-performance global models are produced while strictly adhering to data privacy and security requirements.

[0085] The performance of the federated split learning method for large model collaborative training of the present invention was tested. The hardware and software environment of the experiment is shown in Table 1.

[0086] Table 1 To leverage more powerful computing resources for distributed training, the developed code was deployed on AutoDL's servers. AutoDL provides a pre-configured deep learning environment, supports multiple frameworks, and offers various instance types for users to choose from. The experiment used two devices, each containing one GPU and one CPU, to simulate client devices, and one device containing one CPU to simulate the server.

[0087] The specific experimental environment is shown in Table 1, where the GPU memory is 120GB. The performance data displayed on the front end after the system is running is as follows: Figure 7 As shown, Figure 7 The data shows the training progress and metric trends (trending_up). The training progress is at Epoch 7 / 10, the current batch size is 145 / 200, the training loss is 0.2437, the training accuracy is 87.5%, the validation loss is 0.3182, the validation accuracy is 85.2%, the training time is 01:45:32, and the estimated time remaining (ETA) is 00:32:15. The show_chart also displays the trends of training metrics, including loss, accuracy, memory usage, and time performance. The training loss and validation loss show a continuous decrease with increasing Epoch count. Experimental results show that as the number of training rounds increases, both the training loss and validation loss of the model show a continuous downward trend, verifying that the proposed method effectively improves the model's performance and generalization ability on both the training and validation sets while ensuring the model's convergence stability.

[0088] Table 2 compares and evaluates the experimental data of existing representative federated learning and split learning systems with the data obtained from the test of this method. After analysis, the training accuracy of the method is between that of federated learning and split learning, and the loss value of training and validation is better than most split learning methods and close to that of federated learning.

[0089] Table 2 The federated split learning device for large model collaborative training provided by the present invention will be described below. The federated split learning device for large model collaborative training described below can be referred to in correspondence with the federated split learning method for large model collaborative training described above.

[0090] The federated splitting learning device for large model collaborative training provided by this invention refers to... Figure 8 As shown, it includes: The model partitioning management module 210 is used to split the large model into multiple sub-models based on the computation graph of the large model to be trained using a computation graph partitioning algorithm; and to distribute the multiple sub-models to the corresponding client devices respectively. The model aggregation and update module 220 is used to receive model parameters uploaded by the multiple client devices and trained based on local data; and to aggregate the received model parameters from the multiple client devices to generate a global model, wherein the global model is a large model that integrates the training knowledge of all client devices.

[0091] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions from the memory 330 to execute a federated split learning method for large model collaborative training.

[0092] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the federated split learning method for large model collaborative training provided by the above methods.

[0094] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the federated split learning method for large model collaborative training provided by the methods described above.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated splitting learning method for collaborative training of large models, characterized in that, include: Based on the computation graph of the large model to be trained, the large model is split into multiple sub-models using a computation graph segmentation algorithm. The multiple sub-models are distributed to their respective client devices; Receive model parameters uploaded by the multiple client devices, which are trained based on local data; The model parameters received from multiple client devices are aggregated to generate a global model, which is a large model that integrates the training knowledge of all client devices.

2. The federated splitting learning method for collaborative training of large models according to claim 1, characterized in that, The computational graph based on the large model to be trained is divided into multiple sub-models using a computational graph segmentation algorithm, including: By symbolic tracing or model structure parsing, a directed acyclic graph corresponding to the large model is generated; Analyze the topology of the directed acyclic graph and estimate the computational and storage overhead of each node in the graph; Based on the computation and storage overhead of each node and the resource status of the client device, a partitioning strategy is selected; wherein, the partitioning strategy includes a memory-constrained partitioning strategy, a communication-optimized partitioning strategy, or a hybrid partitioning strategy, and the resource status of the client device includes one or more of the following: computing power, memory capacity, and network bandwidth. Based on the selected segmentation strategy, segmentation points are determined and segmentation is performed on the directed acyclic graph to generate the multiple sub-models; When segmenting the directed acyclic graph, the segmentation operation is preferentially performed at the boundary of the Transformer module.

3. The federated splitting learning method for collaborative training of large models according to claim 2, characterized in that, When the large model is transformed into a directed acyclic graph, the large model is divided into multiple predefined components at the module level. The predefined components include an embedding layer, an encoding layer, a pooling layer, and a classification layer.

4. The federated splitting learning method for collaborative training of large models according to claim 1, characterized in that, The method further includes: Configure a communication protocol for the client device, the communication protocol supporting a multi-channel parallel transmission mechanism, the multi-channel including: The first channel is configured to transmit activation values ​​and control commands generated by the forward propagation of the transmission model. The second channel is configured to transmit gradient data generated by the backpropagation of the model. The third channel is configured to transmit logs and monitoring data during system operation; Among them, the data transmission priority of the first channel is higher than that of the second channel, and the data transmission priority of the second channel is higher than that of the third channel.

5. The federated splitting learning method for collaborative training of large models according to claim 4, characterized in that, The communication protocol employs a packet-splitting strategy for data transmission. This strategy is executed based on a packet-splitting threshold and a time window parameter, and is achieved through the following mechanism: When the accumulated amount of data to be sent reaches the packetization threshold, the accumulated data is immediately packaged and sent. If the data merging threshold is not reached, the accumulated data will be packaged and sent when the data merging time window expires.

6. The federated splitting learning method for collaborative training of large models according to claim 1, characterized in that, The aggregation of model parameters received from multiple client devices generates a global model, including: According to the preset privacy protection policy, perform the following operations: If the strategy is configured to add noise to the gradient, then noise is injected into the gradients received from each client device, and the gradients after the noise injection are aggregated and calculated to generate the global model. If the strategy is configured to add noise to the aggregation process, then the updated model parameters received from each client device are aggregated to obtain the parameter aggregation result, and noise is injected into the parameter aggregation result to generate the global model.

7. A federated split learning device for collaborative training of large models, characterized in that, include: The model partitioning management module is used to split the large model to be trained into multiple sub-models based on the computation graph of the large model; And, the multiple sub-models are distributed to their respective client devices; The model aggregation and update module is used to receive model parameters uploaded by the multiple client devices and trained based on local data; and to aggregate the received model parameters from the multiple client devices to generate a global model, which is a large model that integrates the training knowledge of all client devices.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the federated split learning method for collaborative training of large models as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the federated split learning method for collaborative training of large models as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the federated split learning method for collaborative training of large models as described in any one of claims 1 to 6.

Images