Federal neural network parallel optimization method, system and device
By constructing topology information and generating parallel training strategies, the problem of insufficient GPU resource utilization in federated learning is solved, achieving efficient neural network training, improving resource utilization, and ensuring communication security.
Patent Information
- Application Number
- CN202511008732.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-22
- Publication Date
- 2025-11-14
AI Technical Summary
Existing federated learning frameworks suffer from insufficient GPU resource utilization during large-scale training, leading to resource waste and low training efficiency.
By setting up a GPU distributed training architecture that supports multiple nodes, the configuration resource information of each participant is obtained, the topology information is constructed, and a parallel training strategy is generated based on the topology information and the data to be trained, so that GPUs on different nodes can work together to train the neural network.
It achieves comprehensive awareness of the GPU connection relationships and communication capabilities inside and outside the node, reduces cross-node communication overhead, makes reasonable use of hardware resources, improves the overall system resource utilization efficiency, and ensures communication security.
Smart Images
Figure CN120952103A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of federated model training, and in particular to a method, system and apparatus for parallel optimization of federated neural networks. Background Technology
[0002] With increasing demands for data privacy, Federated Learning (FL) has gradually become the mainstream solution for cross-institutional machine learning model training. Existing federated learning frameworks typically rely on a single server or a single graphics processing unit (GPU) to complete local model training, and then transmit and aggregate model parameters or gradients via the network in an encrypted manner. This approach can meet basic needs when the model size is small or the number of participants is limited, but in large-scale training, it still suffers from the problem of insufficient GPU resource utilization leading to resource waste and low training efficiency.
[0003] There is currently no effective solution to the above problems in existing technologies. Summary of the Invention
[0004] To address the aforementioned issues, this invention provides a federated neural network parallel optimization method, system, and apparatus. By setting up a GPU distributed training architecture that supports multiple nodes, GPUs on different nodes can collaboratively train the neural network. At the same time, training strategies are designed according to the actual conditions of different scenarios to maximize the utilization of processing power, thereby solving the problem of low data processing efficiency in the prior art.
[0005] To achieve the above objectives, this invention provides a parallel optimization method for federated neural networks, comprising: acquiring configuration resource information of nodes corresponding to each participant; constructing topology information of each participant based on the configuration resource information; wherein the topology information includes the interconnection relationship of GPUs available to all nodes within each participant; generating a parallel training strategy based on the topology information and training data; and executing a neural network model training task on the GPUs within each participant according to the parallel training strategy to obtain a target model for the target application task.
[0006] Further optionally, the step of constructing the topology information of each participant based on the configuration resource information includes: for any participant, detecting the interconnection information of the graphics processing unit (GPU) within each node and the network connection information of the GPUs between nodes; and generating topology information based on the interconnection information within the nodes and the network connection information between the nodes.
[0007] Further optionally, the step of generating a parallel training strategy based on the topology information and the data to be trained includes: counting the number of all graphics processing units (GPUs) within the participating party in the topology information; and calculating the batch data size to be allocated to each GPU within the participating party based on the node network bandwidth, the data to be trained, and the number of all GPUs in the topology information.
[0008] Optionally, the step of executing the neural network model training task on the GPU within each participant includes: calculating intermediate features on the GPU within each participant; the task collaborator encrypting the intermediate features to obtain encrypted intermediate features and sending the encrypted intermediate features to the task initiator; wherein the participants include the task collaborator and the task initiator; the task initiator calculating a predicted value on its local GPU based on the prediction function, local intermediate features, and the received encrypted intermediate features; the task initiator calculating an error term on its local GPU based on the predicted value and local labels, calculating the initiator gradient based on the error term, and updating local parameters; the task initiator encrypting the error term and sending it to the task collaborator; the task collaborator calculating the collaborator gradient on its local GPU based on the decrypted error term, and updating local parameters.
[0009] On the other hand, the present invention also provides a federated neural network parallel optimization system, characterized in that it includes: a topology structure establishment module, used to obtain the configuration resource information of the nodes corresponding to each participant, and construct the topology structure information of each participant according to the configuration resource information; wherein, the topology structure information includes the interconnection relationship of the GPUs available to all nodes in each participant; a training strategy generation module, used to generate a parallel training strategy according to the topology structure information and the data to be trained; and a model training module, used to execute a neural network model training task on the GPUs in each participant according to the parallel training strategy, to obtain a target model for the target application task.
[0010] Further optionally, the topology establishment module includes: a connection detection submodule, used to detect, for any participant, the interconnection information of the graphics processing unit (GPU) within each node, and the network connection information of the GPUs between nodes; and a topology information generation submodule, used to generate topology information based on the interconnection information within the nodes and the network connection information between the nodes.
[0011] Further optionally, the training strategy generation module includes: a statistics submodule, used to count the number of all graphics processing units (GPUs) within the participants in the topology information; and a batch data size determination submodule, used to calculate the batch data size to be allocated to each GPU within the participants based on the node network bandwidth, the data to be trained, and the number of all GPUs in the topology information.
[0012] Further optionally, the model training module includes: an intermediate feature calculation submodule, used to calculate intermediate features on the GPUs of each participant, whereby the task collaborator encrypts the intermediate features to obtain encrypted intermediate features and sends the encrypted intermediate features to the task initiator; wherein the participants include the task collaborator and the task initiator; a prediction submodule, used by the task initiator on its local GPU to calculate a prediction value based on a prediction function, local intermediate features, and the received encrypted intermediate features; a first parameter update submodule, used by the task initiator on its local GPU to calculate an error term based on the prediction value and local labels, calculate the initiator gradient based on the error term, and update local parameters; and a second parameter update submodule, used by the task initiator to encrypt the error term and send it to the task collaborator, whereby the task collaborator calculates the collaborator gradient on its local GPU based on the decrypted error term and updates local parameters.
[0013] On the other hand, the present invention also provides a federated neural network parallel optimization device for cross-node GPU collaboration, including the aforementioned federated neural network parallel optimization system.
[0014] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described federated neural network parallel optimization method.
[0015] The above technical solution has the following beneficial effects: it collects GPU-related information from multiple nodes of each participating party to construct topology information, thereby enabling a comprehensive perception of the GPU connection relationships and communication capabilities inside and outside the nodes, reducing cross-node communication overhead; based on topology information and model requirements, it generates parallel training strategies, rationally utilizes hardware resources, and improves the overall system resource utilization efficiency; and it combines multi-party secure computation with distributed parallel processing to ensure communication security. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is a flowchart of the parallel optimization method for federated neural networks provided in an embodiment of the present invention;
[0018] Figure 2 This is a flowchart of the topology information construction method provided in the embodiments of the present invention;
[0019] Figure 3 This is a flowchart of the parallel training strategy generation method provided in the embodiments of the present invention;
[0020] Figure 4 This is a flowchart of the neural network model training method provided in the embodiments of the present invention;
[0021] Figure 5 This is a schematic diagram of the structure of the federated neural network parallel optimization system provided in an embodiment of the present invention;
[0022] Figure 6 This is a schematic diagram of the topology establishment module provided in an embodiment of the present invention;
[0023] Figure 7 This is a schematic diagram of the training strategy generation module provided in an embodiment of the present invention;
[0024] Figure 8 This is a schematic diagram of the structure of the model training module provided in an embodiment of the present invention.
[0025] Figure labeling: 100-Topology structure establishment module; 1001-Connection detection submodule; 1002-Topology information generation submodule; 200-Training strategy generation module; 2001-Statistics submodule; 2002-Batch data size determination submodule; 300-Model training module; 3001-Intermediate feature calculation submodule; 3002-Prediction submodule; 3003-First parameter update submodule; 3004-Second parameter update submodule. Detailed Implementation
[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0027] To address the problem that existing federated neural networks struggle to maintain computational efficiency while ensuring privacy and security, this invention provides a parallel optimization method for federated neural networks. Figure 1 This is a flowchart of the parallel optimization method for federated neural networks provided in an embodiment of the present invention, such as... Figure 1 As shown, the method includes:
[0028] S1. Obtain the configuration resource information of the node corresponding to each participant, and construct the topology information of each participant based on the configuration resource information; wherein, the topology information includes the interconnection relationship of the graphics processing units (GPUs) available to all nodes in each participant.
[0029] Vertical federated learning is a federated learning approach in which multiple participants, with overlapping users but different features, jointly train the system based on encryption technology.
[0030] For each participant in vertical federated learning, each participant may have one or more nodes, which refer to the physical server units that make up the participant. Each node may include multiple graphics processing units (GPUs) or it may have only one GPU.
[0031] First, obtain the configuration resource information of each participant. The configuration resource information includes: the number of GPUs, computing power, video memory capacity, network connection parameters and NCCL communication library availability on each node.
[0032] The topology information of GPU resources is constructed by designing a topology analysis function. This topology information includes at least the interconnections of all available GPUs within the participating parties. The topology information can be represented using a graph structure.
[0033] Based on this, a process group partitioning scheme can be further generated, such as allocating a process to each graphics processing unit (GPU) to provide communication and task scheduling support for parallel training.
[0034] S2. Generate a parallel training strategy based on the topology information and the data to be trained;
[0035] Key factors included in the topology information are the number of nodes within each participant and the number of graphics processing units (GPUs) configured on each node. Based on the actual detection results, it is determined whether the current participant has the conditions to implement parallel training. The parallel training strategy is enabled when any of the following conditions are met:
[0036] A single node exists that is configured with multiple graphics processing units (GPUs), enabling single-node multi-GPU parallel training.
[0037] There are multiple nodes, and each node is configured with at least one graphics processing unit (GPU), enabling multi-node distributed parallel training.
[0038] Additionally, if a participant has only one node, and that node contains only one graphics processing unit (GPU), training can also be performed in this way.
[0039] If the above parallel conditions are met, allocate an appropriate amount of training data to each graphics processing unit (GPU).
[0040] S3. According to the parallel training strategy, the neural network model training task is executed on the graphics processing unit (GPU) of each participant to obtain the target model for the target application task.
[0041] The parallel training strategies generated by each participant are collaboratively executed on their respective graphics processing units (GPUs) to train the neural network model. The training task includes basic processes such as forward propagation, back propagation, gradient calculation, and parameter updates.
[0042] Multiple graphics processing units (GPUs) can work collaboratively based on the process group partitioning results, process different data slices in a data parallel manner, and communicate and synchronize according to the set parameter update interval and gradient aggregation frequency.
[0043] During training, techniques such as secure multi-party computation and homomorphic encryption can be used to encrypt local gradient information, ensuring that the original data content is not leaked during parameter exchange and aggregation. Through multiple rounds of gradient aggregation and parameter updates, each participant continuously optimizes its local model parameters, ultimately obtaining a target neural network model for the intended application task.
[0044] As an optional implementation method, Figure 2 This is a flowchart of the topology information construction method provided in the embodiments of the present invention, such as... Figure 2 As shown, the topology information for each participant is constructed based on the configured resource information, including:
[0045] S101. For any participant, detect the interconnection information of the graphics processing unit (GPU) within each node, and the network connection information of the GPUs between nodes.
[0046] For each participant in the training of the neural network model, all nodes within it are traversed, and the resource interconnection relationship of the graphics processing unit (GPU) in each node is detected.
[0047] The interconnect information of the graphics processing units (GPUs) within a node includes: the physical connection method between GPUs, such as whether they are connected via high-speed interconnect buses such as PCI Express (PCIe) or NVLink; whether each GPU has the ability to exchange data directly; and performance parameters such as interconnect bandwidth and communication latency.
[0048] Simultaneously, the network connection information of the graphics processing units (GPUs) between the nodes within the participating party is detected. Inter-node connections are typically established via Ethernet, InfiniBand, or similar methods. The network connection information between the graphics processing units includes: the network type between nodes; the actual available bandwidth (in Gbps or Mbps); and the round-trip time (RTT) of network communication.
[0049] The detection results are used to characterize the data transfer capabilities between each graphics processing unit (GPU) and other GPUs, as well as their physical or logical location relationships.
[0050] S102. Generate topology information based on the interconnection information within nodes and the network connection information between nodes.
[0051] Based on the collected GPU connection information, the internal topology information of the participating parties is constructed. Topology information is an abstract representation structure used to express resource connection relationships, usually implemented using a graph structure model. In this model, the vertices of the graph represent individual GPUs, and the edges represent the connection relationships between GPUs. Each edge can be attached with attributes to identify the connection's bandwidth, communication latency, or priority.
[0052] Based on the direct connections of GPUs within a node (e.g., whether they are NVLink direct connections) and the network paths and bandwidth between nodes, an overall connectivity graph is formed. This topology information provides a foundation for subsequent parallel partitioning of training strategies, communication optimization, and process group configuration.
[0053] The generated topology information is further divided into process groups. The division of process groups follows certain rules. For example, graphics processing units with communication latency less than a preset threshold and communication bandwidth higher than a preset lower limit are divided into the same process group; if there is no direct connection between graphics processing units that meets the conditions, they are divided into different process groups.
[0054] As an optional implementation method, Figure 3 This is a flowchart of the parallel training strategy generation method provided in the embodiments of the present invention, such as... Figure 3 As shown, a parallel training strategy is generated based on the topology information and the data to be trained, including:
[0055] S201, Count the number of all graphics processing units (GPUs) within the participating parties in the statistical topology information;
[0056] S202. Based on the node network bandwidth, training data, and the number of all graphics processing units (GPUs) in the topology information, calculate the batch data size to be allocated to each GPU within the participating party.
[0057] First, the constructed topology information is parsed to identify all computing nodes and their configured GPUs within each participant. Each participant may consist of one or more physical servers (i.e., multiple nodes), and each node may contain one or more GPUs. Based on this, the total number of available GPUs across all nodes is counted, and the unique identifier of each GPU and its associated node information are recorded. This statistical result will serve as the basis for subsequent data load partitioning.
[0058] In addition, the operating status, memory size, and communication capabilities of each GPU can be optionally recorded to comprehensively consider the adaptability of training task scheduling.
[0059] Then, based on the number of GPUs in each graphics processing unit contained in the topology information and the network bandwidth between their nodes, the data training load that each GPU should bear is reasonably allocated, specifically reflected in the calculation and distribution of the batch size of the data passed in each round of training.
[0060] Batch size refers to the number of samples fed into the neural network model for training in each iteration. Its size directly affects the GPU's memory usage, computational load, and training efficiency.
[0061] Specifically, the training data is divided equally according to the total number of GPUs. Then, it is adjusted according to the network bandwidth of the node to which each GPU belongs. GPUs with higher bandwidth can be allocated a larger batch size, while GPUs with lower bandwidth can have a smaller batch size.
[0062] Furthermore, to reduce communication volume, the gradients of various parameters during aggregation transmission can be compressed.
[0063] As an optional implementation, the computational load or memory usage of each graphics processing unit (GPU) is monitored during subsequent training. If the computational load or memory usage reaches a preset value, data sharding and data allocation are re-performed.
[0064] As an optional implementation method, Figure 4This is a flowchart of the neural network model training method provided in an embodiment of the present invention, such as... Figure 4 As shown, the neural network model training task is performed on the graphics processing unit (GPU) within each participant, including:
[0065] S301. The intermediate features are calculated on the graphics processing units (GPUs) within each participating party. The task collaborator encrypts the intermediate features to obtain encrypted intermediate features and sends the encrypted intermediate features to the task initiator. The participating parties include the task collaborator and the task initiator.
[0066] Each participant calculates its local intermediate features based on its local dataset and local initial parameters. For participants who are task collaborators, after completing the calculation of the intermediate features, they encrypt the intermediate features to obtain encrypted intermediate features, and then send the encrypted intermediate features to the task initiator through a secure channel.
[0067] S302. The task initiator calculates the predicted value on the local graphics processing unit (GPU) based on the prediction function, local intermediate features, and received encrypted intermediate features.
[0068] The task initiator decrypts the encrypted intermediate features and performs forward propagation on its local graphics processing unit (GPU). That is, based on the preset prediction function, it calculates the model output prediction value by combining the intermediate features calculated locally and the received intermediate features.
[0069] S303. The task initiator calculates the error term based on the predicted value and local label on the local graphics processing unit (GPU), calculates the initiator gradient based on the error term, and updates the local parameters.
[0070] Based on the above predicted values and local labels, the task initiator performs error term (i.e. loss) calculation on its local graphics processing unit (GPU) and performs backpropagation based on the error term to obtain the local gradient (i.e., the initiator gradient), thereby updating the parameters of its local neural network model.
[0071] S304. The task initiator encrypts the error term and sends it to the task collaborator. The task collaborator calculates the collaborator gradient on its local graphics processing unit (GPU) based on the decrypted error term and updates the local parameters.
[0072] The task initiator encrypts the calculated error term and sends the encrypted error term to the task collaborator. After decrypting the error term locally, the task collaborator performs backpropagation on its local GPU to calculate the collaborator gradient and update the local model parameters accordingly.
[0073] The above gradient aggregation process can be performed by first performing intra-node gradient aggregation, and then performing node gradient aggregation.
[0074] The local model parameters are updated based on the gradients obtained from the participants, completing one training round and preparing for the next iteration. This process is repeated until the model converges or reaches the predetermined number of iterations, resulting in the final model, i.e., the target model. This process can be based on cryptographic protocols (such as secure multi-party computation, homomorphic encryption, etc.) to protect data privacy.
[0075] This invention also provides a federated neural network parallel optimization system. Figure 5 This is a schematic diagram of the structure of the federated neural network parallel optimization system provided in an embodiment of the present invention, as shown below. Figure 5 As shown, it includes:
[0076] The topology building module 100 is used to obtain the configuration resource information of the nodes corresponding to each participant, and to construct the topology information of each participant based on the configuration resource information; wherein, the topology information includes the interconnection relationship of the graphics processing units (GPUs) available to all nodes in each participant.
[0077] Vertical federated learning is a federated learning approach in which multiple participants, with overlapping users but different features, jointly train the system based on encryption technology.
[0078] For each participant in vertical federated learning, each participant may have one or more nodes, which refer to the physical server units that make up the participant. Each node may include multiple graphics processing units (GPUs) or it may have only one GPU.
[0079] First, obtain the configuration resource information of each participant. The configuration resource information includes: the number of GPUs, computing power, video memory capacity, network connection parameters and NCCL communication library availability on each node.
[0080] The topology information of GPU resources is constructed by designing a topology analysis function. This topology information includes at least the interconnections of all available GPUs within the participating parties. The topology information can be represented using a graph structure.
[0081] Based on this, a process group partitioning scheme can be further generated, such as allocating a process to each graphics processing unit (GPU) to provide communication and task scheduling support for parallel training.
[0082] The training strategy generation module 200 is used to generate parallel training strategies based on topology information and training data.
[0083] Key factors included in the topology information are the number of nodes within each participant and the number of graphics processing units (GPUs) configured on each node. Based on the actual detection results, it is determined whether the current participant has the conditions to implement parallel training. The parallel training strategy is enabled when any of the following conditions are met:
[0084] A single node exists that is configured with multiple graphics processing units (GPUs), enabling single-node multi-GPU parallel training.
[0085] There are multiple nodes, and each node is configured with at least one graphics processing unit (GPU), enabling multi-node distributed parallel training.
[0086] Additionally, if a participant has only one node, and that node contains only one graphics processing unit (GPU), training can also be performed in this way.
[0087] If the above parallel conditions are met, allocate an appropriate amount of training data to each graphics processing unit (GPU).
[0088] The model training module 300 is used to perform neural network model training tasks on the graphics processing unit (GPU) of each participant according to the parallel training strategy, so as to obtain the target model for the target application task.
[0089] The parallel training strategies generated by each participant are collaboratively executed on their respective graphics processing units (GPUs) to train the neural network model. The training task includes basic processes such as forward propagation, back propagation, gradient calculation, and parameter updates.
[0090] Multiple graphics processing units (GPUs) can work collaboratively based on the process group partitioning results, process different data slices in a data parallel manner, and communicate and synchronize according to the set parameter update interval and gradient aggregation frequency.
[0091] During training, techniques such as secure multi-party computation and homomorphic encryption can be used to encrypt local gradient information, ensuring that the original data content is not leaked during parameter exchange and aggregation. Through multiple rounds of gradient aggregation and parameter updates, each participant continuously optimizes its local model parameters, ultimately obtaining a target neural network model for the intended application task.
[0092] As an optional implementation method, Figure 6 This is a schematic diagram of the topology establishment module provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the topology establishment module 100 includes:
[0093] The connection detection submodule 1001 is used to detect the interconnection information of the graphics processing unit (GPU) within each node and the network connection information of the GPUs between nodes for any participant.
[0094] For each participant in the training of the neural network model, all nodes within it are traversed, and the resource interconnection relationship of the graphics processing unit (GPU) in each node is detected.
[0095] The interconnect information of the graphics processing units (GPUs) within a node includes: the physical connection method between GPUs, such as whether they are connected via high-speed interconnect buses such as PCI Express (PCIe) or NVLink; whether each GPU has the ability to exchange data directly; and performance parameters such as interconnect bandwidth and communication latency.
[0096] Simultaneously, the network connection information of the graphics processing units (GPUs) between the nodes within the participating party is detected. Inter-node connections are typically established via Ethernet, InfiniBand, or similar methods. The network connection information between the graphics processing units includes: the network type between nodes; the actual available bandwidth (in Gbps or Mbps); and the round-trip time (RTT) of network communication.
[0097] The detection results are used to characterize the data transfer capabilities between each graphics processing unit (GPU) and other GPUs, as well as their physical or logical location relationships.
[0098] The topology information generation submodule 1002 is used to generate topology information based on the interconnection information within nodes and the network connection information between nodes.
[0099] Based on the collected GPU connection information, the internal topology information of the participating parties is constructed. Topology information is an abstract representation structure used to express resource connection relationships, usually implemented using a graph structure model. In this model, the vertices of the graph represent individual GPUs, and the edges represent the connection relationships between GPUs. Each edge can be attached with attributes to identify the connection's bandwidth, communication latency, or priority.
[0100] Based on the direct connections of GPUs within a node (e.g., whether they are NVLink direct connections) and the network paths and bandwidth between nodes, an overall connectivity graph is formed. This topology information provides a foundation for subsequent parallel partitioning of training strategies, communication optimization, and process group configuration.
[0101] The generated topology information is further divided into process groups. The division of process groups follows certain rules. For example, graphics processing units with communication latency less than a preset threshold and communication bandwidth higher than a preset lower limit are divided into the same process group; if there is no direct connection between graphics processing units that meets the conditions, they are divided into different process groups.
[0102] As an optional implementation method, Figure 7 This is a schematic diagram of the training strategy generation module provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the training policy generation module 200 includes:
[0103] The statistics submodule 2001 is used to count the number of all graphics processing units (GPUs) in the participating parties in the topology information.
[0104] The batch data size calculation submodule 2002 is used to calculate the batch data size to be allocated to each GPU in the participating party based on the node network bandwidth, the data to be trained, and the number of all graphics processing units (GPUs) in the topology information.
[0105] First, the constructed topology information is parsed to identify all computing nodes and their configured GPUs within each participant. Each participant may consist of one or more physical servers (i.e., multiple nodes), and each node may contain one or more GPUs. Based on this, the total number of available GPUs across all nodes is counted, and the unique identifier of each GPU and its associated node information are recorded. This statistical result will serve as the basis for subsequent data load partitioning.
[0106] In addition, the operating status, memory size, and communication capabilities of each GPU can be optionally recorded to comprehensively consider the adaptability of training task scheduling.
[0107] Then, based on the number of GPUs in each graphics processing unit contained in the topology information and the network bandwidth between their nodes, the data training load that each GPU should bear is reasonably allocated, specifically reflected in the calculation and distribution of the batch size of the data passed in each round of training.
[0108] Batch size refers to the number of samples fed into the neural network model for training in each iteration. Its size directly affects the GPU's memory usage, computational load, and training efficiency.
[0109] Specifically, the training data is divided into equal portions based on the total number of GPUs. Then, adjustments are made based on the network bandwidth of the node to which each GPU belongs. GPUs with higher bandwidth are allocated larger batch sizes, while those with lower bandwidth have smaller batch sizes. Furthermore, to reduce communication overhead, the gradients of various parameters during aggregation and transmission can be compressed.
[0110] As an optional implementation, the computational load or memory usage of each graphics processing unit (GPU) is monitored during subsequent training. If the computational load or memory usage reaches a preset value, data sharding and data allocation are re-performed.
[0111] As an optional implementation, the model training module 300 includes:
[0112] The intermediate feature calculation submodule 3001 is used to calculate intermediate features on the graphics processing units (GPUs) within each participating party. The task collaborator encrypts the intermediate features to obtain encrypted intermediate features and sends the encrypted intermediate features to the task initiator. The participating parties include the task collaborator and the task initiator.
[0113] Each participant calculates its local intermediate features based on its local dataset and local initial parameters. For participants who are task collaborators, after completing the calculation of the intermediate features, they encrypt the intermediate features to obtain encrypted intermediate features, and then send the encrypted intermediate features to the task initiator through a secure channel.
[0114] The prediction submodule 3002 is used by the task initiator to calculate the prediction value on the local graphics processing unit (GPU) based on the prediction function, local intermediate features, and received encrypted intermediate features.
[0115] The task initiator decrypts the encrypted intermediate features and performs forward propagation on its local graphics processing unit (GPU). That is, based on the preset prediction function, it calculates the model output prediction value by combining the intermediate features calculated locally and the received intermediate features.
[0116] The first parameter update submodule 3003 is used by the task initiator to calculate the error term based on the predicted value and local label on the local graphics processing unit (GPU), calculate the initiator gradient based on the error term, and update the local parameters.
[0117] Based on the above predicted values and local labels, the task initiator performs error term (i.e. loss) calculation on its local graphics processing unit (GPU) and performs backpropagation based on the error term to obtain the local gradient (i.e., the initiator gradient), thereby updating the parameters of its local neural network model.
[0118] The second parameter update submodule 3004 is used by the task initiator to encrypt the error term and send it to the task collaborator. The task collaborator calculates the collaborator gradient on its local graphics processing unit (GPU) based on the decrypted error term and updates the local parameters.
[0119] The task initiator encrypts the calculated error term and sends the encrypted error term to the task collaborator. After decrypting the error term locally, the task collaborator performs backpropagation on its local GPU to calculate the collaborator gradient and update the local model parameters accordingly.
[0120] The above gradient aggregation process can be performed by first performing intra-node gradient aggregation, and then performing node gradient aggregation.
[0121] The local model parameters are updated based on the gradients obtained from the participants, completing one training round and preparing for the next iteration. This process is repeated until the model converges or reaches the predetermined number of iterations, resulting in the final model, i.e., the target model. This process can be based on cryptographic protocols (such as secure multi-party computation, homomorphic encryption, etc.) to protect data privacy.
[0122] This invention also provides a cross-node GPU collaborative federated neural network parallel optimization device, including the aforementioned federated neural network parallel optimization system.
[0123] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described federated neural network parallel optimization method.
[0124] The aforementioned storage media include, but are not limited to: optical discs, floppy disks, hard disks, and rewritable memory.
[0125] The above technical solution has the following beneficial effects: it collects GPU-related information from multiple nodes of each participating party to construct topology information, thereby enabling a comprehensive perception of the GPU connection relationships and communication capabilities inside and outside the nodes, reducing cross-node communication overhead; based on topology information and model requirements, it generates parallel training strategies, rationally utilizes hardware resources, and improves the overall system resource utilization efficiency; and it combines multi-party secure computation with distributed parallel processing to ensure communication security.
[0126] The above-described specific embodiments of the invention further illustrate the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above content is only for specific embodiments of the invention and is not intended to limit the scope of protection of the invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of protection of the invention.
Claims
1. A parallel optimization method for federated neural networks, characterized in that, include: Obtain the configuration resource information of the node corresponding to each participant, and construct the topology information of each participant based on the configuration resource information; wherein, the topology information includes the interconnection relationship of the graphics processing units (GPUs) available to all nodes within each participant; A parallel training strategy is generated based on the topology information and the data to be trained. According to the parallel training strategy, a neural network model training task is performed on the graphics processing unit (GPU) of each participant to obtain a target model for the target application task.
2. The parallel optimization method for federated neural networks according to claim 1, characterized in that, The step of constructing the topology information for each participant based on the configured resource information includes: For any participant, detect the interconnection information of the graphics processing units (GPUs) within each node, and the network connection information of the GPUs between nodes. Topology information is generated based on the interconnection information within the nodes and the network connection information between the nodes.
3. The parallel optimization method for federated neural networks according to claim 1, characterized in that, The step of generating a parallel training strategy based on the topology information and the data to be trained includes: Count the number of all graphics processing units (GPUs) within the participating parties in the aforementioned topology information; Based on the node network bandwidth, training data, and the number of all graphics processing units (GPUs) in the topology information, calculate the batch data size to be allocated to each GPU within the participating party.
4. The parallel optimization method for federated neural networks according to claim 1, characterized in that, The step of performing neural network model training tasks on the graphics processing unit (GPU) within each participant includes: Intermediate features are computed on the GPUs within each participating party. The task collaborator encrypts the intermediate features to obtain encrypted intermediate features, and then sends the encrypted intermediate features to the task initiator. The participating parties include the task collaborator and the task initiator. The task initiator calculates the predicted value on its local graphics processing unit (GPU) based on the prediction function, local intermediate features, and received encrypted intermediate features. The task initiator calculates the error term based on the predicted value and local label on the local graphics processing unit (GPU), calculates the initiator gradient based on the error term, and updates the local parameters. The task initiator encrypts the error term and sends it to the task collaborator. The task collaborator calculates the collaborator gradient on its local graphics processing unit (GPU) based on the decrypted error term and updates its local parameters.
5. A federated neural network parallel optimization system, characterized in that, include: The topology building module is used to obtain the configuration resource information of the nodes corresponding to each participant, and construct the topology information of each participant based on the configuration resource information; wherein, the topology information includes the interconnection relationship of the graphics processing units (GPUs) available to all nodes in each participant; The training strategy generation module is used to generate a parallel training strategy based on the topology information and the data to be trained. The model training module is used to perform neural network model training tasks on the graphics processing unit (GPU) of each participant according to the parallel training strategy, so as to obtain a target model for the target application task.
6. The federated neural network parallel optimization system according to claim 5, characterized in that, The topology establishment module includes: The connection detection submodule is used to detect the interconnection information of the graphics processing units (GPUs) within each node and the network connection information of the GPUs between nodes for any participant. The topology information generation submodule is used to generate topology structure information based on the interconnection information within the nodes and the network connection information between the nodes.
7. The federated neural network parallel optimization system according to claim 5, characterized in that, The training strategy generation module includes: The statistics submodule is used to count the number of all graphics processing units (GPUs) within the participating parties in the topology information. The batch data size determination submodule is used to calculate the batch data size to be allocated to each GPU in the participating party based on the node network bandwidth, the data to be trained, and the number of all graphics processing units (GPUs) in the topology information.
8. The federated neural network parallel optimization system according to claim 5, characterized in that, The model training module includes: The intermediate feature calculation submodule is used to calculate intermediate features on the graphics processing units (GPUs) within each participating party. The task collaborator encrypts the intermediate features to obtain encrypted intermediate features and sends the encrypted intermediate features to the task initiator. The participating parties include the task collaborator and the task initiator. The prediction submodule is used by the task initiator to calculate the prediction value on the local graphics processing unit (GPU) based on the prediction function, local intermediate features, and received encrypted intermediate features. The first parameter update submodule is used by the task initiator to calculate the error term based on the predicted value and local label on the local graphics processing unit (GPU), calculate the initiator gradient based on the error term, and update the local parameters. The second parameter update submodule is used by the task initiator to encrypt the error term and send it to the task collaborator. The task collaborator calculates the collaborator gradient on its local graphics processing unit (GPU) based on the decrypted error term and updates the local parameters.
9. A parallel optimization device for federated neural networks with cross-node GPU collaboration, characterized in that, Including the federated neural network parallel optimization system as described in any one of claims 5-8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the federated neural network parallel optimization method as described in any one of claims 1-4.