An adaptive dynamic split federated learning method based on a double-stream expert architecture

Abstract

The application discloses a self-adaptive dynamic separated federated learning method based on a double-flow expert framework, which comprises the following steps: 1) constructing a separated federated learning system and loading a global model thereonto; 2) initializing a communication round and parameters of the separated federated learning system; 3) randomly selecting a client as an active client of a current round and distributing corresponding client parameters; 4) updating parameters of each active client based on the double-flow expert framework; 5) recombining and updating the double-flow expert framework based on a server parameter state of the active client; 6) judging whether to update hyperparameters of the current round, if yes, updating the hyperparameters, and entering step 7); 7) judging whether a preset total communication round is reached, if yes, obtaining a final global model, and if not, increasing the round by one and returning to step 3). The application solves the stability-diversity dilemma problem of the separated federated learning in a non-independent and identically distributed data scene.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and distributed machine learning technology, specifically to an adaptive dynamic split federated learning method based on a two-stream expert architecture. Background Technology

[0002] With the widespread application of artificial intelligence technology, large-scale, high-quality training data has become a key driver for improving model performance. However, in privacy-sensitive scenarios such as healthcare, finance, and smart terminals, raw data often cannot be directly collected centrally and used collaboratively, a problem known as the data silo problem. To address this issue, federated learning, as a distributed machine learning paradigm, has emerged. Its core idea is that each participant independently trains the model on local data, only uploading the model parameters or gradients to a central server for aggregation, thereby achieving collaborative training without exposing the raw data. Although federated learning effectively solves the data silo problem, the complete model training process places high demands on the client's computing and storage capabilities, making it difficult to deploy federated learning directly on resource-constrained edge devices. To alleviate this resource constraint problem, split federated learning divides the deep learning neural network into two parts: a lightweight, shallow sub-network is maintained by the federated server, distributed to each client for forward training, responsible for extracting low-level features and sending intermediate activation values ​​to the main server; the main server then hosts the computationally intensive, complex deep sub-network, completing the remaining forward and backward propagation processes. This new collaborative paradigm significantly reduces the computational and storage load on the client side while maintaining the privacy protection features of federated learning.

[0003] Based on the different synchronization processes on the master server side, existing discrete federated learning can be divided into two categories: parallel optimization paradigm and serial optimization paradigm. In the parallel paradigm, the master server maintains an independent parallel branch for each client. Each branch is trained in parallel, and the model is kept globally consistent through periodic parameter averaging. This paradigm is stable when the client data is evenly distributed, but it implicitly assumes that all layers of the network respond consistently to the federated aggregation paradigm. In the serial paradigm, the master server maintains only one deep model, and the clients update sequentially. Although this reduces synchronization overhead, it is very sensitive to the client training order and is prone to catastrophic forgetting when the client data is unevenly distributed. The common limitation of the two existing paradigms is that they both adopt static and layer-independent aggregation strategies, that is, the same update operator is applied to all server-side layers and the aggregation strategy remains unchanged throughout the training. However, recent studies have found that different layers of deep neural networks exhibit different representational characteristics. Shallow networks tend to learn low-level features that are common across tasks, such as edges and textures, and have strong transferability and cross-client consistency; while deep networks are responsible for encoding complex semantically related high-level features. When client data is heterogeneously distributed, the deep parameters corresponding to different clients exhibit increasingly significant semantic discrepancies, which accumulate gradually during training. Therefore, decoupled federated learning faces a fundamental problem of balancing stability and diversity. On the one hand, shallow networks in the early stages of training need to maintain global consistency to learn common features; on the other hand, deep networks in the later stages of training need to retain a moderate degree of local diversity to accommodate the personalized semantics of different clients and avoid destructive interference caused by forced uniformity. Existing unified static aggregation frameworks cannot organically coordinate these two contradictory goals, leading to a decrease in model accuracy and convergence stability in scenarios with non-independent and identically distributed data.

[0004] Therefore, how to design a dynamic server-side optimization mechanism for sensing network depth and training phases, and achieve an adaptive balance between stability and diversity, has become a key technical problem that urgently needs to be solved in the field of decoupled federated learning. Summary of the Invention

[0005] The purpose of this invention is to provide an adaptive dynamic split-federated learning method based on a two-stream expert architecture, comprising the following steps:

[0006] 1) Construct a separate federated learning system and mount the global deep neural network model to be learned on the separate federated learning system.

[0007] The separated federated learning system includes There are three distributed clients, a federated server, and a master server. It is a positive integer.

[0008] 2) Build a dual-stream expert architecture on the main server and initialize the communication rounds and parameters of the separate federated learning system.

[0009] The parameters of the split federated learning system include client parameters, server parameters, and hyperparameters.

[0010] 3) The federated server is from Randomly selected from distributed clients Each client acts as the active client for the current communication round, and client parameters are distributed to the corresponding active clients. Indicates the communication round. Indicates the number of active clients. This indicates the activation of the client set.

[0011] 4) Update the parameters of each activated client based on the dual-stream expert architecture to obtain the updated client parameters.

[0012] 5) Based on the status of the server-side parameters corresponding to all activated clients, the master server reorganizes and updates the dual-stream expert architecture to obtain the updated dual-stream expert architecture.

[0013] 6) Determine whether the preset communication round period has been reached. If yes, merge and update the parameters of the separate federated learning system and proceed to step 7. If no, proceed directly to step 7.

[0014] 7) Determine whether the preset total number of communication rounds has been reached. If yes, obtain the final global deep neural network model applicable to autonomous driving, industrial IoT, or smart security. If no, increment the number of communication rounds by one and return to step 3.

[0015] 8) Execute user-uploaded tasks using the final global deep neural network model.

[0016] Furthermore, the global deep neural network model to be learned includes shallow sub-networks and deep sub-networks.

[0017] The shallow sub-network is mounted on a distributed client, and the parameters of the shallow sub-network are maintained by the federated server.

[0018] The deep sub-network is hosted on the federated server and the main server, and the parameters of the deep sub-network are maintained by the main server.

[0019] Each distributed client stores a local dataset, and the local datasets of different distributed clients differ.

[0020] When applied to autonomous driving, the local dataset includes a dataset of road scene images captured by onboard cameras.

[0021] When applied to the Industrial Internet of Things (IIoT), the local dataset includes multivariate time-series datasets collected by sensors in industrial equipment.

[0022] When applied to smart security, the local dataset includes a dataset of video frame images captured by surveillance cameras.

[0023] The client parameters include parameters of the shallow sub-network.

[0024] The server-side parameters include parameters of the deep sub-network and parameters of the two-stream expert architecture.

[0025] The hyperparameters include half-life parameters and gating routing policies.

[0026] The parameters of the separate federated learning system also include the total number of communication rounds, the number of clients randomly activated in each communication round, the number of rounds of local training for each distributed client, the learning rate, and the batch size.

[0027] Furthermore, when the final global deep neural network model is applied to autonomous driving, the input is road scene images captured by the vehicle's camera, and the output is object detection boxes and category probabilities. The task uploaded by the user is to perform real-time object detection on the perception data collected by the autonomous vehicle.

[0028] When the final global deep neural network model is applied to the Industrial Internet of Things (IIoT), the input is multivariate time series data collected by sensors of industrial equipment, and the output is equipment operating status classification or fault warning signal. The task uploaded by the user is to diagnose faults in the real-time sensor data uploaded by the industrial equipment.

[0029] When the final global deep neural network model is applied to smart security, the input is video frame images captured by surveillance cameras, and the output is target detection boxes and category or abnormal behavior recognition results. The task uploaded by the user is to perform target detection or abnormal behavior recognition on the uploaded surveillance video frames.

[0030] Furthermore, the dual-stream expert architecture includes a master expert module and N reorganization expert modules, where N is a positive integer.

[0031] The steps for updating parameters for each active client based on the dual-stream expert architecture are as follows:

[0032] 4.1) Calculate the dynamic participation weight based on the current communication round, as shown below:

[0033] (1)

[0034] In the formula, Indicates the communication round. Indicates the first Dynamic participation weights for each communication round. This represents the half-life parameter.

[0035] 4.2) Each activated client performs forward propagation based on the local dataset and the client parameters distributed by the federated server to extract the shredded data and the corresponding real label information.

[0036] 4.3) The master server receives the shredded data and uses the master expert module and N recombination expert modules to perform forward propagation to generate a hybrid prediction result.

[0037] 4.4) The main server calculates the loss function based on the mixed prediction results and the real label information.

[0038] 4.5) The master server generates a composite gradient corresponding to the pulverized data of the activated client based on the loss function, and performs preliminary updates on the parameters of the master expert module and N recombination expert modules based on the composite gradient.

[0039] 4.6) Activate the client to update the client parameters locally based on the composite gradient, and obtain the locally updated client parameters.

[0040] 4.7) The federated server performs federated aggregation on the locally updated client parameters of all active clients to obtain the updated client parameters.

[0041] Furthermore, the hybrid prediction results are as follows:

[0042] (2)

[0043] In the formula, Indicates the communication round. Indicates the first Dynamic participation weights for each communication round. This indicates the mixed prediction results. This indicates data to be shredded. This represents the prediction output of the master expert module. This indicates the index of the Reorganization Experts module. This indicates the total number of reorganization expert modules. This indicates a gated network. Indicates the first Parameters of the Reorganization Expert Module. Indicates the first The predicted output of the reorganization expert module.

[0044] Furthermore, the loss function is as follows:

[0045] (3)

[0046] In the formula, This represents the loss function. This indicates the sample index. This represents the current smallest learning batch, which includes several samples. This indicates a category index. Indicates the number of categories. Indicates the first The first sample Real label information for each category. Indicates the first The first sample Mixed prediction results for each category.

[0047] Furthermore, the composite gradient includes a first gradient component and a second gradient component.

[0048] The first gradient component is dynamically weighted. The parameters of the main expert module are initially updated.

[0049] The second gradient component according to the weight The parameters of the top m reorganization expert modules in terms of gating score among the N reorganization expert modules are initially updated, where m is a positive integer greater than or equal to 1.

[0050] After the activated client receives the composite gradient, it performs backpropagation according to the chain rule to update the parameters of the activated client locally.

[0051] The methods for federal aggregation include weighted average.

[0052] Furthermore, the main server reorganizes and updates the dual-stream expert architecture based on the status of the server-side parameters corresponding to all active clients, as follows:

[0053] 5.1) The master server obtains the status of the server-side parameters corresponding to all active clients in the current communication round, and reorganizes and updates the dual-stream expert architecture based on the obtained status, resulting in the reorganized dual-stream expert architecture, as shown below:

[0054] (4)

[0055] In the formula, Indicates the communication round. This indicates the index of the Reorganization Experts module. This indicates a server-side contiguous layer index. This represents the rearrangement operator. Indicates the server-side first Layer The parameters after the restructuring of the restructuring expert module. Indicates the server-side first Layer parameter state set. This indicates that the client index is activated. This indicates the activation of the client set. This represents the reorganized and reordered active client index. This represents the server-side parameters corresponding to the kth activated client after the rearrangement.

[0056] 5.2) Determine whether the current communication round meets the fusion cycle condition. If yes, proceed to step 5.3). If no, obtain the updated dual-stream expert architecture.

[0057] 5.3) The master server performs periodic bidirectional fusion operations on the reorganized dual-stream expert architecture to obtain the updated dual-stream expert architecture.

[0058] Furthermore, the fusion cycle condition includes the number of communication rounds reaching an integer multiple of the fusion cycle interval.

[0059] Furthermore, the periodic bidirectional fusion operation includes alignment operation and absorption operation.

[0060] The alignment operation is as follows:

[0061] (5)

[0062] In the formula, Indicates the communication round. This indicates the index of the Reorganization Experts module. This indicates a server-side contiguous layer index. Indicates the first Dynamic participation weights for each communication round. Indicates the first The server-side communication round number Layer The parameters after the reorganization expert module alignment operation. Indicates the first The server-side communication round number Parameters before the alignment operation of the layer master expert module. Indicates the first The server-side communication round number Layer The parameters before the reorganization expert module alignment operation.

[0063] The absorption operation is as follows:

[0064] (6)

[0065] In the formula, Indicates the first The server-side communication round number The layer master expert module absorbs the parameters after the operation. This indicates the total number of reorganization expert modules.

[0066] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the aforementioned adaptive dynamic split federated learning method based on a two-stream expert architecture.

[0067] The technical effectiveness of this invention is undeniable. It solves the stability-diversity dilemma of separate federated learning in scenarios with non-independent, identically distributed data. This invention clearly separates the responsibilities of the federated server and the master server. The federated server specializes in client-side model aggregation, while the master server specializes in training and evolving the server-side dual-stream model. This ensures that the optimization processes of the client-side and server-side models each perform their respective functions without interference. On the master server side, this invention constructs a dual-stream architecture of master experts and recombination experts, spatially decoupling global stability anchors from local diversity exploration. This achieves organic coordination between the global consistency of shallow representations and the individual diversity of deep representations within the same training process. By introducing a stage-aware evolution mechanism based on exponential decay of half-life, a smooth transition from the early stabilization stage to the later diversification stage is achieved, allowing the optimization objective to adaptively evolve according to the training stage. Through the design of periodic bidirectional fusion operations (alignment first, absorption later), the master expert constrains the behavior of the recombination expert in the early stages, and in the later stages, the master expert efficiently integrates the general knowledge captured by the recombination stream, effectively coordinating information exchange between the two expert streams and preventing semantic drift and training instability in diversity exploration. This invention significantly improves the model accuracy and training robustness of separate federated learning on non-independent and identically distributed data without increasing the communication overhead between the client and the master server or between the client and the federated server. It is especially suitable for practical federated learning application scenarios with high semantic complexity and strong data heterogeneity. Attached Figure Description

[0068] Figure 1 This is the overall algorithm flowchart for the separate federated learning method. Detailed Implementation

[0069] The present invention will be further described below with reference to embodiments, but it should not be construed that the scope of the present invention is limited to the following embodiments. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the above-described technical concept of the present invention should be included within the scope of protection of the present invention.

[0070] Example 1:

[0071] See Figure 1 An adaptive dynamic decoupled federated learning method based on a two-stream expert architecture includes the following steps:

[0072] 1) Construct a separate federated learning system and mount the global deep neural network model to be learned on the separate federated learning system.

[0073] There are many types of global deep neural network models. Common models such as ResNet and CNN can be applied to this method.

[0074] The separated federated learning system includes There are three distributed clients, a federated server, and a master server. It is a positive integer.

[0075] 2) Build a dual-stream expert architecture on the main server and initialize the communication rounds and parameters of the separate federated learning system.

[0076] The parameters of the split federated learning system include client parameters, server parameters, and hyperparameters.

[0077] 3) The federated server is from Randomly selected from distributed clients Each client acts as the active client for the current communication round, and client parameters are distributed to the corresponding active clients. Indicates the communication round. Indicates the number of active clients. This indicates the activation of the client set.

[0078] 4) Update the parameters of each activated client based on the dual-stream expert architecture to obtain the updated client parameters.

[0079] 5) Based on the status of the server-side parameters corresponding to all activated clients, the master server reorganizes and updates the dual-stream expert architecture to obtain the updated dual-stream expert architecture.

[0080] 6) Determine whether the preset communication round period has been reached. If yes, merge and update the parameters of the separate federated learning system and proceed to step 7. If no, proceed directly to step 7.

[0081] Using half-life parameter Dynamic adaptive weights are updated periodically for each communication round. Each time it reaches Each cycle will involve a fusion.

[0082] Fusion is a separate mechanism. At the end of each round, a fusion of the primary expert and the hybrid recombination experts (MORE) is performed. The weights of the primary expert are then aggregated and updated to each recombination expert, and the weights of the recombination experts are then updated to the primary expert. Adaptive weights are also updated in each round.

[0083] 7) Determine whether the preset total number of communication rounds has been reached. If yes, obtain the final global deep neural network model applicable to autonomous driving, industrial IoT, or smart security. If no, increment the number of communication rounds by one and return to step 3.

[0084] 8) Execute user-uploaded tasks using the final global deep neural network model.

[0085] Example 2:

[0086] An adaptive dynamic split federated learning method based on a two-stream expert architecture is described in Example 1. Further, the global deep neural network model to be learned includes shallow sub-networks and deep sub-networks.

[0087] The shallow sub-network is mounted on a distributed client, and the parameters of the shallow sub-network are maintained by the federated server.

[0088] The deep sub-network is hosted on the federated server and the main server, and the parameters of the deep sub-network are maintained by the main server.

[0089] Each distributed client stores a local dataset, and the local datasets of different distributed clients differ.

[0090] When applied to autonomous driving, the local dataset includes a dataset of road scene images captured by onboard cameras.

[0091] When applied to the Industrial Internet of Things (IIoT), the local dataset includes multivariate time-series datasets collected by sensors in industrial equipment.

[0092] When applied to smart security, the local dataset includes a dataset of video frame images captured by surveillance cameras.

[0093] The client parameters include parameters of the shallow sub-network.

[0094] The server-side parameters include parameters of the deep sub-network and parameters of the two-stream expert architecture.

[0095] The hyperparameters include half-life parameters and gating routing policies.

[0096] The parameters of the separate federated learning system also include the total number of communication rounds, the number of clients randomly activated in each communication round, the number of rounds of local training for each distributed client, the learning rate, and the batch size.

[0097] Example 3:

[0098] An adaptive dynamic split federated learning method based on a two-stream expert architecture is described in any one of Examples 1 and 2. Further, when the final global deep neural network model is applied to autonomous driving, the input is a road scene image collected by an onboard camera, and the output is a target detection box and category probability. The task uploaded by the user is to perform real-time target detection on the perception data collected by the autonomous vehicle.

[0099] When the final global deep neural network model is applied to the Industrial Internet of Things (IIoT), the input is multivariate time series data collected by sensors of industrial equipment, and the output is equipment operating status classification or fault warning signal. The task uploaded by the user is to diagnose faults in the real-time sensor data uploaded by the industrial equipment.

[0100] When the final global deep neural network model is applied to smart security, the input is video frame images captured by surveillance cameras, and the output is target detection boxes and category or abnormal behavior recognition results. The task uploaded by the user is to perform target detection or abnormal behavior recognition on the uploaded surveillance video frames.

[0101] Example 4:

[0102] An adaptive dynamic split federated learning method based on a two-stream expert architecture is provided. The main technical contents are described in any one of embodiments 1 to 3. Further, the two-stream expert architecture includes a master expert module and N reorganization expert modules, where N is a positive integer.

[0103] The steps for updating parameters for each active client based on the dual-stream expert architecture are as follows:

[0104] 4.1) Calculate the dynamic participation weight based on the current communication round, as shown below:

[0105] (1)

[0106] In the formula, Indicates the communication round. Indicates the first Dynamic participation weights for each communication round. This represents the half-life parameter.

[0107] 4.2) Each activated client performs forward propagation based on the local dataset and the client parameters distributed by the federated server to extract the shredded data and the corresponding real label information.

[0108] 4.3) The master server receives the shredded data and uses the master expert module and N recombination expert modules to perform forward propagation to generate a hybrid prediction result.

[0109] 4.4) The main server calculates the loss function based on the mixed prediction results and the real label information.

[0110] 4.5) The master server generates a composite gradient corresponding to the pulverized data of the activated client based on the loss function, and performs preliminary updates on the parameters of the master expert module and N recombination expert modules based on the composite gradient.

[0111] 4.6) Activate the client to update the client parameters locally based on the composite gradient, and obtain the locally updated client parameters.

[0112] 4.7) The federated server performs federated aggregation on the locally updated client parameters of all active clients to obtain the updated client parameters.

[0113] Example 5:

[0114] An adaptive dynamic split federated learning method based on a two-stream expert architecture is described in any one of Examples 1 to 4. Furthermore, the hybrid prediction results are as follows:

[0115] (2)

[0116] In the formula, Indicates the communication round. Indicates the first Dynamic participation weights for each communication round. This indicates the mixed prediction results. This indicates data to be shredded. This represents the prediction output of the master expert module. This indicates the index of the Reorganization Experts module. This indicates the total number of reorganization expert modules. This indicates a gated network. Indicates the first Parameters of the Reorganization Expert Module. Indicates the first The predicted output of the reorganization expert module.

[0117] Example 6:

[0118] An adaptive dynamic split federated learning method based on a two-stream expert architecture is provided. The main technical details are described in any one of Examples 1 to 5. Furthermore, the loss function is as follows:

[0119] (3)

[0120] In the formula, This represents the loss function. This indicates the sample index. This represents the current smallest deep learning batch. A batch contains several samples, such as 32 samples in a batch (this can be set according to specific circumstances and will be dynamically allocated to different experts). This represents a category index, such as a task requiring classification of a handwritten digit dataset from 1 to 10. Or it could be used to identify whether a person's image is that of a particular individual. Indicates the number of categories. Indicates the first The first sample Real label information for each category. Indicates the first The first sample Mixed prediction results for each category.

[0121] Example 7:

[0122] An adaptive dynamic split federated learning method based on a two-stream expert architecture is provided. The main technical contents are described in any one of Examples 1 to 6. Further, the composite gradient includes a first gradient component and a second gradient component.

[0123] The first gradient component is dynamically weighted. Perform a preliminary update to the parameters of the main expert module.

[0124] The second gradient component is weighted according to The parameters of the top m reorganization expert modules in terms of gating score among the N reorganization expert modules are initially updated, where m is a positive integer greater than or equal to 1.

[0125] For each sample, the top expert of that sample is selected. The aggregated MORE contains four experts, and the top expert is selected using a gating network (two layers of MLP are sufficient). During inference, each expert has already been aggregated (e.g., if there are 10 clients, each will have its own four corresponding experts, numbered 1-4, so experts with the same number will be aggregated into a global expert).

[0126] After the activated client receives the composite gradient, it performs backpropagation according to the chain rule to calculate the gradient of the client's part of the model, and then updates the parameters of the activated client locally.

[0127] The methods for federal aggregation include weighted average.

[0128] Example 8:

[0129] An adaptive dynamic split-federated learning method based on a two-stream expert architecture, the main technical contents of which are described in any one of embodiments 1 to 7, further wherein the master server reorganizes and updates the two-stream expert architecture based on the state of the server-side parameters corresponding to all active clients as follows:

[0130] 5.1) The master server obtains the status of the server-side parameters corresponding to all active clients in the current communication round, and reorganizes and updates the dual-stream expert architecture based on the obtained status, resulting in the reorganized dual-stream expert architecture, as shown below:

[0131] (4)

[0132] In the formula, Indicates the communication round. This indicates the index of the Reorganization Experts module. This indicates a server-side continuous-level index. This represents the rearrangement operator. Indicates the server-side first Layer The parameters after the restructuring of the restructuring expert module. Indicates the server-side first Layer parameter state set. This indicates that the client index is activated. This indicates the activation of the client set. This represents the reorganized and reordered active client index. This represents the server-side parameters corresponding to the kth activated client after the rearrangement.

[0133] 5.2) Determine whether the current communication round meets the fusion cycle condition. If yes, proceed to step 5.3). If no, obtain the updated dual-stream expert architecture.

[0134] 5.3) The master server performs periodic bidirectional fusion operations on the reorganized dual-stream expert architecture to obtain the updated dual-stream expert architecture.

[0135] Example 9:

[0136] An adaptive dynamic split federated learning method based on a two-stream expert architecture is provided. The main technical contents are described in any one of Examples 1 to 8. Further, the fusion cycle condition includes the number of communication rounds reaching an integer multiple of the fusion cycle interval.

[0137] Example 10:

[0138] An adaptive dynamic split federated learning method based on a two-stream expert architecture is provided. The main technical contents are described in any one of embodiments 1 to 9. Furthermore, the periodic bidirectional fusion operation includes an alignment operation and an absorption operation.

[0139] The alignment operation is as follows:

[0140] (5)

[0141] In the formula, Indicates the communication round. This indicates the index of the Reorganization Experts module. This indicates a server-side continuous-level index. Indicates the first Dynamic participation weights for each communication round. Indicates the first The server-side communication round number Layer The parameters after the reorganization expert module alignment operation. Indicates the first The server-side communication round number Parameters before the alignment operation of the layer master expert module. Indicates the first The server-side communication round number Layer The parameters before the reorganization expert module alignment operation.

[0142] The absorption operation is as follows:

[0143] (6)

[0144] In the formula, Indicates the first The server-side communication round number The layer master expert module absorbs the parameters after the operation. This indicates the total number of reorganization expert modules.

[0145] Example 11:

[0146] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an adaptive dynamic split federated learning method based on a two-stream expert architecture as described in any one of embodiments 1 to 10.

[0147] Example 12:

[0148] See Figure 1 An adaptive dynamic separation federated learning method based on a two-stream expert architecture, the main technical contents of which include:

[0149] 1) Obtain the set of clients participating in training, the federated server, and the master server. The system includes... Distributed clients One federated server and one master server. The global deep neural network model is logically divided into two parts: the client holds a shallow sub-network, whose global parameters are maintained by the federated server, denoted as... The server-side deep subnet is maintained by the main server and includes... A continuous layer, whose parameters are denoted as... Each client Having a local private dataset The data distribution varies across different clients, that is... This constitutes a non-independent, identically distributed data scenario.

[0150] 2) Initialize training parameters. Set the total number of communication rounds. The number of clients randomly activated in each round Basic parameters for local training on each client include the number of training epochs, learning rate, and batch size. A dual-stream expert architecture is constructed on the main server: a master expert... Maintain a set of server-side parameters The Mixture of Recombination Experts (MORE) module contains... Parallel Reorganization Expert , No. The parameters of each reorganization expert are denoted as follows: Set the half-life parameter. Hyperparameters such as gating routing policies. Initialize the current communication round. .

[0151] 3) Determine whether the preset total number of communication rounds has been reached. If yes, training ends and the final model is output; otherwise, proceed to step 4.

[0152] 4) The server determines the current communication round. Calculate dynamic participation weights This weight monotonically decreases during training. Specifically, Calculated according to the exponential decay law of half-life:

[0153] ,

[0154] in This is the half-life parameter, representing the number of rounds required for the weights to drop to half their initial value. When hour Maintaining a high value indicates that training is in a stable phase; when Every time, The training gradually diminishes, entering a more diversified phase.

[0155] 5) The federated server randomly selects from the client set. Each client is designated as the active client in the current round. The federated server will set the current global client model parameters. Distribute to each activated client.

[0156] 6) To Each activated client in Perform the following operations:

[0157] 6.1) Client Using local datasets and current client parameters Perform forward propagation to extract fragmented data. and will The corresponding tag information is sent to the main server.

[0158] 6.2) The main server receives the shredded data. The master expert and the reorganization expert maintained by it are used to perform forward propagation to obtain the master expert's output. and restructuring expert output Through a gating network The outputs of each restructuring expert are weighted to generate a mixed prediction:

[0159]

[0160] 6.3) The main server is based on hybrid prediction. Calculate the loss function with the true labels:

[0161]

[0162] in This is the smallest batch currently available. This represents the number of categories.

[0163] 6.4) The master server performs backpropagation. The loss function automatically adjusts the gradient of the mixed predictions according to the dynamic weights. and The proportions are allocated to the master expert parameters respectively. and parameters of various reorganization experts The main server updates the main expert parameters and the parameters of each recombination expert accordingly.

[0164] The gradient of the hybrid prediction is shown below:

[0165]

[0166] in: It is the predicted probability of the chief expert. It is the prediction probability of the top expert.

[0167] 6.5) The master server returns the composite gradient of the corresponding client-side shredded data to the client. .

[0168] 7) Client Update local client parameters based on the gradient returned by the master server. The updated client parameters are then uploaded to the federated server.

[0169] 8) The federated server performs a federated aggregation operation (such as weighted average) on the client parameters uploaded by all active clients in the current round to obtain the updated global client parameters. .

[0170] 9) The master server performs a reorganization update operation on the reorganization expert module. Specifically, for the server-side... The layer retrieves the server-side parameter status set corresponding to each activated client in the current round. For each reorganization expert Assign a permutation operator The operator is defined as a mapping function in the client index space. . No. The restructuring expert in the first The parameters of the layer are obtained through permutation and recombination:

[0171]

[0172] By assigning different permutations (random permutations) to different reorganization experts, an implicit blending of heterogeneous client representation paths is achieved, avoiding representation distortion caused by directly averaging incompatible deep parameters. The permutation and reorganization operation can be selectively performed at the shallow and deep network layers; that is, reorganization is applied only at the deep server-side layer, while conventional aggregation is still used at the shallow layer.

[0173] 10) Determine whether the current round meets the predetermined fusion cycle conditions. If If the interval is an integer multiple of the fusion cycle interval, the master server performs the periodic bidirectional fusion operation in step 11); otherwise, proceed to step 13.

[0174] 11) The master server performs periodic bidirectional fusion operations, which include two steps: alignment and absorption.

[0175] 11.1) Alignment Operation: Soft alignment is performed on the parameters of each recombinant expert using the master expert parameters to correct semantic drift that may occur during diversity exploration. For the server-side... layer:

[0176]

[0177] 11.2) Absorption Operation: The aligned parameters of each recombination expert are weighted and averaged, then injected into the master expert, allowing the master expert to internalize the general knowledge explored by the recombination flow. For the server-side... layer:

[0178]

[0179] This bidirectional fusion strategy in the early stages ( The high level primarily functions as a regularization mechanism, using the lead expert to constrain the behavior of the restructuring experts; in the later stages ( The low-level approach primarily achieves knowledge internalization, enabling the master expert to efficiently integrate and reconstruct the diverse representations captured by the stream. Since both alignment and absorption operations are convex combinations, the updated parameters always remain within the convex hull of the original expert parameters, ensuring the stability of training.

[0180] 12) Determine whether the gating network or other hyperparameters need to be adjusted in the current round, and update them accordingly if necessary.

[0181] 13) Communication rounds Add 1, then return to step 3).

[0182] 14) Training process ends. The federated server saves the final global client model parameters. The master server stores the final server-side model parameters (master expert). and restructuring experts .

[0183] Example 13:

[0184] See Figure 1 An adaptive dynamic separation federated learning method based on a two-stream expert architecture, the main technical contents of which include:

[0185] The global deep neural network model is a unified model deployed across the entire distributed federated learning system. It aggregates local updates from distributed clients to form a collaborative model with generalization capabilities. Its core functions include:

[0186] 1. Knowledge aggregation carrier: Integrate privacy data knowledge that is scattered across various clients and cannot be directly shared into a unified global model.

[0187] 2. Prediction service provider: After training, the global model can be deployed to various clients or the cloud to provide real-time inference predictions for new input data.

[0188] 3. Optimize target anchor point: During training, it serves as the target of federated aggregation, guiding each client model to converge toward the global optimum.

[0189] It can be applied to fields such as smart healthcare, autonomous driving, industrial IoT, smart security, and mobile internet. For example, in smart healthcare, consider a cross-hospital collaborative skin disease classification and diagnosis system:

[0190] Data: Each of the three hospitals possesses dermoscopy image data. Due to the different service populations, the data distribution varies significantly (label offset), and they cannot be directly shared due to privacy regulations.

[0191] Model Function: After being trained using the method described in this embodiment, a global deep neural network model (such as ResNet18) can accurately identify various skin disease categories such as melanoma, basal cell carcinoma, and benign nevi.

[0192] Prediction results: When a new dermoscopic image is input, the model outputs the probability distribution of each category, such as [0.02, 0.91, 0.05, 0.02], where the highest probability corresponds to "basal cell carcinoma", providing doctors with AI-assisted diagnostic reference.

[0193] This embodiment effectively solves the problem of model performance degradation caused by data heterogeneity through a dual-stream expert architecture and an adaptive dynamic mechanism, and significantly improves prediction accuracy without increasing communication overhead.

Claims

1. An adaptive dynamic split-federated learning method based on a two-stream expert architecture, characterized in that, Includes the following steps: 1) Construct a separate federated learning system and mount the global deep neural network model to be learned on the separate federated learning system; The separated federated learning system includes There are three distributed clients, a federated server, and a master server. It is a positive integer; 2) Build a dual-stream expert architecture on the main server and initialize the communication rounds and parameters of the separate federated learning system; The parameters of the split federated learning system include client parameters, server parameters, and hyperparameters; 3) The federated server is from Randomly selected from distributed clients Each client acts as the active client for the current communication round, and client parameters are distributed to the corresponding active clients. Indicates the communication round; Indicates the number of active clients. Indicates the set of clients to be activated; 4) Update the parameters of each activated client based on the dual-stream expert architecture to obtain the updated client parameters; 5) Based on the status of the server-side parameters corresponding to all activated clients, the master server reorganizes and updates the dual-stream expert architecture to obtain the updated dual-stream expert architecture. 6) Determine whether the preset communication round period has been reached. If yes, merge and update the parameters of the separate federated learning system and proceed to step 7). If no, proceed directly to step 7). 7) Determine whether the preset total number of communication rounds has been reached. If yes, obtain the final global deep neural network model applicable to autonomous driving, industrial IoT, or smart security. If no, increment the number of communication rounds by one and return to step 3). 8) Execute user-uploaded tasks using the final global deep neural network model.

2. The adaptive dynamic split-federated learning method based on a two-stream expert architecture according to claim 1, characterized in that, The global deep neural network model to be learned includes shallow sub-networks and deep sub-networks; The shallow sub-network is mounted on a distributed client, and the parameters of the shallow sub-network are maintained by the federated server. The deep sub-network is hosted on the federated server and the main server, and the parameters of the deep sub-network are maintained by the main server. Each distributed client stores a local dataset, and the local datasets of different distributed clients differ. When applied to autonomous driving, the local dataset includes a dataset of road scene images captured by onboard cameras; When applied to the Industrial Internet of Things (IIoT), the local dataset includes multivariate time-series datasets collected by sensors in industrial equipment. When applied to smart security, the local dataset includes a dataset of video frame images captured by surveillance cameras; The client parameters include parameters of the shallow sub-network; The server-side parameters include parameters of the deep sub-network and parameters of the two-stream expert architecture; The hyperparameters include half-life parameters and gating routing strategies; The parameters of the separate federated learning system also include the total number of communication rounds, the number of clients randomly activated in each communication round, the number of rounds of local training for each distributed client, the learning rate, and the batch size.

3. The adaptive dynamic split-federated learning method based on a two-stream expert architecture according to claim 1, characterized in that, When the final global deep neural network model is applied to autonomous driving, the input is the road scene image captured by the vehicle camera, and the output is the target detection box and the category probability. The task uploaded by the user is to perform real-time target detection on the perception data collected by the autonomous vehicle. When the final global deep neural network model is applied to the Industrial Internet of Things, the input is multivariate time series data collected by industrial equipment sensors, and the output is equipment operating status classification or fault warning signal. The task uploaded by the user is to diagnose faults based on real-time sensor data uploaded by industrial equipment. When the final global deep neural network model is applied to smart security, the input is video frame images captured by surveillance cameras, and the output is target detection boxes and category or abnormal behavior recognition results. The task uploaded by the user is to perform target detection or abnormal behavior recognition on the uploaded surveillance video frames.

4. The adaptive dynamic split-federated learning method based on a two-stream expert architecture according to claim 1, characterized in that, The dual-stream expert architecture includes a master expert module and N reorganization expert modules, where N is a positive integer; The steps for updating parameters for each active client based on the dual-stream expert architecture are as follows: 4.1) Calculate the dynamic participation weight based on the current communication round, as shown below: （1） In the formula, Indicates the communication round; Indicates the first Dynamic participation weights for each communication round; This represents the half-life parameter; 4.2) Each activated client performs forward propagation based on the local dataset and the client parameters distributed by the federated server to extract the shredded data and the corresponding real label information; 4.3) The main server receives the shredded data and uses the main expert module and N recombination expert modules to perform forward propagation to generate a hybrid prediction result; 4.4) The main server calculates the loss function based on the mixed prediction results and the true label information; 4.5) The master server generates a composite gradient corresponding to the shredded data of the activated client based on the loss function, and performs preliminary updates on the parameters of the master expert module and N recombination expert modules based on the composite gradient; 4.6) Activate the client to update the client parameters locally based on the composite gradient, and obtain the locally updated client parameters; 4.7) The federated server performs federated aggregation on the locally updated client parameters of all active clients to obtain the updated client parameters.

5. The adaptive dynamic split-federated learning method based on a two-stream expert architecture according to claim 4, characterized in that, The results of the hybrid prediction are shown below: （2） In the formula, Indicates the communication round; Indicates the first Dynamic participation weights for each communication round; This indicates a mixed prediction result; This indicates data shredding; This indicates the prediction output of the main expert module; Indicates the index of the Reorganization Experts module; Indicates the total number of reorganization expert modules; Indicates a gating network; Indicates the first Parameters of the Recombination Expert module; Indicates the first The predicted output of the reorganization expert module.

6. The adaptive dynamic split-federated learning method based on a two-stream expert architecture according to claim 4, characterized in that, The loss function is as follows: （3） In the formula, Represents the loss function; Indicates the sample index; This represents the current smallest learning batch, which includes a number of samples. Indicates a category index; Indicates the number of categories; Indicates the first The first sample Real label information for each category; Indicates the first The first sample Mixed prediction results for each category.

7. The adaptive dynamic split-federated learning method based on a two-stream expert architecture according to claim 4, characterized in that, The composite gradient includes a first gradient component and a second gradient component; The first gradient component is dynamically weighted. Perform a preliminary update of the parameters in the main expert module; The second gradient component is weighted according to The parameters of the top m recombination expert modules in terms of gating score among the N recombination expert modules are initially updated, where m is a positive integer greater than or equal to 1; After the activated client receives the composite gradient, it performs backpropagation according to the chain rule to update the parameters of the activated client locally. The methods for federal aggregation include weighted average.

8. The adaptive dynamic split-federated learning method based on a two-stream expert architecture according to claim 1, characterized in that, The steps for the master server to reorganize and update the dual-stream expert architecture based on the status of server-side parameters corresponding to all active clients are as follows: 5.1) The master server obtains the status of the server-side parameters corresponding to all active clients in the current communication round, and reorganizes and updates the dual-stream expert architecture based on the obtained status, resulting in the reorganized dual-stream expert architecture, as shown below: （4） In the formula, Indicates the communication round; Indicates the index of the Reorganization Experts module; This indicates a server-side contiguous layer index; Represents the rearrangement operator; Indicates the server-side first Layer The parameters after reorganization by the reorganization expert module; Indicates the server-side first Layer parameter state set; This indicates that the client index is activated; Indicates the set of clients to be activated; This represents the active client index after rearrangement; This represents the server-side parameters corresponding to the kth active client after the rearrangement; 5.2) Determine whether the current communication round meets the fusion cycle condition. If yes, proceed to step 5.3). If no, obtain the updated dual-stream expert architecture. 5.3) The master server performs periodic bidirectional fusion operations on the reorganized dual-stream expert architecture to obtain the updated dual-stream expert architecture.

9. The adaptive dynamic split-federated learning method based on a two-stream expert architecture according to claim 8, characterized in that, The fusion cycle condition includes the number of communication rounds reaching an integer multiple of the fusion cycle interval.

10. The adaptive dynamic split-federated learning method based on a two-stream expert architecture according to claim 8, characterized in that, The periodic bidirectional fusion operation includes alignment operation and absorption operation; The alignment operation is as follows: （5） In the formula, Indicates the communication round; Indicates the index of the Reorganization Experts module; This indicates a server-side contiguous layer index; Indicates the first Dynamic participation weights for each communication round; Indicates the first The server-side communication round number Layer Parameters after alignment operation of each reorganization expert module; Indicates the first The server-side communication round number Parameters before the alignment operation of the layer master expert module; Indicates the first The server-side communication round number Layer Parameters before the reorganization expert module alignment operation; The absorption operation is as follows: （6） In the formula, Indicates the first The server-side communication round number The layer master expert module absorbs the parameters after the operation; This indicates the total number of reorganization expert modules.

Images