Federal learning method, system and device based on global feature sharing and medium

Abstract

The invention provides a federal learning method, system and device based on global feature sharing and a medium, and the method comprises the steps: initializing a bottom feature extraction network of a global model through employing a server, and sharing the parameters of the bottom feature extraction network to all clients; the server dynamically calculates contribution utility scores of the clients according to differences between underlying model parameters uploaded by the clients and current global underlying network parameters and local data scales of the clients, and constructs probability distribution according to the utility scores, then selecting a client set participating in the training in a weighted random sampling mode; and the selected client recombines the updated global underlying network parameters with locally reserved high-level network parameters to form a new composite model, and the training process is repeatedly executed until the model reaches a preset convergence condition, so that the data transmission quantity between each client and the server can be remarkably reduced, and the data transmission efficiency is improved. Bandwidth resources are greatly saved, and the efficiency of large-scale distributed training is improved.

Description

Technical Field

[0001] This application belongs to the technical field of federated learning, and particularly relates to a federated learning method, system, electronic device, and storage medium based on global feature sharing. Background Art

[0002] In recent years, with the wide application of deep learning in fields such as image recognition and natural language processing, the scale of model parameters has been continuously increasing, and it is difficult for a single device to independently complete the training requirements of large-scale data. Therefore, federated learning has emerged. Traditional federated learning methods usually require each client to upload complete model parameters (including the underlying feature extraction layer and the high-level personalized layer) so that the server can uniformly aggregate the model updates of each client. This approach protects data privacy to a certain extent and utilizes distributed computing resources, but at the same time, it also brings significant communication overhead problems. Especially in the context of a large number of deep learning model parameters, transmitting complete model parameters each time communication is required, which greatly increases bandwidth consumption and communication latency and is difficult to meet the requirements of actual large-scale applications and bandwidth-constrained environments. In addition, existing technologies generally assume that client data is independently and identically distributed (iid), yet in actual applications, client data often has obvious non-independent and identically distributed (non-iid) characteristics, which leads to slow convergence or even performance degradation during global model aggregation, thus restricting the universality and personalized application of the model. Summary of the Invention

[0003] In view of this, it is necessary to provide a federated learning method, system, electronic device, and storage medium based on global feature sharing, which can at least overcome one of the above defects.

[0004] In a first aspect, an embodiment of the present application provides a federated learning method based on global feature sharing, which is applied to a configured server and multiple clients. The method includes:

[0005] The server initializes the underlying feature extraction network of the global model and shares the parameters of the underlying feature extraction network with all the clients;

[0006] Each client configures its local high-level feature extraction network according to its local data distribution characteristics, so as to form a composite model including the underlying feature extraction network and its local high-level feature extraction network at each client;

[0007] The server dynamically calculates the contribution utility scores of each client according to the differences between the underlying model parameters uploaded by each client and the current global underlying network parameters and the local data scale of each client, constructs a probability distribution according to the utility scores, and then selects a set of clients participating in this round of training through weighted random sampling;

[0008] The server is used to send the global underlying network parameters to the selected client;

[0009] After receiving the global underlying network parameters sent by the server, the selected client combines the global underlying network parameters with the local high-level network parameters to form a complete local model;

[0010] The selected client performs forward propagation on the local data using the global underlying network, extracts global features, and caches the statistical information of the global features for establishing benchmark parameters for feature distribution alignment;

[0011] The selected client trains the local model. After the training is completed, only the updated underlying network parameters are uploaded to the server;

[0012] The server weights and aggregates the uploaded underlying network parameters according to the contribution utility scores of each selected client to generate a new generation of global underlying network parameters;

[0013] The selected client recombines the updated global underlying network parameters with the local high-level network parameters reserved locally to form a new composite model until the model reaches the preset convergence condition.

[0014] In one embodiment, the server dynamically calculates the contribution utility scores of each client according to the difference between the underlying model parameters uploaded by each client and the current global underlying network parameters and the local data scale of each client, including:

[0015] u i =n i ·max(d i ,∈)

[0016] where u i is the dynamic utility score, n i is the local data volume, is the Euclidean distance between the underlying model uploaded by client i in the previous round and the global underlying model, is the parameter extracted by the underlying feature extraction network, W base,t-1 is the parameter extracted by the underlying feature extraction network in the previous round, and ∈ is a minimum value to ensure that when d i =0, the client still has a certain probability of participating in this round of training.

[0017] In one embodiment, the method further includes:

[0018] The server generates a probability distribution for client selection according to the dynamic utility scores of each client, and the calculation formula of the probability distribution is:

[0019]

[0020] p i represents the sampling weight obtained by the client based on its contribution degree, and N is the total number of clients currently participating in federated learning. is the sum of the utility scores of all N clients;

[0021] The server performs probability-weighted random sampling and selects the set of clients to participate in the current round of update.

[0022] In one embodiment, the client uses a composite loss function to evaluate during local training, and the composite loss function includes a task loss, a parameter alignment regularization loss, and a feature distribution alignment regularization loss;

[0023] The parameter alignment regularization loss is calculated by the following formula:

[0024]

[0025] where λ is a regularization hyperparameter used to control the deviation between the local underlying model and the global underlying model of the previous round, are the parameters extracted by the underlying feature extraction network.

[0026] In one embodiment, the feature distribution alignment regularization loss is calculated by the maximum mean discrepancy algorithm, and the calculation formula of the maximum mean discrepancy algorithm is:

[0027]

[0028] where, is the feature distribution alignment regularization loss, is the maximum mean discrepancy, μ is the hyperparameter of the feature alignment regularization term, φ is the Gaussian kernel function, H is the reproducing kernel Hilbert space, m is the number of samples used to calculate the mean in the local feature set, and n is the number of samples used to calculate the mean in the global feature set.

[0029] In one embodiment, the client updates the local model using the gradient descent method, and the update rule of the client is:

[0030]

[0031] where η is the learning rate, are the local model parameters updated after the end of the (t + 1)-th round of training for client i, are the local model parameters of client i during the t-th round of training, is the gradient of the composite loss function calculated for the local model parameters of client i during the t-th round of training.

[0032] In one embodiment, the server is further configured to:

[0033] Perform weighted average on the underlying models uploaded by each client according to the corresponding dynamic utility scores to update the global underlying model;

[0034]

[0035] Where W base,t+1 is the updated global underlying model, is the underlying model uploaded by client i, d i is the dynamic utility score of client i, and S t is the set of clients participating in training in this round.

[0036] In a second aspect, an embodiment of the present application provides a federated learning system based on global feature sharing, which is applied to implement the federated learning method based on global feature sharing as described in the first aspect. The system includes: a server and multiple clients;

[0037] The server is configured to initialize the underlying feature extraction network of the global model and share the parameters of the underlying feature extraction network with all the clients;

[0038] Each client configures its local high-level feature extraction network according to its local data distribution characteristics, so as to form a composite model including the underlying feature extraction network and its local high-level feature extraction network at each client;

[0039] The server dynamically calculates the contribution utility scores of each client according to the differences between the parameters of the underlying models uploaded by each client and the current global underlying network parameters and the local data scales of each client, constructs a probability distribution according to the utility scores, and then selects a set of clients participating in training in this round through weighted random sampling;

[0040] The server is configured to send the global underlying network parameters to the selected clients;

[0041] After receiving the global underlying network parameters sent by the server, the selected clients combine the global underlying network parameters with the local high-level network parameters to form a complete local model;

[0042] The selected clients perform forward propagation on local data using the global underlying network, extract global features, and cache the statistical information of the global features for establishing benchmark parameters for feature distribution alignment;

[0043] The selected client trains the local model, and after the training is completed, only uploads the updated underlying network parameters to the server;

[0044] The server performs weighted aggregation on the uploaded underlying network parameters according to the contribution utility scores of the selected clients to generate a new generation of global underlying network parameters;

[0045] The selected client recombines the updated global underlying network parameters with the locally retained high-level network parameters to form a new composite model until the model reaches the preset convergence condition.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0047] A processor;

[0048] A memory for storing instructions executable by the processor;

[0049] Wherein, when the processor is configured to execute the instructions, it implements the federated learning method based on global feature sharing as described in the first aspect.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions that direct a device to execute the federated learning method based on global feature sharing as described in the first aspect.

[0051] The federated learning method, system, electronic device, and storage medium based on global feature sharing provided by the embodiments of the present application can significantly reduce the data transmission volume between each client and the server, greatly save bandwidth resources, and improve the efficiency of large-scale distributed training. This method combines global underlying feature sharing and local high-level personalized training, effectively balancing global generality and local personalized needs, and ensuring the stable convergence and high-precision performance of the global model in a non-independent and identically distributed data environment. At the same time, the introduction of the dynamic utility score mechanism makes the client selection and model aggregation processes more reasonable, further improving the generalization ability and robustness of the overall model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of the federated learning method based on global feature sharing provided by an embodiment of the present application.

[0053] Figure 2 It is a module diagram of the federated learning system based on global feature sharing provided by an embodiment of the present application.

[0054] Figure 3 It is a module diagram of the electronic device provided by an embodiment of the present application.

[0055] MAIN ELEMENT SYMBOL DESCRIPTION

[0056] Federated Learning System 10 Based on Global Feature Sharing

[0057] Server 11

[0058] Client 12

[0059] Electronic Device 20

[0060] Processor 21

[0061] Memory 22

[0062] Method Steps S100 - S900 Detailed Implementation Manner

[0063] Next, in combination with the accompanying drawings in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments.

[0064] It should be noted that in the embodiments of the present application, "at least one" means one or more, and multiple means two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application.

[0065] It should be noted that in the embodiments of the present application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. Features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0066] Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts are within the scope of protection of the present application.

[0067] In recent years, with the wide application of deep learning in fields such as image recognition and natural language processing, the scale of model parameters has been continuously increasing. It is difficult for a single device to independently complete the training requirements of large-scale data, so federated learning has emerged as the times require. Traditional federated learning methods usually require each client to upload complete model parameters (including the underlying feature extraction layer and the high-level personalized layer) so that the server can uniformly aggregate the model updates of each client. This approach protects data privacy to a certain extent and utilizes distributed computing resources, but at the same time brings significant communication overhead problems. Especially in the context of a large number of deep learning model parameters, transmitting complete model parameters each time communication is carried out greatly increases bandwidth consumption and communication latency, making it difficult to meet the requirements of actual large-scale applications and bandwidth-constrained environments. In addition, existing technologies generally assume that client data is independently and identically distributed (iid), but in actual applications, client data often has obvious non-independent and identically distributed (non-iid) characteristics, which leads to slow convergence or even performance degradation during global model aggregation, thus restricting the universality and personalized application of the model.

[0068] In view of this, this application provides a federated learning method, system, device and medium based on global feature sharing, which can significantly reduce the data transmission volume between each client and the server, greatly save bandwidth resources, and improve the efficiency of large-scale distributed training. This method combines global underlying feature sharing and local high-level personalized training to effectively balance global generality and local personalized needs, ensuring stable convergence and high-precision performance of the global model in a non-independent and identically distributed data environment. At the same time, the introduction of the dynamic utility score mechanism makes the client selection and model aggregation process more reasonable, further improving the generalization ability and robustness of the overall model.

[0069] Figure 1 It is a schematic flow chart of the federated learning method based on global feature sharing provided by an embodiment of this application. As Figure 1The federated learning method based on global feature sharing shown above includes at least the following steps: S100: The application server initializes the underlying feature extraction network of the global model and shares the parameters of the underlying feature extraction network with all clients; S200: Each client configures its local high-level feature extraction network according to its local data distribution characteristics to form a composite model including the underlying feature extraction network and the local high-level feature extraction network on each client; S300: The server dynamically calculates the contribution utility scores of each client according to the differences between the underlying model parameters uploaded by each client and the current global underlying network parameters and the local data scales of each client, constructs a probability distribution based on the utility scores, and then selects a set of clients participating in this round of training through weighted random sampling; S400: The server is used to send the global underlying network parameters to the selected clients; S500: After receiving the global underlying network parameters sent by the server, the selected clients combine the global underlying network parameters with the local high-level network parameters to form a complete local model; S600: The selected clients perform forward propagation on the local data using the global underlying network, extract global features, and cache the statistical information of the global features for establishing the reference parameters for feature distribution alignment; S700: The selected clients train the local model, and after the training is completed, only upload the updated underlying network parameters to the server; S800: The server performs weighted aggregation on the uploaded underlying network parameters according to the contribution utility scores of each selected client to generate a new generation of global underlying network parameters; S900: The selected clients recombine the updated global underlying network parameters with the local high-level network parameters they retain to form a new composite model until the model reaches the preset convergence condition.

[0070] S100: The application server initializes the underlying feature extraction network of the global model and shares the parameters of the underlying feature extraction network with all clients.

[0071] In the embodiment of the present application, the federated learning method based on global feature sharing includes, in step S100, the application server initializes the underlying feature extraction network of the global model and shares the parameters of the underlying feature extraction network with all clients.

[0072] Specifically, the server initializes the underlying network parameters according to the preset model structure and sends these parameters to each client through a secure network channel.

[0073] It can be understood that this step ensures that all clients use the same underlying network in the feature extraction stage, thus providing a consistent basis for subsequent model fusion and global aggregation.

[0074] S200: Each client configures its local high-level feature extraction network according to its local data distribution characteristics, so as to form a composite model including a low-level feature extraction network and the local high-level feature extraction network on each client.

[0075] In the embodiment of the present application, the federated learning method based on global feature sharing includes, in step S200, each client configures its local high-level feature extraction network according to its local data distribution characteristics, so as to form a composite model including a low-level feature extraction network and the local high-level feature extraction network on each client.

[0076] Specifically, each client designs or adjusts the high-level network structure according to the data type and distribution characteristics it processes, and combines it with the shared low-level network.

[0077] It can be understood that this step enables each client to extract personalized features that conform to its own data characteristics while using the general features shared globally, thereby improving the performance of the model in specific application scenarios.

[0078] S300: The server dynamically calculates the contribution utility scores of each client according to the differences between the low-level model parameters uploaded by each client and the current global low-level network parameters, as well as the local data scales of each client, constructs a probability distribution according to the utility scores, and then selects a set of clients participating in this round of training through weighted random sampling.

[0079] In the embodiment of the present application, the federated learning method based on global feature sharing includes, in step S300, the server dynamically calculates the contribution utility scores of each client according to the differences between the low-level model parameters uploaded by each client and the current global low-level network parameters, as well as the local data scales of each client, constructs a probability distribution according to the utility scores, and then selects a set of clients participating in this round of training through weighted random sampling.

[0080] Specifically, after receiving the low-level network parameters uploaded by each client, the server calculates the contribution utility score of each client by comparing the differences between the parameters of each client and the current global parameters, and then combining the information of the local data volume of each client.

[0081] It can be understood that this dynamic evaluation process reflects the actual contributions of each client to the global model update, ensures that clients with large amounts of data and obvious updates have a higher probability of participation in subsequent training, thereby improving the update quality of the overall model.

[0082] S400: The server is used to send the global low-level network parameters to the selected clients.

[0083] In the embodiment of the present application, the federated learning method based on global feature sharing includes, in step S400, that the server is used to send the global underlying network parameters to the selected clients.

[0084] Specifically, the server encapsulates the newly generated global underlying network parameters and transmits them to the selected clients in this round through a secure communication protocol.

[0085] It can be understood that this process ensures that each selected client can obtain the latest global parameters at the same time, providing synchronized basic data for subsequent local model combination and training.

[0086] S500: After receiving the global underlying network parameters sent by the server, the selected clients combine the global underlying network parameters with the local high-level network parameters to form a complete local model.

[0087] In the embodiment of the present application, the federated learning method based on global feature sharing includes, in step S500, that after receiving the global underlying network parameters sent by the server, the selected clients combine the global underlying network parameters with the local high-level network parameters to form a complete local model.

[0088] Specifically, the clients combine the received global underlying network parameters with the pre-configured local high-level network parameters to form a composite model integrating general features and personalized features.

[0089] It can be understood that through this combination method, the clients can not only utilize the general knowledge shared by the server but also extract personalized features according to their own data, thereby improving the adaptability and prediction accuracy of the model.

[0090] S600: The selected clients use the global underlying network to perform forward propagation on the local data, extract global features, and cache the statistical information of the global features for establishing benchmark parameters for feature distribution alignment.

[0091] In the embodiment of the present application, the federated learning method based on global feature sharing includes, in step S600, that the selected clients use the global underlying network to perform forward propagation on the local data, extract global features, and cache the statistical information of the global features for establishing benchmark parameters for feature distribution alignment.

[0092] Specifically, each client uses the global underlying network to perform inference processing on the local data, extracts features therefrom, and calculates and stores the statistical information (such as mean and variance) of these features.

[0093] It can be understood that this step provides a necessary benchmark for subsequent coordination of feature distributions among clients through feature distribution alignment regularization processing, thereby enhancing the stability and consistency of the global model.

[0094] S700: The selected client trains the local model. After the training is completed, only the updated underlying network parameters are uploaded to the server.

[0095] In the embodiment of the present application, the federated learning method based on global feature sharing includes, in step S700, the selected client trains the local model. After the training is completed, only the updated underlying network parameters are uploaded to the server.

[0096] Specifically, when each client performs local training using the composite model, the model parameters are updated according to the set loss function, and after the training is completed, only the update results of the underlying network parameters are uploaded to the server.

[0097] It can be understood that this design not only effectively reduces the amount of data transmission and communication overhead, but also ensures the privacy and stability of the local personalized high-level network parameters, thus realizing an efficient federated learning process.

[0098] S800: The server performs weighted aggregation on the uploaded underlying network parameters according to the contribution utility scores of the selected clients to generate a new generation of global underlying network parameters.

[0099] In the embodiment of the present application, the federated learning method based on global feature sharing includes, in step S800, the server performs weighted aggregation on the uploaded underlying network parameters according to the contribution utility scores of the selected clients to generate a new generation of global underlying network parameters.

[0100] Specifically, the server performs weighted average processing on the parameters uploaded by each client according to their respective contribution utility scores to calculate the updated global underlying network parameters.

[0101] It can be understood that this weighted aggregation method makes full use of the effective update information of each client, enabling the generated global model to more accurately reflect the overall data distribution and improving the global performance and robustness of the model.

[0102] S900: The selected client recombines the updated global underlying network parameters with the high-level network parameters retained locally to form a new composite model until the model reaches the preset convergence condition.

[0103] In the embodiment of the present application, the federated learning method based on global feature sharing includes, in step S900, the selected client recombines the updated global underlying network parameters with the high-level network parameters retained locally to form a new composite model until the model reaches the preset convergence condition.

[0104] Specifically, after each client receives the updated global underlying parameters from the server, it re-integrates them with the local high-level network parameters to form a new composite model, and continues local training and iterative updates based on this.

[0105] It can be understood that this step realizes the continuous collaborative update of the global model and the local model, ensures that the model gradually converges during continuous iteration, takes into account both global generality and personalized requirements, and meets the performance requirements of actual application scenarios.

[0106] In the embodiments of the present application, the server dynamically calculates the contribution utility scores of each client according to the differences between the underlying model parameters uploaded by each client and the current global underlying network parameters, as well as the local data scale of each client, including:

[0107] u i =n i ·max(d i ,∈)

[0108] Where u i is the dynamic utility score, n i is the local data volume, is the Euclidean distance between the underlying model uploaded by client i in the previous round and the global underlying model, is the parameter extracted by the underlying feature extraction network, W base,t-1 is the parameter extracted by the underlying feature extraction network in the previous round, and ∈ is a minimum value to ensure that when d i =0, the client still has a certain probability of participating in this round of training.

[0109] Specifically, the server first compares the differences between the underlying model parameters uploaded by each client and the current global underlying network parameters, calculates the parameter change degree d i of each client, then compares d i with the preset minimum value ∈, takes the larger one, and multiplies it by the local data volume n i of this client, so as to obtain the dynamic utility score u i reflecting the contribution degree of this client to the global model.

[0110] It can be understood that this dynamic utility score combines the amplitude of the client model update and the local data scale, can not only reflect the actual contribution of the client to the improvement of the global model, but also ensures that even if the model parameters of some clients have no obvious change in the previous round of update, as long as they have a large amount of data, they still have the opportunity to participate in this round of training, thus ensuring the fairness and effectiveness of each client in the federated learning process.

[0111] In an embodiment of the present application, the method further includes: the server generates a probability distribution of client selection according to the dynamic utility scores of each client, and the calculation formula of the probability distribution is:

[0112]

[0113] p i represents the sampling weight obtained by the client based on its contribution degree, N is the total number of clients currently participating in federated learning, is the sum of the utility scores of all N clients. The server performs probability-weighted random sampling and selects a client set to participate in the current round of update.

[0114] Specifically, the server first accumulates the utility scores uploaded by each client, calculates the total sum of the utility scores of all clients, and divides the utility score of each client by the total sum, so as to obtain the sampling probability of each client. The server adopts a weighted random sampling method based on this probability distribution and selects some clients with higher contributions as the participants in the current round of update.

[0115] It can be understood that this process ensures that clients with a large amount of data and obvious model updates obtain higher weights in the global model update, so that the global model can more accurately reflect the overall data distribution, improve the update efficiency and convergence performance of federated learning, and at the same time take into account the fairness among different clients.

[0116] In an embodiment of the present application, the client uses a composite loss function for evaluation during the local training process, and the composite loss function includes a task loss, a parameter alignment regularization loss, and a feature distribution alignment regularization loss;

[0117] The parameter alignment regularization loss is calculated by the following formula:

[0118]

[0119] where λ is a regularization hyperparameter used to control the deviation between the local underlying model and the global underlying model of the previous round, are the parameters extracted by the underlying feature extraction network.

[0120] Specifically, the client uses a composite loss function including a task loss, a parameter alignment regularization loss, and a feature distribution alignment regularization loss for model evaluation during the local training process. For the parameter alignment regularization loss, the client first compares the current local underlying model parameters with the global underlying model parameters shared in the previous round, calculates the Euclidean distance between the two, and then multiplies the distance by the preset regularization hyperparameter λ to obtain the gradient of the regularization loss.

[0121] It can be understood that this regularization term is used to constrain the deviation between the local underlying model of the client and the global model, so that while each client conducts personalized training, the model parameters will not deviate too much, thereby ensuring the consistency and stability of the global model.

[0122] In the embodiments of the present application, the feature distribution alignment regularization loss is calculated by the maximum mean discrepancy algorithm, and the calculation formula of the maximum mean discrepancy algorithm is:

[0123]

[0124] Where, is the feature distribution alignment regularization loss, is the maximum mean discrepancy, μ is the hyperparameter of the feature alignment regularization term, φ is the Gaussian kernel function, H is the reproducing kernel Hilbert space, m is the number of samples used to calculate the mean in the local feature set, n is the number of samples used to calculate the mean in the global feature set.

[0125] Specifically, the client calculates the feature distribution alignment regularization loss using the maximum mean discrepancy (MMD) algorithm. The client first calculates the means of the global feature set and the local feature set respectively, then maps the means into the reproducing kernel Hilbert space through the Gaussian kernel function, and further calculates the distance between the mapped means as a measure of the feature distribution difference. The hyperparameter μ in this process is used to adjust the regularization strength, and the sample numbers n and m correspond to the number of samples used for mean calculation in the global feature set and the local feature set respectively.

[0126] It can be understood that the purpose of this step is to make the feature distribution extracted locally as consistent as possible with the global feature distribution, thereby effectively alleviating the model deviation caused by inconsistent data distributions and improving the generalization ability and adaptability of the global model.

[0127] In the embodiments of the present application, the client updates the local model using the gradient descent method, and the update rule of the client is:

[0128]

[0129] Where, η is the learning rate, is the local model parameter updated by the client i after the end of the (t + 1)-th round of training, is the local model parameter of the client i during the t-th round of training, is the gradient of the composite loss function calculated for the local model parameter of the client i during the t-th round of training.

[0130] Specifically, the client uses the gradient descent method to update the local model. In each round of training, the client calculates the gradient information according to the current composite loss function, and then updates the local model parameters according to the preset learning rate η, that is, subtracting the learning rate multiplied by the gradient of the loss function from the current parameters.

[0131] It can be understood that this update mechanism enables the model parameters to be iteratively adjusted in the direction of reducing the overall loss in each round of training, thereby gradually improving the accuracy and robustness of the local model on the task, and finally promoting the entire federated learning system to reach a convergence state.

[0132] In the embodiment of the present application, the server is further configured to: perform weighted average update on the global underlying model according to the corresponding dynamic utility scores of the underlying models uploaded by each client;

[0133]

[0134] Wherein, W base,t+1 is the updated global underlying model, is the underlying model uploaded by client i, d i is the dynamic utility score of client i, and S t is the set of clients participating in the training in this round.

[0135] Specifically, after receiving the updated underlying model parameters uploaded by each client, the server performs weighted average on these parameters according to the dynamic utility scores of each client. The specific process is to multiply the uploaded parameters of each client by their corresponding utility scores, then sum these weighted results, and finally divide by the sum of the utility scores of all participating clients to generate new global underlying model parameters.

[0136] It can be understood that this weighted aggregation method can fully integrate the update information of each client, making the improvement of the global model more in line with the characteristics of the overall data, and at the same time ensuring that clients with greater contributions have a greater impact on the model update, improving the accuracy and convergence speed of the global model.

[0137] In this embodiment, the training process is carried out in a cyclic iteration manner. In each round of training, each client uses local data to train its composite model and only uploads the updated underlying network parameters to the server; the server then performs weighted aggregation according to the parameters uploaded by each client and their contribution utility scores to update the global underlying network parameters. This cycle continues until the preset maximum number of training rounds T is reached, or when the global model changes less than the set threshold in consecutive rounds of updates and converges, the training process ends. Finally, each client forms the final personalized model by recombining the latest updated global underlying network parameters of the server with the personalized high-level network parameters reserved locally. This model can be expressed as:

[0138]

[0139] Thus, this embodiment not only ensures the unity of global shared features, but also fully realizes the personalized optimization of each client based on local data.

[0140] In this embodiment, in addition to the foregoing steps, the server further monitors the dynamic utility scores of each client in real time during each round of training, and combines the historical contribution information to smoothly adjust the weights of each client to ensure the stability and reasonableness of the weight distribution. In addition, the server uses a secure multi-party computation protocol (SMC) to encrypt and aggregate the underlying model parameters uploaded by each client to prevent parameter leakage during transmission, thereby enhancing data privacy protection. Each client also introduces an adaptive learning rate scheduling strategy during the local training process, dynamically adjusting the learning rate according to the magnitude of the current training loss decrease, so as to ensure a faster convergence speed in the initial stage and achieve stable parameter updates in the later stage, further improving the model performance.

[0141] Specifically, after receiving the updates from each client, the server performs smooth filtering on the contribution utility scores of each client, and then completes the encrypted aggregation based on the security protocol; the client automatically adjusts the local learning rate according to the change in the training loss to ensure that the model update is both rapid and stable.

[0142] It can be understood that this embodiment not only improves the security of the global model update, but also effectively balances fast convergence and stable optimization by dynamically adjusting various hyperparameters, and is applicable to practical application scenarios with high data sensitivity and complex network environments.

[0143] In this embodiment, in view of the possible client transmission delay or abnormal situation during the federated learning process, the server introduces a delay tolerance mechanism. For clients that have not uploaded updates for a long time, the server uses historical parameters or the previous round of global model parameters for substitution to ensure the continuity and stability of the overall aggregation process. At the same time, in order to further improve the feature distribution alignment effect, this embodiment adds a domain adaptation module on the basis of the original maximum mean discrepancy (MMD) to make a finer-grained adjustment of the difference between local features and global features and enhance their consistency. In addition, in order to reduce the communication overhead, this embodiment also introduces a model pruning technology on the client side, only transmitting key parameters to further reduce the bandwidth consumption.

[0144] Specifically, when the server detects a client delay, it automatically enables the parameter compensation mechanism and adjusts the feature alignment process through the domain adaptation algorithm; the client prunes the model before local training to reduce the number of parameters to be transmitted.

[0145] It can be understood that in this embodiment, by tolerating client latency and enhancing feature alignment, the global model can still maintain efficient updates and stable convergence in a non-i.i.d. data environment, while significantly reducing the communication burden and data transmission risk, and further improving the generalization performance of the overall model.

[0146] The implementation method of this application combines global feature sharing and local personalized training, significantly reducing the communication overhead between each client and the server, saving bandwidth resources. At the same time, with the support of technical means such as a dynamic utility score mechanism, a composite loss function, and adaptive parameter adjustment, an effective balance between the global model and the local model is achieved. Further introducing measures such as secure aggregation, latency tolerance, and domain adaptation alignment not only improves data privacy protection and transmission security but also significantly enhances the convergence speed, robustness, and generalization ability of the model in a non-i.i.d. data environment. Generally speaking, this implementation method has high practical application value and is applicable to various scenarios such as large-scale distributed data processing, intelligent decision-making, and privacy protection.

[0147] Figure 2 It is a schematic diagram of the modules of a federated learning system based on global feature sharing provided by an embodiment of this application. As Figure 2 shown, the federated learning system 10 based on global feature sharing includes a server 11 and clients 12. The functions of the server 11 and the clients 12 are the same as or similar to those in Figure 1 , and this application will not elaborate here. It can be understood that Figure 2 only shows an embodiment including one client 12. In practical applications, those skilled in the art can set the number of clients 12 according to actual needs, and this application does not limit it here.

[0148] Figure 3 It is a schematic diagram of the modules of an electronic device provided by an embodiment of this application. As Figure 3 shown, the electronic device 20 includes at least the following parts: a processor 21 and a memory 22.

[0149] In the embodiment of this application, the memory 22 is used to store executable instructions of the processor 21, and the processor 21 is configured to implement the federated learning method based on global feature sharing as Figure 1 shown when executing the instructions.

[0150] In the embodiment of this application, a computer-readable storage medium includes instructions that direct the device to execute the method in the first aspect. For example, the instructions direct the device to execute the federated learning method based on global feature sharing shown in steps S1OO to S9OO in Figure 1 .

[0151] The program operating in the electronic device 2O according to an embodiment of the present application may be a program that controls a central processing unit (CPU) or the like to implement the functions of the above-described embodiments related to a solution of the present invention (a program that causes a computer to function). Then, the information processed by these devices is temporarily stored in a random access memory (RAM) during its processing, and then stored in various ROMs such as a read-only memory (Read Only Memory, Flash ROM), a hard disk drive (Hard Disk Drive: HDD), etc. It is read out, corrected, and written by the CPU as needed.

[0152] It should be noted that a part of the electronic device 2O of the above-described embodiment can also be implemented by a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and is implemented by reading the program recorded on the recording medium into a computer system and executing it.

[0153] It should be noted that the "computer system" mentioned here refers to the computer system built into the electronic device 2O, and a computer system including hardware such as an OS and peripheral devices is adopted. In addition, the "computer-readable recording medium" refers to a removable medium such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, etc., and a storage device such as a hard disk built into the computer system.

[0154] Moreover, the "computer-readable recording medium" may include: a medium that stores a program dynamically for a short time, such as a communication line in the case of transmitting a program via a network such as the Internet or a communication line such as a telephone line; a medium that stores a program for a fixed time, such as a volatile memory inside a computer system of a server or a client in this case. In addition, the above program may be a program for implementing a part of the above functions, and may also be a program that can implement the above functions by combining with a program already recorded in the computer system.

[0155] In addition, the electronic device 2O in the above-described embodiment can also be implemented as an aggregate (device group) composed of multiple devices. Each device constituting the device group may have a part or all of the functions or function blocks of the electronic device 2O of the above-described embodiment. As the device group, it is sufficient to have all the functions or function blocks of the electronic device 2O.

[0156] It can be understood that the federated learning method, system 10, electronic device 20, and storage medium provided by the embodiments of the present application based on global feature sharing can significantly reduce the data transmission volume between each client and the server, greatly save bandwidth resources, and improve the efficiency of large-scale distributed training. This method combines global low-level feature sharing and local high-level personalized training, effectively balancing global generality and local personalized needs, and ensuring the stable convergence and high-precision performance of the global model in a non-independent and identically distributed data environment. At the same time, the introduction of the dynamic utility score mechanism makes the client selection and model aggregation processes more reasonable, further enhancing the generalization ability and robustness of the overall model.

[0157] Those of ordinary skill in the art of this technology should recognize that the above embodiments are only used to illustrate the present application, rather than to limit the present application. As long as appropriate changes and variations made to the above embodiments fall within the scope of the spirit of the present application, they fall within the scope of protection required by the present application.

Claims

1. A federated learning method based on global feature sharing, which is applied to a configuration server and multiple clients, and is characterized in that, The method includes: Applying the server to initialize the underlying feature extraction network of the global model and sharing the parameters of the underlying feature extraction network to all the clients; Each of the clients configures its local high-level feature extraction network according to its local data distribution characteristics, so as to form a composite model including the underlying feature extraction network and the local high-level feature extraction network at each client; The server dynamically calculates the contribution utility scores of each client according to the differences between the underlying model parameters uploaded by each client and the current global underlying network parameters, as well as the local data scales of each client, constructs a probability distribution according to the utility scores, and then selects a set of clients participating in the current round of training through weighted random sampling; The server is used to send the global underlying network parameters to the selected clients; After receiving the global underlying network parameters sent by the server, the selected clients combine the global underlying network parameters with the local high-level network parameters to form a complete local model; The selected clients perform forward propagation on the local data using the global underlying network, extract global features, and cache the statistical information of the global features for establishing benchmark parameters for feature distribution alignment; The selected clients train the local model, and after the training is completed, only upload the updated underlying network parameters to the server; The server weights and aggregates the uploaded underlying network parameters according to the contribution utility scores of each selected client to generate a new generation of global underlying network parameters; The selected clients recombine the updated global underlying network parameters with the local high-level network parameters reserved locally to form a new composite model until the model reaches a preset convergence condition.

2. The federated learning method based on global feature sharing according to claim 1, wherein The server dynamically calculates the contribution utility scores of each client according to the differences between the underlying model parameters uploaded by each client and the current global underlying network parameters, as well as the local data scales of each client, including: u i = n i · max(d i , ∈) Among them, u i is the dynamic utility score, n i is the local data volume, is the Euclidean distance between the underlying model uploaded by client i in the previous round and the global underlying model, is the parameter extracted by the underlying feature extraction network, W base,t-1 is the parameter extracted by the underlying feature extraction network in the previous round, ∈ is a minimum value to ensure that when d i = 0, the client still has a certain probability of participating in this round of training.

3. The federated learning method based on global feature sharing according to claim 2, wherein The method further includes: The server generates a probability distribution for client selection according to the dynamic utility scores of each client, and the calculation formula of the probability distribution is: p i represents the sampling weight obtained by the client based on its contribution degree, where N is the total number of clients currently participating in federated learning, is the sum of the utility scores of all N clients; The server performs probability-weighted random sampling and selects the set of clients to participate in the current round of update.

4. The federated learning method based on global feature sharing according to claim 1, wherein The clients evaluate during the local training process using a composite loss function, and the composite loss function includes a task loss, a parameter alignment regularization loss, and a feature distribution alignment regularization loss; The parameter alignment regularization loss is calculated by the following formula: Among them, λ is a regularization hyperparameter used to control the deviation between the local underlying model and the previous-round global underlying model, is the parameter extracted by the underlying feature extraction network.

5. The federated learning method based on global feature sharing according to claim 4, wherein The feature distribution alignment regularization loss is calculated by the maximum mean discrepancy algorithm, and the calculation formula of the maximum mean discrepancy algorithm is: Among them, is the feature distribution alignment regularization loss, is the maximum mean discrepancy, μ is the hyperparameter of the feature alignment regularization term, φ is the Gaussian kernel function, H is the reproducing kernel Hilbert space, m is the number of samples used to calculate the mean in the local feature set, and η is the number of samples used to calculate the mean in the global feature set.

6. The federated learning method based on global feature sharing according to claim 4, wherein The clients update the local model using the gradient descent method, and the update rule of the clients is: where η is the learning rate, is the local model parameter updated after the end of the (t + 1)-th round of training for client i, is the local model parameter of client i during the t-th round of training, is the gradient of the composite loss function calculated for the local model parameter of client i during the t-th round of training.

7. The federated learning method based on global feature sharing according to claim 2, wherein The server is further used for: Performing weighted average update on the global underlying model according to the corresponding dynamic utility scores for the underlying models uploaded by each client; Among them, W base,t+1 is the updated global underlying model, is the underlying model uploaded by client i, d i is the dynamic utility score of client i, S t is the set of clients participating in training in this round.

8. A federated learning system based on global feature sharing, which is applied to implement the federated learning method based on global feature sharing according to any one of claims 1 to 7, characterized in that The system includes: a server and multiple clients; The server is used to initialize the underlying feature extraction network of the global model and share the parameters of the underlying feature extraction network to all the clients; Each of the clients configures the local high-level feature extraction network according to its local data distribution characteristics, so as to form a composite model including the underlying feature extraction network and the local high-level feature extraction network at each client; The server dynamically calculates the contribution utility scores of each client according to the differences between the underlying model parameters uploaded by each client and the current global underlying network parameters and the local data scales of each client, constructs a probability distribution according to the utility scores, and then selects a set of clients participating in this round of training through weighted random sampling; The server is used to send the global underlying network parameters to the selected clients; After receiving the global underlying network parameters sent by the server, the selected clients combine the global underlying network parameters with the local high-level network parameters to form a complete local model; The selected clients perform forward propagation on the local data using the global underlying network, extract global features, and cache the statistical information of the global features for establishing benchmark parameters for feature distribution alignment; The selected clients train the local model, and after the training is completed, only upload the updated underlying network parameters to the server; The server performs weighted aggregation on the uploaded underlying network parameters according to the contribution utility scores of each selected client to generate a new generation of global underlying network parameters; The selected clients recombine the updated global underlying network parameters with the local high-level network parameters reserved locally to form a new composite model until the model reaches the preset convergence condition.

9. An electronic device, characterized in that, Comprising: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to implement the federated learning method based on global feature sharing according to any one of claims 1 to 7 when executing the instructions.

10. A computer-readable storage medium, characterized in that, Including instructions, the instructions instructing the device to execute the federated learning method based on global feature sharing according to any one of claims 1 to 7.

Images