Federal learning model training method based on distributed random convex difference optimization

Abstract

The invention discloses a federated learning model training method based on distributed random convex difference optimization, and belongs to the technical field of federated learning model training, and the method specifically comprises the following steps: S1, system initialization; s2, according to the federated machine learning task selected in the S1 and the machine learning model, constructing a federated learning model training-oriented distributed optimization model with a convex difference structure; s3, the central server designs a solution architecture composed of external iteration and internal iteration and initializes relevant parameters required by the iteration process; s4, the client side executes local model parameter updating in parallel; s5, the central server executes global model parameter updating; and S6, the central server executes algorithm termination condition judgment and outputs optimal global model parameters. According to the federal learning model training method based on distributed random convex difference optimization, convergence is faster and more stable, and better generalization and interpretability can be obtained under non-convex regularization.

Description

Technical Field

[0001] This invention relates to the field of federated learning model training technology, and in particular to a federated learning model training method based on distributed stochastic convexity optimization. Background Technology

[0002] With the rapid development of artificial intelligence and big data technologies, traditional centralized training can no longer meet the efficiency and performance requirements of model training under large-scale data. Against this backdrop, federated learning technology, designed for large-scale distributed training scenarios, has emerged. Efficient and accurate federated learning model training algorithms can not only significantly improve the model's predictive performance and convergence efficiency, but also effectively optimize the utilization efficiency of computing resources. Therefore, designing an efficient and accurate model training algorithm for federated learning models is key to realizing the large-scale deployment of artificial intelligence.

[0003] However, federated learning model training is constrained by multiple factors, including task complexity, the large scale of data samples, and the federated collaborative training architecture, leading to poor performance of traditional model training methods. Specifically, federated learning model training suffers from the following three problems: First, federated learning model training faces the problem of minimizing non-convex and non-smooth loss functions. The training process of federated learning models can be transformed into a problem of minimizing the loss function. In practical applications, to adapt to complex prediction or classification tasks, the deep neural networks used in federated learning models usually contain complex model structures, which inevitably makes their corresponding loss functions non-convex and non-smooth. However, existing federated learning model training algorithms usually assume that the gradient of a non-convex loss function is Lipschitz continuous when dealing with it. Therefore, when these traditional algorithms deal with the minimization problem of non-convex and non-smooth functions that do not have the above-mentioned assumptions, the algorithms are prone to getting trapped in local suboptimal solutions, or even causing the model to diverge.

[0004] Secondly, federated learning model training faces the "large-sum" problem of loss functions. The model's local loss function, mathematically speaking, is a weighted sum of losses over a massive number of local samples, exhibiting a typical "large-sum" structure. Existing optimization algorithms for this structure are generally divided into two categories: one uses gradient descent, calculating the precise gradient based on all sample data, resulting in excessive computational cost; the other uses stochastic gradient descent, estimating the gradient only based on a subset of sample data. While these algorithms reduce computational cost to some extent, they introduce significant random noise, leading to excessively large gradient variance, thus affecting the stability of the model training process and the model's convergence performance.

[0005] Finally, federated learning model training faces the "client drift" problem. Traditional optimization algorithms for non-convex functions typically assume that all sample data can be centrally stored and computed. The algorithm minimizes the loss function composed of all samples on a unified model parameter, and the convergence point of the algorithm can be regarded as the ideal global optimum. When such centralized non-convex function optimization algorithms are transferred to federated learning model training, the model parameters are updated separately on each client based on local sample data. The model parameters of different clients drift towards their respective local optima, resulting in significant differences in the optimal model parameters between different clients. On this basis, the global aggregation performed by the central server can only perform a simple average of these drifted local model parameters, which is difficult to align with the global optimum in the centralized scenario. Client drift then manifests as a systematic shift in the overall model parameters at the global level, directly limiting the convergence accuracy that the algorithm can achieve. The client drift problem is further amplified when there is strong data heterogeneity among clients, and may even lead to instability and divergence in the training process. Summary of the Invention

[0006] The purpose of this invention is to provide a federated learning model training method based on distributed stochastic convexity optimization. This method introduces stochastic variance reduction gradient and convexity decomposition mechanisms into the model training process, and applies near-end operators to the model parameter update process. This significantly solves the optimization problem of non-convex objective functions in federated learning model training, effectively reduces gradient variance and communication overhead caused by massive samples, and reduces consensus bias generated during training. It helps the algorithm efficiently train federated learning models with both high accuracy and strong robustness, improving the model's convergence stability and final performance.

[0007] To achieve the above objectives, this invention provides a federated learning model training method based on distributed stochastic convexity optimization, specifically including the following steps: S1. System initialization: The central server selects clients to form a federated learning training cluster; the central server selects the federated machine learning task and its corresponding machine learning model. S2. Based on the federated machine learning task and machine learning model selected in S1, construct a distributed optimization model with a convex structure for training federated learning models. S3. The central server designs a solution architecture consisting of external and internal iterations and initializes the relevant parameters required for the iteration process. The central server sets the initial global model parameters and deploys the global model parameters, solution architecture, parameters required for the iteration process, and the local model in S2 to the client. S4. Clients perform local model parameter updates in parallel: Each client updates its local model parameters based on the received global model parameters and its local private data, using the convexity algorithm and random variance reduction technique, and then uploads the updated local model parameters to the central server. S5. The central server performs a global model parameter update: Based on all the client local model parameters received in S4, the central server uses a federated averaging algorithm to generate new global model parameters and deploys these global model parameters to all clients. S6. The central server executes the algorithm termination condition judgment and outputs the optimal global model parameters.

[0008] Preferably, the specific steps for system initialization in S1 are as follows: S11, Central Server Proposed A training cluster consisting of [number] clients; S12. The central server determines the parameter vector to be optimized based on the selected federated machine learning task and machine learning model. In addition, the empirical risk function and the regularization term.

[0009] Preferably, in S2, the client selects the federated machine learning task, machine learning model, and parameter vector in S1. Construct the local objective function, and the formula for calculating the local objective function is as follows: ; in, For the client The number of samples in the local private dataset held. For the sample The loss function, whose gradient is about yes -Lipsitz continuous, For the client In parameters The empirical risk function is as follows: For regularization functions, It is a non-negative constant used to control the strength of regularization. For regularization terms; Based on the aforementioned local objective function, the client Build a local model ; Build the corresponding global model ; The empirical risk function in the above global model and regularization term Perform convexity decomposition, that is, decompose it into the difference of two convex functions, and transform the above global model into a global model with convexity structure. The calculation formula is as follows: ; in, For empirical risk function Convexity decomposition, , , For any one greater than constant, yes Convexity decomposition.

[0010] Preferably, the specific operation steps in S3 are as follows: S31. The central server is designed with a solution architecture consisting of external iteration and internal iteration: At the beginning of each round of external iteration, the central server deploys global model parameters to each client. After receiving the global model parameters, the client uses the convexity algorithm and random variance reduction technique based on its local private data to perform multiple rounds of internal iteration updates in parallel, obtain the updated local model parameters, and upload them to the central server. After receiving the local model parameters uploaded by each client, the central server uses the federated average algorithm to perform aggregation updates, obtain the new global model parameters, and complete one round of external iteration update. S32. Parameters required for central server initialization of the iteration process: Setting the maximum number of external iteration rounds in the federated learning training process. Set an external iteration counter This sets the total number of internal iterations that each client needs to execute in each round of external iteration. Define the number of samples required for each client to compute the stochastic gradient during its internal iterations. ; S33. The central server generates initial model parameters for a global model. ; S34, The central server will send the initial model parameters Solving the architecture and the parameters required for the iterative process. , , , And the local model in S2 is deployed to each client.

[0011] Preferably, the client in S4 Accept the Global model parameters of the next outer iteration Client The specific steps for performing local model parameter updates in parallel are as follows: S41. Calculate the full gradient snapshot: Client Based on the Global model parameters of the next outer iteration and its local private data, parallel computing functions The full gradient on the local dataset is used as a baseline for variance reduction, and the specific calculation formula is as follows: ; in, For the client Based on its local dataset, the first Individual sample data, in global model parameters The function calculated at the location The gradient; Set the first The initial model parameter point for the internal iteration of the local model in the next external iteration is: ; S42, Execution Internal iteration step, for internal iteration step The specific steps are as follows: (1) Constructing a mini-batch sample index set: Client Parallel random sampling without replacement is used to sample from its contents. From a local dataset of data samples, construct a dataset of size [size missing]. Mini-batch sample index set ; (2) Constructing the gradient for reducing random variance: Client Based on the sample index set obtained in the previous step And the benchmark point for variance reduction obtained in S41 parallel computing At model parameter points The gradient of random variance reduction at point A is calculated using the following formula: ; in, In the first In the next external iteration, the client The execution of the first The input model parameters used in the next internal iteration; (3) Client parallel computing Input model parameters subgradient at And reduce the gradient with the random variance obtained in the previous step. The result of addition exist The stochastic subgradient estimate is shown below: ; (4) Client Parallel preservation of local objective function Convex function part ,Will Concave function part Replace with affine approximation and add proximal items ,get The convex function is approximated as: ; in, For the first In the next external iteration, the client The execution of the first The learning rate for each internal iteration; (5) Client Parallel construction of local models for updating local model parameters ; (6) Client Solving using proximal gradient descent in parallel To obtain the updated model parameters : ; in, for exist gradient at, yes At point Proximal operators at the location; S43. Complete local update: Client Parallel completion After the first internal iteration, set the last model parameter output after the internal iteration. For the client In the The final model parameters that need to be uploaded to the central server after the next external iteration and will Uploaded to the central server.

[0012] Preferably, in S5, the central server performs global model parameter updates: the central server performs aggregation operations, that is, collects data from all... The client in the first The local model parameters obtained from the second external iteration update The federal average was calculated to obtain the first Global model parameters used in the next outer iteration: ; Subsequently, the central server aggregates the global model parameters. Broadcast to all clients.

[0013] Preferably, in S6, the central server determines the algorithm termination criterion as follows: when the algorithm has not yet converged and the external iteration count has reached a certain threshold... The preset maximum value was not reached. Repeat steps S4 and S5, and update the iteration counter after each round. When the algorithm terminates, it outputs the final global model parameters. As the optimal solution.

[0014] Therefore, the federated learning model training method based on distributed stochastic convexity optimization described above, as used in this invention, has the following beneficial effects: (1) Solve the problem of minimizing non-convex loss function: By performing convex difference decomposition on the loss function, the non-convex loss function is transformed into the difference of two convex functions, and on this basis, a federated iterative architecture is constructed, so that the non-convex objective function can still be stably solved in the federated scenario, reducing the risk of traditional federated model training algorithms easily getting trapped in local suboptimal solutions and training divergence under complex deep neural network models. (2) Solving the “big sum” problem: In the client-side local gradient calculation in the federated scenario, a stochastic variance reduced gradient estimation is introduced. The local full gradient calculated based on all local samples at the global snapshot point obtained in the previous iteration is used as the control variable. The difference between the gradient of the small batch sampled at the current parameter point and the small batch gradient at the snapshot point is superimposed on the above local full gradient, thereby constructing an unbiased gradient estimate with small variance for the local “big sum” structure loss function. The above design does not require traversing all local samples in each round, which significantly reduces the computation and communication overhead of a single iteration and effectively suppresses the gradient variance introduced by the summation of large-scale samples, thereby accelerating the overall convergence speed of the algorithm. (3) Suppressing the "client drift" problem: When each client updates the model parameters locally, it uses the global model parameters broadcast by the central server in the previous round as a unified snapshot point. The local full gradient is calculated on this snapshot point and the gradient of the current mini-batch samples is corrected. This makes the local model parameter update direction aligned with the same reference point in a statistical sense, which significantly suppresses the drift of model parameters between different clients. On this basis, the central server performs weighted aggregation on the local model parameters updated by each client. The resulting global model parameters are closer to the global optimal solution in the centralized scenario. This design alleviates the systematic bias caused by client drift and improves the convergence accuracy and training stability of federated learning model training.

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] Figure 1 This invention provides a visualization of a clothing classification task dataset, based on a federated learning model training method using distributed stochastic convexity optimization. Figure 2 This is a diagram of a convolutional neural network architecture based on a federated learning model training method for distributed stochastic convexity optimization according to an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0018] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0019] Example 1 This invention provides a federated learning model training method based on distributed stochastic convexity optimization, specifically including the following steps: S1. System initialization: The central server selects clients to form a federated learning training cluster; the central server selects the federated machine learning task and its corresponding machine learning model. The specific steps for system initialization are as follows: S11, Central Server Proposed A training cluster consisting of [number] clients; S12. The central server determines the parameter vector to be optimized based on the selected federated machine learning task and machine learning model. In addition, the empirical risk function and the regularization term.

[0020] S2. Based on the federated machine learning task and machine learning model selected in S1, construct a distributed optimization model with a convex structure for training federated learning models. The client selects the federated machine learning task, machine learning model, and parameter vector in S1. Construct the local objective function, and the formula for calculating the local objective function is as follows: ; in, For the client The number of samples in the local private dataset held. For the sample The loss function, whose gradient is about yes -Lipsitz continuous, For the client In parameters The empirical risk function is as follows: For regularization functions, It is a non-negative constant used to control the strength of regularization. For regularization terms; Based on the aforementioned local objective function, the client Build a local model ; Build the corresponding global model ; The empirical risk function in the above global model and regularization term Perform convexity decomposition, that is, decompose it into the difference of two convex functions, and transform the above global model into a global model with convexity structure. The calculation formula is as follows: ; in, For empirical risk function Convexity decomposition, , , For any one greater than constant, yes Convexity decomposition.

[0021] S3. The central server designs a solution architecture consisting of external and internal iterations and initializes the relevant parameters required for the iteration process. The central server sets the initial global model parameters and deploys the global model parameters, solution architecture, parameters required for the iteration process, and the local model in S2 to the client. The specific steps are as follows: S31. The central server is designed with a solution architecture consisting of external iteration and internal iteration: At the beginning of each round of external iteration, the central server deploys global model parameters to each client. After receiving the global model parameters, the client uses the convexity algorithm and random variance reduction technique based on its local private data to perform multiple rounds of internal iteration updates in parallel, obtain the updated local model parameters, and upload them to the central server. After receiving the local model parameters uploaded by each client, the central server uses the federated average algorithm to perform aggregation updates, obtain the new global model parameters, and complete one round of external iteration update. S32. Parameters required for central server initialization of the iteration process: Setting the maximum number of external iteration rounds in the federated learning training process. Set an external iteration counter This sets the total number of internal iterations that each client needs to execute in each round of external iteration. Define the number of samples required for each client to compute the stochastic gradient during its internal iterations. ; S33. The central server generates initial model parameters for a global model. ; S34, The central server will send the initial model parameters Solving the architecture and the parameters required for the iterative process. , , , And the local model in S2 is deployed to each client.

[0022] S4. Clients perform local model parameter updates in parallel: Each client updates its local model parameters based on the received global model parameters and its local private data, using the convexity algorithm and random variance reduction technique, and then uploads the updated local model parameters to the central server. Client Accept the Global model parameters of the next outer iteration Client The specific steps for performing local model parameter updates in parallel are as follows: S41. Calculate the full gradient snapshot: Client Based on the Global model parameters of the next outer iteration and its local private data, parallel computing functions The full gradient on the local dataset is used as a baseline for variance reduction, and the specific calculation formula is as follows: ; in, For the client Based on its local dataset, the first Individual sample data, in global model parameters The function calculated at the location The gradient; Set the first The initial model parameter point for the internal iteration of the local model in the next external iteration is: ; S42, Execution Internal iteration step, for internal iteration step The specific steps are as follows: (1) Constructing a mini-batch sample index set: Client Parallel random sampling without replacement is used to sample from its contents. From a local dataset of data samples, construct a dataset of size [size missing]. Mini-batch sample index set ; (2) Constructing the gradient for reducing random variance: Client Based on the sample index set obtained in the previous step And the benchmark point for variance reduction obtained in S41 parallel computing At model parameter points The gradient of random variance reduction at point A is calculated using the following formula: ; in, In the first In the next external iteration, the client The execution of the first The input model parameters used in the next internal iteration; (3) Client parallel computing Input model parameters subgradient at And reduce the gradient with the random variance obtained in the previous step. The result of addition exist The stochastic subgradient estimate is shown below: ; (4) Client Parallel preservation of local objective function Convex function part ,Will Concave function part Replace with affine approximation and add proximal items ,get The convex function is approximated as: ; in, For the first In the next external iteration, the client The execution of the first The learning rate for each internal iteration; (5) Client Parallel construction of local models for updating local model parameters ; (6) Client Solving using proximal gradient descent in parallel To obtain the updated model parameters : ; in, for exist gradient at, yes At point Proximal operators at the location; S43. Complete local update: Client Parallel completion After the first internal iteration, set the last model parameter output after the internal iteration. For the client In the The final model parameters that need to be uploaded to the central server after the next external iteration and will Uploaded to the central server.

[0023] S5. The central server performs a global model parameter update: Based on all the client local model parameters received in S4, the central server uses a federated averaging algorithm to generate new global model parameters and deploys these global model parameters to all clients. The central server performs global model parameter updates: the central server performs aggregation operations, that is, it collects data from all... The client in the first The local model parameters obtained from the second external iteration update The federal average was calculated to obtain the first Global model parameters used in the next outer iteration: ; Subsequently, the central server aggregates the global model parameters. Broadcast to all clients.

[0024] S6. The central server executes the algorithm termination condition judgment and outputs the optimal global model parameters; The central server determines the algorithm's termination criterion as follows: when the algorithm has not yet converged and the external iteration count has reached a certain threshold... The preset maximum value was not reached. Repeat steps S4 and S5, and update the iteration counter after each round. When the algorithm terminates, it outputs the final global model parameters. As the optimal solution.

[0025] Example 2 This embodiment considers a clothing image classification task, namely a ten-category image classification task that takes grayscale clothing images as input and clothing category labels as output.

[0026] S1. System Initialization: The datasets from the assigned tasks are distributed and stored across 10 clients, each client... Holding local private datasets ,in, For the client The first One sample, For the client The total number of local samples, For each sample, use its local sample index. From the input image tensor and its corresponding category label Composition, input image tensor ,in, This represents the image height, which is the number of pixels in the vertical direction of the image. This represents the image width, which is the number of pixels in the horizontal direction of the image. This refers to the number of image channels, i.e., the dimension of the vector at each pixel. Category Labels ,in, This represents the total number of categories. Conceptually merging all local samples from all clients yields the overall dataset for this classification task. ,in, The total number of samples in the dataset. This is the sample index. The dataset in this embodiment is from the FMNIST dataset, and some examples are shown below. Figure 1 As shown.

[0027] The central server selects a neural network model, such as Figure 2 As shown, this embodiment uses a... A convolutional neural network (CNN) model with parameters set to be used as a classification model for image classification tasks. This model consists of 4 convolutional layers, 2 max-pooling layers, 1 flattening layer, 1 fully connected layer, and 1 output layer. Figure 2 The structure shown is composed of the following components, and its calculation formula is as follows: ; in, This indicates that the CNN model has parameters Below, for the input image tensor The output after the action, i.e., a dimensional prediction probability vector, The Each component is denoted as Indicates the model predicts the input. Belongs to the The probability of a class .

[0028] The central server determines the empirical risk function and regularization term for the clothing image classification task and the CNN model. In this embodiment, the empirical risk function is first determined as follows: ,in For the client The Sample The loss function is applied. This embodiment introduces a regularization term. For the logarithm and the regularization term Log-sum, i.e. ; in, For model parameters The total dimension It is a parameter The One portion, It is a non-negative constant used to control the regularization term. It is a very small positive number to ensure numerical stability.

[0029] S2, the central server builds local and global models. The central server first builds the client based on the system initialization steps in S1. Local objective function: ; based on Central server builds client The local model is .

[0030] Furthermore, based on the aforementioned local model, the central server constructs a global model as follows: .

[0031] The central server is based on the local objective function. Convex decomposition constructs the aforementioned local and global models into a distributed optimization model with a convex structure for federated learning model training. To achieve this goal, the central server first... Perform convexity decomposition: ; in, , , , .

[0032] Then, the central server after convexity decomposition Substitute the above local model and global model This yields a distributed optimization model with a convex structure, designed for federated learning model training.

[0033] S3. The central server designs a solution architecture consisting of external and internal iterations and initializes the relevant parameters required for the iteration process. In steps S1 and S2, the CNN model and the local model are determined. Following the convex structure, this embodiment constructs a federated solution architecture consisting of nested external and internal iterations in step S3. The central server sets the maximum number of external iteration rounds for the entire federated solution architecture. Set an external iteration counter ; Define the total number of internal iterations that each client needs to execute in each round of external iteration. ; Define the number of samples required for each client to compute the stochastic gradient during its internal iterations. .

[0034] The central server sets the initial global model parameters and deploys the global model parameters, solution architecture, parameters required for the iteration process, and the local model in S2 to the client. The central server assigns initial values ​​to all trainable parameters and flattens them, constructing a one-dimensional real vector. These are used as the initial global model parameters. After completing the above steps, the central server will use the global model parameters... Solving the architecture and the parameters required for the iterative process. , , , And the local model in S2 is deployed to each client.

[0035] S4. Clients perform local model parameter updates in parallel: After completing the federated solution architecture, parameter initialization, and parameter deployment in S3, this step is initiated, and each client performs the updates. The internal iterative update is performed in parallel locally based on the private dataset and the local model to obtain the updated local model parameters. And upload it to the central server. Specific steps include: Client Based on the Global model parameters of the next outer iteration Parallel computation function In its local dataset Full gradient on: ; in, For the indicator function, when the local sample category label Time to take Otherwise take . for In the sample Unnormalized category scores on satisfy: .

[0036] Then, the client Set the first The initial model parameter point for the internal iteration of the local model in the next external iteration is: and execute Internal iteration step, for internal iteration step : Client parallel computing In model parameters The gradient of random variance reduction at point is denoted as . Client First, parallel processing from its contents A dataset of size is randomly drawn without replacement from a local dataset containing data samples. Mini-batch sample index set Further based on the sample index set And the full gradient obtained in the previous step parallel computing In model parameters Gradient reduction of random variance at point: ; in, For the first In the next external iteration, the client The execution of the first The input model parameters used in each internal iteration.

[0037] Client parallel computing exist stochastic subgradient : ; in, Based on Calculated; for The identity matrix, It is composed of vectors of The diagonal matrix generated by each component.

[0038] Client Parallel construction Approximate convex objective function and local model Client reserve convex function part Then, based on the stochastic subgradient estimation from the previous step... , the concave function part Replace with affine approximation and add proximal items ,get The convex function is approximated as follows: ; in, This represents the current learning rate at the local step. Based on this convex objective function, the client... Obtain the local model used to update the local model parameters. .

[0039] Client Parallel solution of the model yields updated model parameters In this embodiment, the client... Solving the above local model in parallel using the proximal gradient descent method yields: ; in, ( ) is an element-wise symbolic function. It represents the Hadamah accumulation. For elements all equal to 1 dimensional vector, For elements all equal to 0 dimensional vector, The expression is: ; Client Parallel completion After the first internal iteration, take As the first The local update results of the external iteration are then uploaded to the central server to update the model parameters.

[0040] S5. The central server performs a global model parameter update: The central server receives all client-local model parameters received in S4, and the... The local model parameters obtained from external iteration updates The federal average was calculated to obtain the first Global model parameters used in the next outer iteration: ; Subsequently, the central server aggregates the global model parameters. Broadcast to all clients.

[0041] S6. The central server executes the algorithm termination condition judgment and outputs the optimal global model parameters: The central server repeats steps S4 and S5, updating the iteration counter after each round. Until the algorithm bracelet or external iteration count is reached. Reaching the preset maximum value At that time, the final global model parameters are output. As the optimal solution.

[0042] Therefore, this invention employs a federated learning model training method based on distributed stochastic convexity optimization. By integrating the convexity algorithm, stochastic variance reduction algorithm, and federated averaging algorithm, it achieves efficient collaborative training on data that is not independent and identically distributed with non-convex loss functions. Compared to traditional methods, it converges faster and more stably, and achieves better generalization and interpretability under non-convex regularization. This invention's method is model and task-friendly, directly applicable to classification tasks, and can be replaced with other prediction models. It possesses advantages such as modularity, pluggability, and ease of engineering implementation, significantly improving the training efficiency and accuracy of federated learning in heterogeneous scenarios and resource-constrained environments.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A federated learning model training method based on distributed stochastic convexity optimization, characterized in that: Specifically, the following steps are included: S1. System initialization: The central server selects clients to form a federated learning training cluster; the central server selects the federated machine learning task and its corresponding machine learning model. S2. Based on the federated machine learning task and machine learning model selected in S1, construct a distributed optimization model with a convex structure for training the federated learning model. S3. The central server designs a solution architecture consisting of external and internal iterations and initializes the relevant parameters required for the iteration process. The central server sets the initial global model parameters and deploys the global model parameters, solution architecture, parameters required for the iteration process, and the local model in S2 to the client. S4. Clients perform local model parameter updates in parallel: Each client updates its local model parameters based on the received global model parameters and its local private data, using the convexity algorithm and random variance reduction technique, and then uploads the updated local model parameters to the central server. S5. The central server performs a global model parameter update: Based on all the client local model parameters received in S4, the central server uses a federated averaging algorithm to generate new global model parameters and deploys these global model parameters to all clients. S6. The central server executes the algorithm termination condition judgment and outputs the optimal global model parameters.

2. The federated learning model training method based on distributed stochastic convexity optimization according to claim 1, characterized in that: The specific steps for system initialization in S1 are as follows: S11, Central Server Proposed A training cluster consisting of [number] clients; S12. The central server determines the parameter vector to be optimized based on the selected federated machine learning task and machine learning model. In addition, the empirical risk function and the regularization term.

3. The federated learning model training method based on distributed stochastic convexity optimization according to claim 2, characterized in that: In S2, the client selects the federated machine learning task, machine learning model, and parameter vector from S1. Construct the local objective function, and the formula for calculating the local objective function is as follows: ； in, For the client The number of samples in the local private dataset held. For the sample The loss function, whose gradient is about yes -Lipsitz continuous, For the client In parameters The empirical risk function is as follows: For regularization functions, It is a non-negative constant used to control the strength of regularization. For regularization terms; Based on the aforementioned local objective function, the client Build a local model ; Build the corresponding global model ; The empirical risk function in the above global model and regularization term Perform convexity decomposition, that is, decompose it into the difference of two convex functions, and transform the above global model into a global model with convexity structure. The calculation formula is as follows: ； in, For empirical risk function Convexity decomposition, , , For any one greater than The constant, yes Convexity decomposition.

4. The federated learning model training method based on distributed stochastic convexity optimization according to claim 3, characterized in that: The specific operation steps in S3 are as follows: S31. The central server is designed with a solution architecture consisting of external iteration and internal iteration: At the beginning of each round of external iteration, the central server deploys global model parameters to each client. After receiving the global model parameters, the client uses the convexity algorithm and random variance reduction technique based on its local private data to perform multiple rounds of internal iteration updates in parallel, obtain the updated local model parameters, and upload them to the central server. After receiving the local model parameters uploaded by each client, the central server uses the federated average algorithm to perform aggregation updates, obtain the new global model parameters, and complete one round of external iteration update. S32. Parameters required for central server initialization of the iteration process: Setting the maximum number of external iteration rounds in the federated learning training process. Set an external iteration counter This sets the total number of internal iterations that each client needs to execute in each round of external iteration. Define the number of samples required for each client to compute the stochastic gradient during its internal iterations. ; S33. The central server generates initial model parameters for a global model. ; S34, The central server will send the initial model parameters Solving the architecture and the parameters required for the iterative process , , , And the local model in S2 is deployed to each client.

5. The federated learning model training method based on distributed stochastic convexity optimization according to claim 4, characterized in that: S4 client Accept the Global model parameters of the next outer iteration Client The specific steps for performing local model parameter updates in parallel are as follows: S41. Calculate the full gradient snapshot: Client Based on the Global model parameters of the next outer iteration and its local private data, parallel computing functions The full gradient on the local dataset is used as a baseline for variance reduction, and the specific calculation formula is as follows: ； in, For the client Based on its local dataset, the first Individual sample data, in global model parameters The function calculated at the location The gradient; Set the first The initial model parameter point for the internal iteration of the local model in the next external iteration is: ; S42, Execution Internal iteration step, for internal iteration step The specific steps are as follows: (1) Constructing a mini-batch sample index set: Client Parallel random sampling without replacement is used to sample from its contents. From a local dataset of data samples, construct a dataset of size [size missing]. Mini-batch sample index set ; (2) Constructing the gradient for reducing random variance: Client Based on the sample index set obtained in the previous step And the benchmark point for variance reduction obtained in S41 parallel computing At model parameter points The gradient of random variance reduction at point A is calculated using the following formula: ； in, In the first In the next external iteration, the client The execution of the first The input model parameters used in the next internal iteration; (3) Client parallel computing Input model parameters subgradient at And reduce the gradient with the random variance obtained in the previous step. The result of addition exist The stochastic subgradient estimate is shown below: ； (4) Client Parallel preservation of local objective function Convex function part ,Will Concave function part Replace with affine approximation and add proximal items ,get The convex function is approximated as: ； in, For the first In the next external iteration, the client The execution of the first The learning rate for each internal iteration; (5) Client Parallel construction of local models for updating local model parameters ; (6) Client Solving using proximal gradient descent in parallel To obtain the updated model parameters : ； in, for exist gradient at, yes At point Proximal operators at the location; S43. Complete local update: Client Parallel completion After the first internal iteration, set the last model parameter output after the internal iteration. For the client In the The final model parameters that need to be uploaded to the central server after the next external iteration and will Uploaded to the central server.

6. The federated learning model training method based on distributed stochastic convexity optimization according to claim 5, characterized in that: In S5, the central server performs global model parameter updates: the central server performs aggregation operations, that is, it collects data from all... The client in the first The local model parameters obtained from the second external iteration update The federal average was calculated to obtain the first Global model parameters used in the next outer iteration: ； Subsequently, the central server aggregates the global model parameters. Broadcast to all clients.

7. The federated learning model training method based on distributed stochastic convexity optimization according to claim 6, characterized in that: In S6, the central server determines the algorithm termination criterion as follows: when the algorithm has not yet converged and the external iteration count has reached a certain threshold... The preset maximum value was not reached. Repeat steps S4 and S5, and update the iteration counter after each round. When the algorithm terminates, it outputs the final global model parameters. As the optimal solution.

Images