Intelligent model training method and system for distributed device collaborative optimization and storage medium

Abstract

The invention belongs to the technical field of federal learning, and discloses an intelligent model training method and system for distributed equipment collaborative optimization and a storage medium, and the algorithm comprises the steps that a client receives an original model sent by a server, and completes an initialization task of the model; the client carries out local training on the received initialization model based on the local data set to obtain a local training model, all basic parameters of local training are obtained through calculation, and a memory-free SR1 matrix is generated; the server randomly extracts part of clients, and the selected clients upload local model training parameters of the clients; updating an ADMM direction, and updating an internal problem; solving a variable-scale near-end mapping problem to obtain a near-end direction, and solving an external problem; and updating the global model according to the obtained near-end direction until the model training is completed. Under the condition that statistical heterogeneity exists, compared with an existing first-order composite optimization algorithm, the algorithm has the advantages that the convergence speed is higher, the convergence condition of the algorithm can be met more quickly, and progressive convergence is achieved.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of federated learning technology, and in particular relates to an intelligent model training method, system and storage medium for distributed device collaborative optimization. Background Art

[0002] The rapid development of Internet technology has brought unprecedented opportunities and challenges to machine learning. In the context of massive data, traditional centralized machine learning model training methods face the problem of high storage and computing costs. At the same time, due to the needs of privacy protection and data security, data silos are prevalent, which limits the quality and effectiveness of model training.

[0003] Federated learning, as an emerging solution, aims to solve these problems. It allows multiple parties to jointly train a global model without sharing their own data. This not only protects data privacy, but also makes it possible to use data scattered in different places to improve the performance of the model. However, federated learning also faces some challenges in practical applications. On the one hand, federated learning usually requires a large number of devices, and there are node restrictions and network bandwidth restrictions, which causes the communication speed between the device and the central server to be much slower than the calculation speed within the node, resulting in low communication efficiency; on the other hand, there are differences in data distribution between devices. Due to the different data sizes and data categories owned by different devices, the trained local models also have large differences. Therefore, how to train a high-performance model on a global scale in the case of heterogeneous data needs to be solved urgently. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides an intelligent model training method, system and storage medium for distributed device collaborative optimization.

[0005] The present invention is implemented as follows: an intelligent model training method for distributed device collaborative optimization, the algorithm

[0006] include:

[0007] S1: The client receives the original model sent by the server and completes the model initialization task;

[0008] S2: The client performs local training on the received initialization model based on the local data set to obtain a local training model, calculates the basic parameters of the local training, and generates a memoryless SR1 matrix;

[0009] S3: The server randomly selects some clients, and the selected clients upload their local model training parameters;

[0010] S4: Update ADMM direction and update internal issues;

[0011] S5: Solve the variable-scale proximal mapping problem, obtain the proximal direction, and solve the external problem;

[0012] S6: Update the global model according to the obtained proximal direction until the model training is completed.

[0013] Further, step S1 is the initialization stage in the federated learning process, in which the client first receives the initial global model from the server, which is designed to be easily adapted to the diverse data sets held by different clients, and considering the communication efficiency and computational cost, a convolutional neural network (CNN) or a recurrent neural network (RNN) is usually selected; then, the client generates a series of initialization parameters based on the received global model, including local primitive parameters, local dual parameters, and key initial parameters for subsequent generation of memoryless SR1 matrices; the generation of these parameters is an important preparation before local training, ensuring that each client can start its training task at a unified starting point; in addition, the client also needs to set up a local training environment to facilitate the use of its unique data for model training and parameter adjustment;

[0014] The parameters generated by S1 are mainly divided into two parts: (1) parameters generated by the server: global original parameter node d0, global model (2) Initialization parameters generated by the client: local original parameters Local dual parameters Generate initial parameters for memoryless SR1 matrix With η.

[0015] Furthermore, S2 is a key part of the federated learning process, in which the client conducts in-depth training on the initialized model received from the server based on its local unique dataset; in this process, the client first initiates the local training task, and then applies its local dataset to carefully adjust and optimize the model; through this training, the client is able to calculate basic parameters that reflect the characteristics of the local data, which are crucial for capturing specific patterns and trends in the dataset;

[0016] After the client trains the model based on local data, it will get the local gradient of this round of updates Then according to the local gradient obtained from the previous round of training Global Modelx k and x k-1 , the memoryless SR1 matrix can be generated. The specific generation algorithm is as follows:

[0017] Input: Iteration round k, node i∈S k , global model x k ,x k -1, local gradient difference

[0018]

[0019] if k = 0 then

[0020] B k =I,u k =v k =0

[0021] else

[0022]

[0023] end

[0024] Output:

[0025] In the algorithm, ρ and Basic parameters set for the approximate matrix generation algorithm, The calculation intermediate variables in the process of generating memoryless SR1 matrix for the client, This is the generated memoryless SR1 matrix;

[0026] The generated memoryless SR1 matrix is ​​used to approximate the Hessian matrix, which has many advantages. First, it paves the way for solving external problems later: the proximal calculation of variable scale is difficult, and the generated memoryless SR1 matrix can make the proximal mapping defined and simple to calculate, degenerating the proximal mapping calculation of variable scale into ordinary proximal mapping calculation, which reduces the difficulty of calculation. Second, it reduces the transmission cost of communication: each client originally needs to upload the Hessian matrix to the central node, and its transmission cost is O(p 2 ), the cost is high. Using an approximate matrix instead of SR1 instead of the Hessian matrix can effectively reduce the communication cost. Third, more effective privacy protection: The Hessian matrix is ​​based on the second-order model gradient of local data, which contains a lot of effective information of local data. Once it is attacked by gradient leakage, the local data set may be stolen. Using its approximate matrix SR1 to replace it can effectively prevent such attacks.

[0027] Furthermore, the S4 is responsible for updating the ADMM direction in the FedQuasiNewton algorithm, wherein the server receives and processes the local model training parameters uploaded by the client, adjusts the search direction by optimizing the augmented Lagrangian function, solves internal problems, and prepares for the next iterative update of the global model;

[0028] In this step, the original parameter d k+1 The update method is as follows:

[0029]

[0030] In this step, the server is required to receive two parameters from the client. and Then the ADMM direction is updated; however, after simplifying the formula, it can be seen that only the client needs to calculate the intermediate calculation variables locally And upload, the server can complete the update of ADMM direction. Through the improvement of this step, the computing pressure of the server can be reduced, and the computing task can be distributed to the selected client; at the same time, the number of parameters transmitted is reduced from two to one, which not only reduces the communication pressure, but also reduces the risk of client privacy leakage.

[0031] Further, the internal problem is solved to obtain d k+1 After that, there is At this point, we only need to ask Then the proximal direction p can be calculated k ;

[0032] Now consider how to calculate the variable scale proximal mapping In fact, the computational difficulty of the variable-scale proximal mapping is usually no less than that of the original problem; for the above problem, when B k When is a positive definite matrix, the above problem has a unique solution. Usually, we take However, this means that each node needs to upload the Hessian matrix to the central node, and the transmission cost is O(p 2 ), the cost is high; at the same time, since the proximal calculation of the variable scale is difficult and the approximate matrix is ​​required to be a positive definite matrix, it is necessary to select a suitable approximate matrix so that the proximal mapping is defined and the calculation is simple; however, since the matrix generated in step S2 of the present invention is a memoryless SR1 matrix, its variable scale proximal mapping can be degenerated into a common proximal mapping, which just meets the requirements;

[0033] The global approximate Hessian matrix of each round of update is S k For all the selected clients; It can be seen that in each iteration, there is no need for nodes to upload the entire Hessian matrix to the central end, only the clients need to Upload it to the central end, and the variable scale near-end mapping can be completed.

[0034] From the proximal direction p k The update formula is:

[0035]

[0036] where x k is the global model parameter of the previous round, dk+1 Calculated in the previous step, prox is the proximal operator;

[0037] At this point, the proximal direction p of the global model update can be obtained k .

[0038] Furthermore, S6 is the final model update stage in the FedQuasiNewton algorithm, which uses the proximal direction solved in step S5 to iteratively update the global model through a specific update formula until the conditions for model training completion are met, such as reaching a predetermined number of iterations or performance indicators, thereby completing the entire federated learning training process;

[0039] The update formula of the global model is:

[0040] x k+1 =x k +tp k

[0041] The step size t is the sequence t∈{1,β,β 2 ,…} the first number that satisfies the following relationship

[0042]

[0043] Specifically, considering a federated learning framework consisting of a central server and m nodes, we have:

[0044]

[0045] in is the local loss function of the ith node, is a convex regularization function.

[0046] Another object of the present invention is to provide an intelligent model training method system for distributed device collaborative optimization based on the intelligent model training method for distributed device collaborative optimization as described in claims 1-6, characterized in that the system specifically includes:

[0047] Model initialization module: the client receives the original model sent by the server and completes the model initialization task;

[0048] The local training module is connected to the model initialization module. The client performs local training on the received initialization model based on the local data set to obtain the local training model, calculates the basic parameters of the local training, and generates a memoryless SR1 matrix.

[0049] The parameter training module is connected to the local training module. The server randomly selects some clients, and the selected clients upload their local model training parameters.

[0050] The update module is connected with the parameter training module to update the ADMM direction and internal issues;

[0051] The solving module is connected with the updating module to solve the variable-scale proximal mapping problem, obtain the proximal direction, and solve the external problem;

[0052] The model updating module is connected to the solving module and updates the global model according to the obtained proximal direction until the model training is completed.

[0053] Another object of the present invention is to provide a computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the intelligent model training method for distributed device collaborative optimization.

[0054] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the intelligent model training method for distributed device collaborative optimization.

[0055] Another object of the present invention is to provide an information data processing terminal, which is used to implement the intelligent model training method system for distributed device collaborative optimization.

[0056] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0057] First, the convergence speed is accelerated: Experimental results show that under the condition of statistical heterogeneity, the algorithm of the present invention has a faster convergence speed than the existing first-order composite optimization algorithm, can reach the convergence condition of the algorithm more quickly, and has asymptotic convergence.

[0058] Improved communication efficiency: By using a single ADMM algorithm to solve the iteration direction of the Newton-type algorithm, the present invention reduces the number of communication rounds required for each iteration, thereby significantly reducing the communication cost. Compared with the traditional Newton-type federated learning algorithm, this algorithm reduces the data transmission requirements between nodes while maintaining the algorithm accuracy.

[0059] Reduced computational complexity: Through the application of the improved ADMM algorithm and the memoryless SR1 formula, the present invention simplifies the computational process of the variable-scale proximal mapping, converts the originally complex computational task into a form that is easier to handle, and reduces the computational difficulty of the algorithm.

[0060] Enhanced privacy protection: This paper uses the memoryless SR1 formula to approximate the Hessian matrix, avoiding the direct transmission of the Hessian matrix and reducing the risk of privacy leakage. In addition, the locality of node data is taken into account in the algorithm design, further protecting the privacy of user data.

[0061] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0062] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0063] The composite optimization second-order algorithm of the present invention effectively reduces the cost of federated learning in actual deployment by reducing communication rounds and data transmission volume, especially in scenarios where model training is required across multiple devices or nodes. This improvement in cost-effectiveness makes federated learning technology more suitable for small and medium-sized enterprises with limited budgets, or for application in areas with underdeveloped network infrastructure. In addition, the algorithm's enhanced protection of data privacy gives it broad application potential in industries with strict requirements for data security, such as healthcare and financial services. With the promotion of technology, it can be foreseen that it will promote the digital transformation of these industries, improve the efficiency and security of data processing, and also bring new growth points and competitive advantages to related companies.

[0064] (2) The technical solution of the present invention solves a technical problem that people have been eager to solve but have never been able to solve successfully:

[0065] The advent of the big data era has made the channels for data generation more diverse. Traditional large-scale machine learning methods train models by concentrating a large amount of data on a single server. However, with the surge in data volume, this method not only requires huge storage space, but also has very high training costs. In the real world, data is often scattered in different organizations and institutions. Due to the need for privacy protection, these data can usually only be stored in their respective systems. Federated learning, as an emerging distributed learning framework, allows model training without migrating user data out of the local device, thereby building a high-performance model across devices. However, federated learning also faces challenges in its implementation, mainly including: (1) high communication costs; (2) statistical heterogeneity of data between devices. To address these challenges, researchers have developed a series of first-order algorithms. However, the convergence speed of these first-order algorithms is affected by the condition number of the problem and is usually slow. Multiple communications are required to achieve satisfactory accuracy, which undoubtedly increases the communication cost. In order to solve these problems, the present invention extends the traditional proximal quasi-Newton method to federated learning and proposes an intelligent model training method for distributed device collaborative optimization. The algorithm is based on the proximal quasi-Newton method and uses a single ADMM algorithm to solve the iteration direction of the Newton-type algorithm. Considering that only some nodes usually participate in each round of updates in federated learning, this paper improves the ADMM algorithm to adapt to the update of some nodes. In addition, this paper also uses the memoryless SR1 formula to approximate the Hessian matrix, which not only simplifies the calculation of the proximal mapping and reduces the difficulty of calculation, but also reduces the transmission cost between nodes, thereby better protecting privacy.

[0066] Third, the technical solution of the present invention aims at the multi-client optimization problem in the federated learning environment, and proposes a federated composite optimization algorithm based on the proximal quasi-Newton method, which solves several key problems in the prior art and has made significant technological progress in industrial applications, as shown in the following:

[0067] 1. Technical problems solved

[0068] (1) Uneven data distribution among clients

[0069] Existing federated learning algorithms usually assume that each client has similar data distribution, but in actual applications, the data distribution between clients is often non-uniform, and there are problems such as large differences in data volume and categories. This will lead to unstable performance of the global model. The present invention introduces the memoryless SR1 matrix and the proximal optimization of the quasi-Newton method, so that each client can perform effective training based on its own local data characteristics, thereby solving the problem of decreased model accuracy caused by uneven data distribution.

[0070] (2) Low communication efficiency in federated learning

[0071] The traditional federated learning process requires frequent exchange of model parameters between the client and the server, which has high communication costs, especially in the case of large-scale participants, which greatly affects the efficiency of the system. The present invention adopts a lightweight global model (such as CNN or RNN), combined with the proximal quasi-Newton method, to optimize local parameters on the client, reducing the communication frequency between the global model and the local model, and significantly improving the communication efficiency of federated learning.

[0072] (3) Excessive consumption of computing resources in model training

[0073] The computational burden in existing federated learning is too heavy, especially on resource-constrained devices such as mobile phones or IoT devices. The present invention uses a memoryless SR1 matrix for optimization locally on the client, avoiding the direct solution of the Hessian matrix, reducing the computational complexity and the consumption of computing resources, thereby enabling efficient training in resource-constrained environments.

[0074] (4) Global models have poor adaptability to diverse data sets

[0075] The global model performs poorly on different clients due to differences in data sets, and is difficult to generalize to the data distribution of all participants. Through the technical solution of the present invention, after receiving the global model, the client generates local primitive parameters and dual parameters in combination with the local data set, and gradually optimizes them through the memoryless SR1 matrix, so that the global model can better adapt to diverse data sets and improve the generalization ability of the model.

[0076] 2. Technological advancement

[0077] (1) Significant improvement in model adaptability and performance

[0078] Through the optimization algorithm of the present invention, the client can perform deep training in combination with the local data set, generate local dual parameters and apply the memoryless SR1 matrix, thereby more effectively capturing the characteristic patterns of local data. This method effectively improves the performance of the model on each client, while enhancing the adaptability of the global model, making it suitable for diverse data environments, and improving the overall performance of federated learning in practical applications.

[0079] (2) Communication costs are significantly reduced

[0080] By using a lightweight global model (such as CNN or RNN) combined with the SR1 matrix in the quasi-Newton method, the client can perform most of the optimization calculations locally, thereby reducing the communication requirements when the model is updated. Compared with the traditional federated learning method, the present invention greatly reduces the communication frequency between the client and the server, is particularly suitable for bandwidth-constrained network environments, and has great advantages in large-scale applications.

[0081] (3) Improve computing efficiency and reduce resource consumption

[0082] By introducing the proximal quasi-Newton method, the memoryless SR1 matrix can effectively reduce resource consumption in the calculation process and avoid tedious second-order optimization calculations, especially when frequent model updates are required, the calculation efficiency is greatly improved. This makes the present invention suitable for resource-constrained computing environments such as mobile devices and the Internet of Things, ensuring the operability of federated learning.

[0083] (4) Optimization flexibility of the global model

[0084] Since the optimization algorithm adopted in the present invention has strong flexibility, it allows the global model to be dynamically adjusted according to the local optimization of the client, which means that the global model can be flexibly adapted between different clients, thereby improving the overall efficiency and flexibility of federated learning, especially for data sets with large heterogeneity.

[0085] (5) Reduce server burden

[0086] In traditional federated learning, the server often needs to centrally process a large number of model updates and parameter calculations, which is a heavy burden. However, the present invention assigns most of the optimization tasks to the client for execution, reduces the workload of the server through local optimization of the memoryless SR1 matrix, and improves the scalability and processing power of the system.

[0087] (6) Strengthen security and privacy protection

[0088] The present invention reduces the frequency of communication and performs most of the model optimization and parameter calculation on the client, thereby reducing the frequency of sensitive data transmission on the network and enhancing the privacy protection effect of federated learning. At the same time, the parameters generated and optimized locally are only related to local data, effectively avoiding the risk of data leakage.

[0089] In summary, the present invention solves many bottleneck problems in current technology through a composite optimization algorithm based on the proximal quasi-Newton method in a federated learning environment, and brings significant technological progress, including improving model adaptability, reducing communication and computing costs, improving the flexibility of the global model and the scalability of the system, and strengthening data privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 This is a schematic diagram of the FedQuasiNewton algorithm flow proposed by the present invention;

[0091] Figure 2 This is a schematic diagram of the performance experimental results of the sparse linear regression model in Example 2;

[0092] Figure 3 Schematic diagram of the training loss, test loss, training accuracy and test accuracy results under the MNIST data set in Example 4. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0094] The following are two industrial application examples of the federated composite optimization algorithm based on the present invention:

[0095] Example 1: Federated Learning Model Optimization for Medical Data

[0096] In the medical industry, hospitals or clinics usually have sensitive patient data that cannot be shared or centrally processed, so federated learning becomes an effective solution. The federated composite optimization algorithm of the present invention can be deployed in multiple medical institutions. Through federated learning, each hospital trains the model locally and combines the proximal quasi-Newton method and memoryless SR1 matrix of the present invention for optimization to form a local model that adapts to the characteristics of each data.

[0097] Application scenarios:

[0098] Pathological image classification model: Each hospital has a different amount of pathological image data. Using CNN as the global model, the local data sets of each hospital are significantly different. Through the algorithm of the present invention, each hospital can perform in-depth training of the local model based on its own pathological image data, optimize the local dual parameters and synchronize with the global model, and achieve efficient model optimization under different data environments.

[0099] Data privacy and security: Due to the privacy of medical data, hospitals cannot directly share data. This invention ensures data privacy by reducing communication frequency, local optimization and parameter updates, and realizes efficient calculation through memoryless SR1 matrix, reducing the calculation and communication burden of each hospital.

[0100] Industrial benefits:

[0101] Hospitals can improve the predictive performance of local models and accurately diagnose patients' pathological conditions. While ensuring data privacy, hospitals jointly train a more powerful global model to improve the breadth and accuracy of diagnosis.

[0102] Example 2: Predictive maintenance in smart manufacturing

[0103] In smart manufacturing, data collection of factory production equipment is usually decentralized, and each factory has different types of equipment and data, which makes it difficult to centrally train models. Federated learning can achieve collaborative model training among different factories through the composite optimization algorithm of the present invention.

[0104] Application scenarios:

[0105] Equipment failure prediction model: Each factory has different equipment types, usage environments, and data characteristics. Through federated learning, the central server provides a lightweight initial prediction model (such as a time series model based on RNN). The algorithm of the present invention enables each factory to perform local optimization based on the sensor data of its own equipment, generate local dual parameters, achieve fast optimization through the memoryless SR1 matrix, and synchronize the global model with the central server.

[0106] Dynamically adjust equipment parameters: The optimization algorithm of the present invention can adjust model parameters according to the local training data of each factory, improve the adaptability of the prediction model, reduce equipment downtime, reduce maintenance costs, and improve equipment utilization.

[0107] Industrial benefits:

[0108] Each factory can improve the prediction accuracy of equipment failures through local training and reduce the risk of unexpected equipment downtime. At the same time, the global model improved by federated learning can help all factories optimize their equipment maintenance plans, further reduce production costs and improve production efficiency.

[0109] These two embodiments demonstrate the effective application of the present invention in scenarios with high data sensitivity and strong device heterogeneity. Utilizing the algorithm of the present invention can significantly improve system performance, save costs and protect data privacy.

[0110] Client, Server: Clients refer to independent entities participating in the model training process, and they usually have their own private data sets. The client is responsible for performing computing tasks locally, such as training the model with its data and generating updates, which are then sent back to the server. The client maintains the locality of the data during this process, thereby protecting data privacy and security; the server is the central coordination node, responsible for initializing the global model and distributing it to each client, while collecting and aggregating model updates or gradients uploaded by the client to generate and optimize the global model. The server also manages the communication process to ensure the synchronization and efficiency of the training process, and balances the heterogeneity of the participants through algorithm design and resource scheduling.

[0111] Federated learning: As an advanced machine learning technology, it allows multiple parties to jointly build and optimize a global model without sharing the original data. The core advantage of this approach is data privacy protection, because the data does not have to leave its original storage location, thereby reducing the risk of data leakage or abuse. Federated learning enhances the generalization ability of the model by integrating diverse data from different sources, helps solve the problem of data silos, and promotes collaboration between different organizations.

[0112] Alternating Direction Method of Multipliers (ADMM): is an efficient optimization algorithm for solving large-scale problems with multiple block variables and coupled constraints. It gradually approaches the optimal solution by decomposing the problem, introducing multipliers, alternately solving subproblems and updating the multipliers. The advantages of ADMM lie in its flexibility, parallel processing capability and global convergence, making it an important tool for solving complex optimization problems in fields such as machine learning and signal processing.

[0113] SR1 matrix: is a preprocessing technique used in finite element analysis that improves the efficiency of solving a system of linear equations by constructing an approximate stiffness matrix. This matrix consists of a subset of the rows and columns of the original stiffness matrix, usually containing only elements directly connected to each degree of freedom. The SR1 method approximates the global stiffness matrix by iteratively updating these sub-matrices, thereby reducing computational and storage requirements.

[0114] Hessian matrix: The Hessian matrix is ​​a matrix of second-order partial derivatives of a multivariate function, which is used to describe the local curvature or convexity of the function. For a function f(x) with n variables, its Hessian matrix is ​​an n×n symmetric matrix whose elements are the second-order partial derivatives of the function. In optimization problems, the positive or negative definiteness of the Hessian matrix can provide information about the local minimum or maximum of the function. The Hessian matrix plays a core role in second-order optimization algorithms such as Newton's method, and is used to guide the search direction and step size to accelerate convergence to the optimal solution.

[0115] Proximal Quasi-Newton Method (PQNM): is an advanced optimization algorithm that combines the convergence speed of the quasi-Newton method with the ability of the proximal algorithm to handle non-smoothness. The method iteratively updates the inverse of the approximate Hessian matrix while applying proximal operations to handle non-smooth constraints such as sparsity. In each iteration, the algorithm updates the approximation matrix using the quasi-Newton formula and combines the proximal step to ensure that the solution satisfies specific constraints. This method performs well in solving optimization problems with complex constraints, especially in the fields of machine learning, signal processing, and image restoration, and is favored for its effectiveness and robustness.

[0116] Variable Scale Proximal Mapping (VMPM): is an optimization algorithm that combines gradient descent with proximal operations. This method searches for the optimal solution by adjusting the step size and introducing non-smooth constraints. In each iteration, VMPM uses gradient information to determine the search direction and adjusts the solution based on the proximal operation to meet specific constraints, such as sparsity or non-negativity. This method is particularly suitable for solving optimization problems with complex constraints and can balance the convergence speed and the quality of the solution.

[0117] The present invention discloses a federated composite optimization algorithm based on the proximal quasi-Newton method, the FedQuasiNewton algorithm, which is a second-order optimization algorithm for solving composite optimization problems under the framework of federated learning. In order to facilitate the solution, the algorithm divides the entire federated learning model update problem into two parts, namely, internal problems and external problems:

[0118] Internal problem: refers to a sub-problem that needs to be solved in each iteration of the algorithm, the purpose of which is to determine a search direction. Specifically, the internal problem is solved by the ADMM (Alternating Direction Method of Multipliers) algorithm, which involves each participating node calculating the update of its local parameters, and only some clients participate in the update in each iteration.

[0119] External problem: After solving the internal problem and obtaining the search direction, the external problem involves how to use this direction to update the global model. Specifically, the external problem refers to solving the variable-scale proximal mapping problem, that is, based on a given search direction, finding a new model that can minimize the sum of the loss function and the regularization term of the original problem. Using the memoryless SR1 formula to approximate the Hessian matrix can simplify the calculation of the variable-scale proximal mapping and reduce the difficulty of calculation.

[0120] The internal problem focuses on how to determine a suitable search direction, while the external problem focuses on how to use this direction to update the global model. Both jointly promote the iterative process of the algorithm.

[0121] The algorithm is based on the proximal quasi-Newton algorithm in traditional machine learning, and uses a single ADMM algorithm to solve the iteration direction, avoiding the increase in communication costs and privacy leakage caused by the proximal quasi-Newton method transferring the Hessian matrix. At the same time, considering that the traditional ADMM algorithm requires all nodes to participate in the update, it does not conform to the actual application scenario of federated learning. Therefore, the present invention improves the ADMM algorithm to adapt it to the situation of partial node updates. In terms of variable-scale proximal mapping calculation, considering its high computational difficulty, the present invention uses the memoryless SR1 formula instead of the node Hessian matrix, simplifies the variable-scale proximal mapping calculation, and further reduces the risk of privacy leakage. Compared with the existing federated learning algorithms, the algorithm proposed in the present invention solves the problem that the second-order algorithm cannot solve the composite optimization target under the federated learning framework, makes up for the vacancy of the second-order algorithm in federated composite optimization learning, and expands the application of the second-order algorithm in federated learning.

[0122] According to the attached Figure 1 ,The federated composite optimization algorithm based on the proximal quasi-Newton method mainly includes the following steps:

[0123] S1: Each client receives the original model sent by the server and completes the model initialization task.

[0124] The present invention is mainly implemented on the client side in a federated learning environment. The server side is the central server in federated learning. The initial global learning model usually uses a lightweight model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN), to ensure the flexibility and scalability of the model and adapt to the diverse local data sets with different data volumes owned by different participants.

[0125] The S1 step is the initialization phase in the federated learning process, in which the client first receives the initial global model from the server. This model is designed to be easily adapted to the diverse data sets held by different clients, and considering communication efficiency and computational costs, it is usually selected such as convolutional neural network (CNN) or recurrent neural network (RNN). Subsequently, the client generates a series of initialization parameters based on the received global model, including local primitive parameters, local dual parameters, and key initial parameters for the subsequent generation of memoryless SR1 matrices. The generation of these parameters is an important preparation before local training, ensuring that each client can start its training task at a unified starting point. In addition, the client also needs to set up a local training environment to facilitate model training and parameter adjustment using its unique data.

[0126] Specifically, the parameters generated in this step are mainly divided into two parts: (1) Parameters generated by the server: global original parameter node d0, global model (2) Initialization parameters generated by the client: local original parameters Local dual parameters Generate initial parameters for memoryless SR1 matrix With η.

[0127] S2: The client performs local training on the received initialization model based on the local data set to obtain a local training model, calculates the basic parameters of the local training, and generates a memoryless SR1 matrix.

[0128] Step S2 is a key part of the federated learning process, where the client performs in-depth training on the initialized model received from the server based on its local unique dataset. In this process, the client first initiates a local training task, and then applies its local dataset to carefully adjust and optimize the model. Through this training, the client is able to calculate basic parameters that reflect the characteristics of the local data, which are critical to capturing specific patterns and trends in the dataset.

[0129] After the client trains the model based on local data, it will get the local gradient of this round of updates Then according to the local gradient obtained from the previous round of training Global Modelx k and x k-1 , the memoryless SR1 matrix can be generated. The specific generation algorithm is as follows:

[0130]

[0131] In the algorithm, ρ and Basic parameters set for the approximate matrix generation algorithm, The calculation intermediate variables in the process of generating memoryless SR1 matrix for the client, This is the generated memoryless SR1 matrix.

[0132] The generated memoryless SR1 matrix is ​​used to approximate the Hessian matrix, which has many advantages. First, it paves the way for solving external problems later: the variable-scale proximal calculation is difficult, and the generated memoryless SR1 matrix can make the proximal mapping defined and simple to calculate, degenerating the variable-scale proximal mapping calculation into a common proximal mapping calculation, which reduces the difficulty of calculation; second, it reduces the transmission cost of communication: each client originally needs to upload the Hessian matrix to the central node, and the transmission cost is O(p 2), the cost is high. Using an approximate matrix instead of SR1 instead of the Hessian matrix can effectively reduce the communication cost. Third, more effective privacy protection: The Hessian matrix is ​​based on the second-order model gradient of local data, which contains a lot of effective information of local data. Once it is attacked by gradient leakage, the local data set may be stolen. Using its approximate matrix SR1 to replace it can effectively prevent such attacks.

[0133] S3: The server randomly selects some clients and uploads their local model training parameters.

[0134] For solving the algorithm iteration direction of internal problems, the ADMM algorithm is usually used for solution, but the traditional ADMM algorithm requires multiple rounds of communication between the client and the service segment to achieve algorithm convergence, which means an increase in communication costs. In the actual application of federated learning, there is usually a problem of system heterogeneity, and it cannot be guaranteed that all clients can upload local models to participate in the update in each round of iteration. Therefore, in order to alleviate the communication pressure and speed up the model iteration speed, the present invention considers introducing a method of randomly extracting client updates on the basis of a single ADMM algorithm to ensure that only some nodes participate in the update in each round of iteration.

[0135] After the server randomly selects some clients that have completed local training, the selected clients need to upload the parameters obtained in this round of training. and in, This is obtained when the memoryless SR1 matrix is ​​generated in step S2.

[0136] parameter The calculation process is as follows:

[0137]

[0138] In the formula, for each selected node i, its original parameter And the dual parameter The update formula is as follows:

[0139]

[0140] Wherein, L represents the augmented Lagrangian function, d represents the original parameter, is an approximation of the Hessian matrix, Represents the local gradient of the node update λ is the set of dual variables, ρ is the penalty parameter set and is usually greater than 0.

[0141] S4: Update ADMM direction and internal issues.

[0142] Step S4 is responsible for updating the ADMM direction in the FedQuasiNewton algorithm, where the server receives and processes the local model training parameters uploaded by the client, adjusts the search direction by optimizing the augmented Lagrangian function, solves internal problems, and prepares for the next iterative update of the global model.

[0143] In this step, the original parameter d k+1 The update method is as follows:

[0144]

[0145] In this step, the server is required to receive two parameters from the client. and Then the ADMM direction is updated. However, after simplifying the formula, it can be seen that only the client needs to calculate the intermediate calculation variables locally. And upload, the server can complete the update of ADMM direction. Through the improvement of this step, the computing pressure of the server can be reduced, and the computing task can be distributed to the selected client. At the same time, the number of parameters transmitted is reduced from two to one, which not only reduces the communication pressure, but also reduces the risk of client privacy leakage.

[0146] S5: Solve the variable-scale proximal mapping problem and obtain the proximal direction.

[0147] Solve the internal problem to get d k+1 After that, there is At this point, we only need to ask Then the proximal direction p can be calculated k .

[0148] Now consider how to calculate the variable scale proximal mapping In fact, the computational difficulty of the variable-scale proximal mapping is usually no less than that of the original problem. k When is a positive definite matrix, the above problem has a unique solution. Usually, we take However, this means that each node needs to upload the Hessian matrix to the central node, and the transmission cost is O(p 2 ), the cost is high. At the same time, since the proximal calculation of the variable scale is difficult and the approximate matrix is ​​required to be a positive definite matrix, it is necessary to select a suitable approximate matrix so that the proximal mapping is defined and the calculation is simple. However, since the matrix generated in step S2 of the present invention is a memoryless SR1 matrix, its variable scale proximal mapping can be degenerated into a common proximal mapping, which just meets the requirements.

[0149] The global approximate Hessian matrix of each round of update is S k For all the selected clients. It can be seen that in each iteration, there is no need for nodes to upload the entire Hessian matrix to the central end, only the clients need to Upload it to the central end, and the variable scale near-end mapping can be completed.

[0150] From the proximal direction p k The update formula is:

[0151]

[0152] where x k is the global model parameter of the previous round, d k+1 Calculated in the previous step, prox is the proximal operator.

[0153] At this point, the proximal direction p of the global model update can be obtained k .

[0154] S6: Update the global model according to the obtained proximal direction, and repeat the above steps until the model training is completed.

[0155] Step S6 is the final model update stage in the FedQuasiNewton algorithm. It uses the proximal direction solved in step S5 to iteratively update the global model through a specific update formula until the conditions for completing model training are met, such as reaching a predetermined number of iterations or performance indicators, thereby completing the entire federated learning training process.

[0156] The update formula of the global model is:

[0157] x k+1 =x k +tp k

[0158] The step size t is the sequence t∈{1,β,β 2 ,…} the first number that satisfies the following relationship

[0159]

[0160] Specifically, considering a federated learning framework consisting of a central server and m nodes, we have:

[0161]

[0162] in is the local loss function of the ith node, is a convex regularization function.

[0163] 1. Specific application fields or related products of the present invention.

[0164] The embodiment of the present invention provides an intelligent model training method system for distributed device collaborative optimization based on the intelligent model training method for distributed device collaborative optimization as described in claims 1-6, characterized in that the system specifically includes:

[0165] Model initialization module: the client receives the original model sent by the server and completes the model initialization task;

[0166] The local training module is connected to the model initialization module. The client performs local training on the received initialization model based on the local data set to obtain the local training model, calculates the basic parameters of the local training, and generates a memoryless SR1 matrix.

[0167] The parameter training module is connected to the local training module. The server randomly selects some clients, and the selected clients upload their local model training parameters.

[0168] The update module is connected with the parameter training module to update the ADMM direction and internal issues;

[0169] The solving module is connected with the updating module to solve the variable-scale proximal mapping problem, obtain the proximal direction, and solve the external problem;

[0170] The model updating module is connected to the solving module and updates the global model according to the obtained proximal direction until the model training is completed.

[0171] An embodiment of the present invention provides a computer device, characterized in that the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the intelligent model training method for distributed device collaborative optimization.

[0172] An embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the intelligent model training method for distributed device collaborative optimization.

[0173] An embodiment of the present invention provides an information data processing terminal, which is used to implement the intelligent model training method system for distributed device collaborative optimization.

[0174] 2. Relevant evidence of the technical effects obtained by the embodiments of the present invention. Specific embodiments

[0176] Example 1: Federated Regularized Logistic Regression Problem

[0177] In this embodiment, the logistic regression composite optimization problem of the smooth regularization term is considered, and the loss function is:

[0178]

[0179] in is the data of the ith node. The regular term is Regularization term

[0180] The datasets used are three basic datasets in LibSVM: a1a, w7a, and phishing, which are binary classification datasets. Their basic information is shown in Table 1:

[0181] Table 1: Basic description of the LibSVM dataset

[0182]

[0183] In order to impose statistical heterogeneity on the data between nodes, two methods of data label heterogeneity are used: quantity-based label heterogeneity and distribution-based label heterogeneity.

[0184] (1) Label heterogeneity based on quantity: Suppose there are n data distributed in m nodes. First, sort the data according to the label and divide the data into m·α parts, each containing Each node is assigned α pieces of data. As you can see, α controls the degree of label skew. The larger α is, the greater the label skew is. In this skew mode, each node only has label data of some classes.

[0185] (2) Distribution-based label heterogeneity: According to the Dirichlet distribution, each node is assigned a certain proportion of samples of each label. Specifically, by simulating p k ~Dir(β), and divide the samples in the kth class into proportion p k，j Assigned to node j. It can be seen that β controls the degree of label tilt. For this tilted allocation method, each node has more label categories, but the number varies greatly, which is more in line with reality.

[0186] Selection of comparison method: Considering that the FedQuasiNewton algorithm proposed in the invention adopts a similar processing method as the FedNew algorithm to solve the Newton iteration direction when solving internal problems, but the FedQuasiNewton algorithm only requires some nodes to participate in the update. Therefore, this embodiment selects FedNew as the comparison method to compare the performance difference between the FedQuasiNewton algorithm and the FedNew algorithm with full node participation under the participation of some nodes. In addition, the classic first-order algorithms such as FedAvg and FedProx with partial node participation are selected to verify the prediction performance of the algorithm.

[0187] Parameter selection: For the setting of data heterogeneity, when using quantity-based label tilt, set α = 2; when using distribution-based label tilt, set β = 0.5. For the regularization term coefficient, take λ = 10-4. All experiments were iterated for 1500 rounds.

[0188] The hyperparameters of the algorithm FedQuasiNewton of the present invention are respectively ρ∈{0.01,0.1,1,10}, η∈{0.5,0.6,0.7,0.8,0.9}. In fact, the hyperparameters finally selected in this embodiment are ρ=0.1, η=0.9.

[0189] FedNew sets hyperparameters ρ=0.1, α=0.25; FedAvg and FedProx both take a learning rate of 0.01, perform 10 epochs per round, and the batch size of each update is 20. The penalty coefficient μ in FedProx is selected from the candidate set {0.001, 0.01, 0.1, 1}. This embodiment finally selects μ=1.

[0190] Experimental results: The solution x obtained by using the standard Newton method for 30 iterations is ★ As the optimal solution to the federated composite optimization problem, record the difference between the algorithm function value and the optimal solution value after the iteration. K )-F(x ★ ) and the total number of communication bytes of the algorithm. The experimental results are shown in the following table:

[0191] Table 2: The difference between the function value of each algorithm and the optimal value and the total communication bytes under a1a data

[0192]

[0193] Table 3: The difference between the function value of each algorithm and the optimal value and the total communication bytes under phishing data

[0194]

[0195] Table 4: The difference between the function value of each algorithm and the optimal value and the total communication bytes under the data

[0196]

[0197] Experimental results show that on the a1a and w7a datasets, the FedQuasiNewton algorithm shows excellent performance with the smallest gap to the problem minimum under IID and quantity-based label heterogeneity, although its performance is similar to FedNew and slightly inferior to FedProx under distribution-based label heterogeneity. In the phishing dataset, FedQuasiNewton performs best under distribution-based label heterogeneity, while it is also better than the first-order algorithm in other cases and performs similarly to FedNew under quantity-based label distribution. Although the communication cost of the FedQuasiNewton algorithm is twice that of the first-order algorithm, in practical applications, it can achieve the same accuracy as FedNew with fewer nodes involved and less communication cost, showing its practicality and superiority in the federated learning environment, especially when only some nodes participate in the update.

[0198] Example 2: Federated Sparse Linear Regression

[0199] Consider sparse linear regression, that is, the optimization objective is the federated Lasso problem:

[0200]

[0201] Where λ is the regularization parameter. The linear model of each node is as follows:

[0202]

[0203] in and independent of the covariates Assume ω ★ is s-sparse, that is, ||ω ★ ||1=s,s<<p. At the kth node, first generate a heterogeneous vector δ k ~N(0,I p×p ), covariate i=1,2,…,n k .in Independently drawn from N(0,Σ), the (i,j)th element of the covariance matrix Σ is σ i，j =0.5 i-j| ,1≤i,j≤p.

[0204] Assume p = 1024, s = 512, a total of 64 nodes. Without loss of generality, assume that the amount of data in each node is the same, and both contain 128 data. In each round of iteration, 10 nodes are randomly selected to participate in the update. The regularization parameter λ = 0.5 5 To evaluate the performance of different methods, the estimated coefficients are recorded Error, that is

[0205] Selection of comparison method: Since the second-order algorithm in federated learning can only solve the problem of smooth objective function, it cannot be applied to the setting in this experiment. FedDR is currently a first-order composite optimization algorithm with better performance in federated learning. It has a good performance in the recovery of sparse coefficients, so FedDR is selected as the comparison method of this experiment.

[0206] Hyperparameter selection: The hyperparameters of FedDR are η=150, α=1.95, 10 epochs are performed per round, the learning rate is 0.001, and the batch size of each update is 10. For the algorithm FedQuasiNewton proposed in this invention, the parameter ρ of ADMM is taken as 0.01. When generating the memoryless SR1 matrix, η=0.9.

[0207] Experimental results: as attached Figure 2 As shown, in dealing with sparse linear problems, the algorithm FedQuasiNewton can approach the true sparse coefficient ω in fewer rounds. ★ Compared with the federated first-order algorithm FedDR, FedQuasiNewton has higher accuracy, less curve vibration, more stable descent direction, and faster convergence.

[0208] Example 3: MNIST multi-classification problem

[0209] Data processing: In this experiment, multivariate logistic regression is used to study the convex classification problem of the MNIST dataset. MNIST consists of handwritten digits from 0 to 9, with a total of 70,000 images, including 60,000 training sets and 10,000 test sets, all of which are 28×28 grayscale images. In order to impose statistical heterogeneity, the data is distributed in 1,000 nodes with reference to the processing method of FedProx. Each node has only two types of data, and the number of samples in each node obeys a power law. The input of the model is the flattened 784-dimensional image data, and the output is the classification label of 0-9.

[0210] Training model selection: A single-layer neural network with a size of 784×10 is used in the experiment. In each round of iteration, 250 nodes are randomly selected from 1000 nodes to participate in the update, and a total of 200 rounds of iterations are performed.

[0211] Selection of comparison method: In federated learning, there is no second-order algorithm for compound optimization problems. This paper selects the first-order compound optimization algorithm FedDR as the comparison method. FedDR is an algorithm with excellent performance in solving compound optimization problems in current federated learning, and its effect is better than a number of commonly used federated learning algorithms. Therefore, it is more appropriate to select FedDR as the comparison method.

[0212] Hyperparameter selection: For compound optimization problems, consider the regularization term as Take g(x) = 0.001||x||1. For FedDR, its hyperparameters are set as: η = 500, α = 1.95. For the algorithm of the present invention, take the parameter ρ in ADMM = 5, and when generating the memoryless SR1 matrix, take η=0.99.

[0213] Experimental results: The results are as attached Figure 3 As shown. In the MNIST multi-classification problem, although the data distribution between nodes is Non-IID, the algorithm proposed in the present invention is convergent, so the algorithm proposed in the present invention can solve the statistical heterogeneity of federated learning. At the same time, compared with FedDR, the method proposed in the present invention is higher than FedDR in convergence speed and test classification accuracy, and only 50 rounds are required to achieve 80% classification accuracy, which greatly reduces the number of communication rounds and reduces the communication cost.

[0214] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0215] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. An intelligent model training method for distributed device collaborative optimization, characterized in that: The algorithm includes: S1: The client receives the original model sent by the server and completes the model initialization task; S2: The client performs local training on the received initialization model based on the local data set to obtain a local training model, calculates the basic parameters of the local training, and generates a memoryless SR1 matrix; S3: The server randomly selects some clients, and the selected clients upload their local model training parameters; S4: Update ADMM direction and update internal issues; S5: Solve the variable-scale proximal mapping problem, obtain the proximal direction, and solve the external problem; S6: Update the global model according to the obtained proximal direction until the model training is completed.

2. The intelligent model training method for distributed device collaborative optimization according to claim 1, characterized in that: The step S1 is the initialization phase in the federated learning process, in which the client first receives the initial global model from the server, which is designed to be easily adapted to the diverse data sets held by different clients, and considering the communication efficiency and computational cost, a convolutional neural network (CNN) or a recurrent neural network (RNN) is usually selected; then, the client generates a series of initialization parameters based on the received global model, including local primitive parameters, local dual parameters, and key initial parameters for subsequent generation of memoryless SR1 matrices; the generation of these parameters is an important preparatory work before local training, ensuring that each client can start its training task at a unified starting point; In addition, the client needs to set up a local training environment to use its unique data to train the model and adjust the parameters; The parameters generated by S1 are mainly divided into two parts: (1) parameters generated by the server: global original parameter node d0, global model (2) Initialization parameters generated by the client: local original parameters Local dual parameters Generate initial parameters for memoryless SR1 matrix With η.

3. The intelligent model training method for distributed device collaborative optimization according to claim 1 is characterized in that: S2 is a key part of the federated learning process, in which the client conducts in-depth training on the initialized model received from the server based on its local unique dataset. In this process, the client first initiates the local training task, and then applies its local dataset to carefully adjust and optimize the model. Through this training, the client is able to calculate basic parameters that reflect the characteristics of the local data, which are crucial for capturing specific patterns and trends in the dataset. After the client trains the model based on local data, it will get the local gradient of this round of updates Then according to the local gradient obtained from the previous round of training Global Modelx k and x k-1 , the memoryless SR1 matrix can be generated. The specific generation algorithm is as follows: Input: Iteration round k, node i∈S k , global model x k ,x k -1, local gradient difference if k = 0 then B k =I,u k =v k =0 else end Output: In the algorithm, ρ and Basic parameters set for the approximate matrix generation algorithm, The calculation intermediate variables in the process of generating memoryless SR1 matrix for the client, This is the generated memoryless SR1 matrix; The generated memoryless SR1 matrix is ​​used to approximate the Hessian matrix, which has many advantages. First, it paves the way for solving external problems later: the proximal calculation of variable scale is difficult, and the generated memoryless SR1 matrix can make the proximal mapping defined and simple to calculate, degenerating the proximal mapping calculation of variable scale into ordinary proximal mapping calculation, reducing the difficulty of calculation; Second, it reduces the transmission cost of communication: each client originally needs to upload the Hessian matrix to the central node, and the transmission cost is O(p 2 ), the cost is high, and using approximate matrices instead of SR1 instead of Hessian matrices can effectively reduce communication costs; Third, more effective privacy protection: The Hessian matrix is ​​based on the second-order model gradient of local data, which contains a lot of effective information of local data. Once it is attacked by gradient leakage, the local data set may be stolen. Using its approximate matrix SR1 to replace it can effectively prevent such attacks.

4. The intelligent model training method for distributed device collaborative optimization according to claim 1, characterized in that: The S4 is responsible for updating the ADMM direction in the FedQuasiNewton algorithm, where the server receives and processes the local model training parameters uploaded by the client, adjusts the search direction by optimizing the augmented Lagrangian function, solves internal problems, and prepares for the next iterative update of the global model; In this step, the original parameter d k+1 The update method is as follows: In this step, the server is required to receive two parameters from the client. and Then the ADMM direction is updated; however, after simplifying the formula, it can be seen that only the client needs to calculate the intermediate calculation variables locally And upload, the server can complete the update of ADMM direction. Through the improvement of this step, the computing pressure of the server can be reduced, and the computing task can be distributed to the selected client; at the same time, the number of parameters transmitted is reduced from two to one, which not only reduces the communication pressure, but also reduces the risk of client privacy leakage.

5. The intelligent model training method for distributed device collaborative optimization according to claim 1, characterized in that: The internal problem is solved to obtain d k+1 After that, there is At this point, we only need to ask Then the proximal direction p can be calculated k ; Now consider how to calculate the variable scale proximal mapping In fact, the computational difficulty of the variable-scale proximal mapping is usually no less than that of the original problem; for the above problem, when B k When is a positive definite matrix, the above problem has a unique solution. Usually, we take However, this means that each node needs to upload the Hessian matrix to the central node, and the transmission cost is O(p 2 ), the cost is high; at the same time, since the proximal calculation of the variable scale is difficult and the approximate matrix is ​​required to be a positive definite matrix, it is necessary to select a suitable approximate matrix so that the proximal mapping is defined and the calculation is simple; however, since the matrix generated in step S2 of the present invention is a memoryless SR1 matrix, its variable scale proximal mapping can be degenerated into a common proximal mapping, which just meets the requirements; The global approximate Hessian matrix of each round of update is S k For all the selected clients; It can be seen that in each iteration, there is no need for nodes to upload the entire Hessian matrix to the central end, only the clients need to Upload it to the central end, and the variable scale near-end mapping can be completed. From the proximal direction p k The update formula is: where x k is the global model parameter of the previous round, d k+1 Calculated in the previous step, prox is the proximal operator; At this point, the proximal direction p of the global model update can be obtained k .

6. The intelligent model training method for distributed device collaborative optimization according to claim 1, characterized in that: The S6 is the final model update stage in the FedQuasiNewton algorithm, which uses the proximal direction solved in step S5 to iteratively update the global model through a specific update formula until the conditions for model training completion are met, such as reaching a predetermined number of iterations or performance indicators, thereby completing the entire federated learning training process; The update formula of the global model is: x k+1 =x k +tp k The step size t is the sequence t∈{1,β,β 2 ,…} the first number that satisfies the following relationship Specifically, considering a federated learning framework consisting of a central server and m nodes, we have: in is the local loss function of the ith node, is a convex regularization function.

7. An intelligent model training method system for distributed device collaborative optimization based on the intelligent model training method for distributed device collaborative optimization as described in claims 1-6, characterized in that: The system specifically includes: Model initialization module: the client receives the original model sent by the server and completes the model initialization task; The local training module is connected to the model initialization module. The client performs local training on the received initialization model based on the local data set to obtain the local training model, calculates the basic parameters of the local training, and generates a memoryless SR1 matrix. The parameter training module is connected to the local training module. The server randomly selects some clients, and the selected clients upload their local model training parameters. The update module is connected with the parameter training module to update the ADMM direction and internal issues; The solving module is connected with the updating module to solve the variable-scale proximal mapping problem, obtain the proximal direction, and solve the external problem; The model updating module is connected to the solving module and updates the global model according to the obtained proximal direction until the model training is completed.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the intelligent model training method for distributed device collaborative optimization as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the intelligent model training method for distributed device collaborative optimization as described in any one of claims 1 to 6.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the intelligent model training method system for distributed device collaborative optimization as described in claim 9.

Images