A method for domain-wide large model and training system based on federated learning

Abstract

A method for a domain large model and training system based on federated learning creates a model building task and determines the participants and their data set information; each participant encrypts the gradient obtained by local training using a homomorphic encryption public key, and sends the encrypted gradient to an aggregation node in a cloud data center; in the cloud data center, the encrypted gradient is selected from a parameter queue in a pipeline parallel mode for aggregation, and the aggregated gradient is decrypted using a homomorphic encryption private key in a trusted execution environment; the decrypted gradient is used to update the parameters of the current large model to obtain multiple updated models; parameters of all updated models are aggregated, and the aggregated model is sent to each participant as a new current model to start the next round of training; the present invention uses federated learning technology to jointly train a large model using multi-party data while protecting data privacy, so as to improve the generalization ability and accuracy of the model, and is suitable for various scenarios such as cloud data centers and distributed computing environments.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a domain large model and training system method based on federated learning. Background Art

[0002] With the rapid development of artificial intelligence technology, the training and application of large-scale models has become a trend. However, traditional large-scale model training usually requires a large amount of centrally stored data, which not only increases the cost of data transmission and storage, but also may cause data privacy and security issues.

[0003] At present, although there are some model training methods based on federated learning, most of these methods focus on simple models. In addition, they are difficult to directly apply to large model training in complex fields due to existing problems in some fields, such as data storage is relatively scattered and difficult to aggregate, some participants are unable to communicate over the network, and the data of all parties continues to grow.

[0004] Therefore, how to design and implement a suitable and efficient method for large-scale domain model training system based on federated learning has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a domain large model and training system method based on federated learning. Through federated learning technology, under the premise of protecting data privacy, the large model is jointly trained using multi-party data to improve the generalization ability and accuracy of the model. It is suitable for various scenarios such as cloud data centers and distributed computing environments.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions: a domain big model based on federated learning, the domain big model based on federated learning includes communication services, training services, model management services, a registration center, a distributed computing engine, and a storage system; The communication services include gateway services, high-performance remote procedure call protocols, and hypertext transfer protocols; all requests from external systems will be delegated to the gateway service for request forwarding; The training service includes verification components, task scheduling components, metadata center management components, federated learning components, feature data intersection components, data loading and preprocessing components, and model evaluation components. The verification component is responsible for verifying the submitted configuration parameters. The task scheduling component is responsible for parsing the configuration parameters and scheduling the entire training task. According to the specified component running order, a training task is converted into a responsibility chain and submitted to the task thread pool for execution. The metadata center management component is responsible for recording the progress and running status of each party's training task, configuration parameters, and federated learning partners and roles. The federated learning component is used to complete various functions required in the model training process. The feature data intersection component relies on encryption technology and privacy computing services to allow all parties to align data and combine features without directly sharing the original data. The data loading and preprocessing component obtains data from each participant and integrates it into the system, cleans, converts, and normalizes the data to ensure the security, integrity, and consistency of the data and improve the quality of the data and the training effect of the model. The model evaluation component is used to measure the performance of the model, guide model optimization, support the improvement of the federated learning framework, promote cooperation and trust between participants, and ensure data security and privacy. Model Management Service: After completing the training task, the training service sends the trained model information to the Model Management Service, which then performs persistent storage and grouping operations. Registration center: In order to ensure high availability of services, the registration service of the distributed application coordination service software is introduced. When the service is started, the service information will be registered in the distributed application coordination service software. When a training request is initiated to the gateway service, the gateway service will pull available services from the distributed application coordination service software and complete the service call through the specified load balancing strategy; Distributed computing engine: By improving performance, protecting data privacy, supporting multi-party collaboration, optimizing resource utilization, improving computing efficiency, adapting to multiple scenarios, promoting real-time model updates, and reducing hardware requirements, it can efficiently process large-scale data sets and complex machine learning models while protecting user privacy, and realize cross-organizational collaborative model training and updates; Storage system: Save the model and process parameters. Through various distributed file storage systems, ensure that only model updates or gradients are shared, and the data is retained on the local device or system. At the same time, introducing multiple auxiliary nodes can reduce the impact of single point failures and maintain the stability of the overall learning framework.

[0007] Furthermore, the federated learning components include logistic regression, deep neural network, support vector machine, decision tree, and naive Bayes.

[0008] A method for training large domain models based on federated learning. S1 Model initialization: define the initial large model structure and parameters, create model building tasks, and determine the participants and their data set information; S2 encrypted gradient transmission: Each participant uses the homomorphic encryption public key to encrypt the gradient obtained from local training, and sends the encrypted gradient to the aggregation node in the cloud data center. The online node transmits it through the network, and the offline node transmits it through the data carrier; S3 gradient aggregation and decryption: In the cloud data center, encrypted gradients are selected from the parameter queue in a pipeline parallel mode for aggregation to obtain aggregated gradients. Subsequently, the aggregated gradients are decrypted using the homomorphic encryption private key in the trusted execution environment to obtain decrypted gradients. S4 model update: Use the decrypted gradient to update the parameters of the current large model to obtain an updated model. Repeat this process until the pipeline parallel mode ends and multiple updated models are obtained. S5 Model aggregation and distribution: Aggregate the parameters of all updated models to obtain the final aggregated model, and send the aggregated model to each participant as the new current model to start the next round of training; S6 Model convergence judgment: judge whether the model has converged according to the preset convergence conditions. If not, return to S2 to continue training; S7 incremental iteration: Based on the data growth rate, each participant regularly adds the newly acquired data to the training and returns to S6 to continue iteration.

[0009] The beneficial effects of the present invention are as follows: the present invention ensures that the original data is not leaked during the data transmission and aggregation process through homomorphic encryption technology, thereby protecting data privacy; adopts a pipeline parallel mode to process the gradient aggregation and decryption process, thereby reducing waiting time and improving training efficiency; utilizes multi-party data for joint training to enhance the generalization ability of the model and improve the application effect of the model in complex fields; utilizes storage medium conduction to form an offline joint channel, incorporates partners who cannot participate online into the training and use system, enhances the applicability of the model, and improves the application effect; uses the first trained model as a basis, regularly inputs newly generated and collected data into the model, and fine-tunes the model parameters to reduce training time and computing resource consumption, quickly adapt to new data and task changes, and maintain the performance and adaptability of the model.

[0010] The present invention uses federated learning technology to jointly train a large model using data from multiple parties while protecting data privacy, so as to improve the generalization ability and accuracy of the model. The present invention is suitable for various scenarios such as cloud data centers and distributed computing environments. The parts not described in detail in the present invention are existing commonly used technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention will be further described below in conjunction with the accompanying drawings: Figure 1Schematic diagram of the overall architecture; Figure 2 Schematic diagram of the model framework; Figure 3 Schematic diagram of the training process. DETAILED DESCRIPTION

[0012] The present invention is further described in detail below in conjunction with embodiments and specific implementation modes: Example 1

[0013] like Figure 2 As shown, a domain big model based on federated learning includes a communication service, a training service, a model management service, a registration center, a distributed computing engine, and a storage system; The communication services include gateway services, high-performance remote procedure call protocols, and hypertext transfer protocols; all requests from external systems will be delegated to the gateway service for request forwarding; The training service includes verification components, task scheduling components, metadata center management components, federated learning components, feature data intersection components, data loading and preprocessing components, and model evaluation components. The verification component is responsible for verifying the submitted configuration parameters. The task scheduling component is responsible for parsing the configuration parameters and scheduling the entire training task. According to the specified component running order, a training task is converted into a responsibility chain and submitted to the task thread pool for execution. The metadata center management component is responsible for recording the progress and running status of each party's training task, configuration parameters, and federated learning partners and roles. The federated learning component is used to complete various functions required in the model training process. The feature data intersection component relies on encryption technology and privacy computing services to allow all parties to align data and combine features without directly sharing the original data. The data loading and preprocessing component obtains data from each participant and integrates it into the system, cleans, converts, and normalizes the data to ensure the security, integrity, and consistency of the data and improve the quality of the data and the training effect of the model. The model evaluation component is used to measure the performance of the model, guide model optimization, support the improvement of the federated learning framework, promote cooperation and trust between participants, and ensure data security and privacy. Model Management Service: After completing the training task, the training service sends the trained model information to the Model Management Service, which then performs persistent storage and grouping operations. Registration center: In order to ensure high availability of services, the registration service of the distributed application coordination service software is introduced. When the service is started, the service information will be registered in the distributed application coordination service software. When a training request is initiated to the gateway service, the gateway service will pull available services from the distributed application coordination service software and complete the service call through the specified load balancing strategy; Distributed computing engine: By improving performance, protecting data privacy, supporting multi-party collaboration, optimizing resource utilization, improving computing efficiency, adapting to multiple scenarios, promoting real-time model updates, and reducing hardware requirements, it can efficiently process large-scale data sets and complex machine learning models while protecting user privacy, and realize cross-organizational collaborative model training and updates; Storage system: Save the model and process parameters. Through various distributed file storage systems, ensure that only model updates or gradients are shared, and the data is retained on the local device or system. At the same time, introducing multiple auxiliary nodes can reduce the impact of single point failures and maintain the stability of the overall learning framework.

[0014] The federated learning components include logistic regression, deep neural network, support vector machine, decision tree, and naive Bayes. Example 2

[0015] like Figure 1 , Figure 3 As shown, S1 creates a model building task: creates a model building task in a cloud data center and determines the initial large model structure and parameters; S2 Invite Participants: Send invitations to potential data holders to participate in model training; S3 collects data set information: collects data set information of each participant, including data volume and data type; S4 Encrypted Gradient Transmission: Each participant uses the homomorphic encryption public key to encrypt the gradient obtained by the local training end, and sends the encrypted gradient to the aggregation node of the cloud data center. The online node transmits it to the local training end through the network, and the offline node transmits it to the local training end through the data carrier; S5 gradient aggregation and decryption: The cloud data center processes encrypted gradients in a pipeline parallel mode to perform gradient aggregation and decryption operations; S6 Model update and aggregation: Use the decrypted gradient to update the large model parameters to obtain the updated model, aggregate the parameters of all updated models to obtain the final aggregated model; S7 Model distribution and convergence judgment: The cloud data center distributes the aggregated model to each participant as the new current model, and judges whether the model converges according to the preset convergence conditions. If not, it returns to S4 to continue training; S8 regular update: Each participant regularly adds newly collected data to the dataset, performs incremental training on the model, and returns to S4 for iterative update.

[0016] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A domain-wide model based on federated learning, characterized by: The domain big model based on federated learning includes communication services, training services, model management services, registration centers, distributed computing engines, and storage systems; The communication services include gateway services, high-performance remote procedure call protocols, and hypertext transfer protocols; all requests from external systems will be delegated to the gateway service for request forwarding; The training service includes verification components, task scheduling components, metadata center management components, federated learning components, feature data intersection components, data loading and preprocessing components, and model evaluation components; the verification component is responsible for verifying the submitted configuration parameters; The task scheduling component is responsible for parsing configuration parameters and scheduling the entire training task. It converts a training task into a responsibility chain according to the specified component running order and submits it to the task thread pool for execution. The metadata center management component is responsible for recording the progress and running status of each party's training tasks, configuration parameters, and federated learning partners and roles; the federated learning component is used to complete various functions required in the model training process; the feature data intersection component relies on encryption technology and privacy computing services to allow all parties to align data and combine features without directly sharing the original data; the data loading and preprocessing component obtains data from each participant and integrates it into the system, cleans, converts, and normalizes the data to ensure the security, integrity, and consistency of the data and improve the quality of the data and the training effect of the model; The model evaluation component is used to measure the performance of the model, guide model optimization, support the improvement of the federated learning framework, promote cooperation and trust among participants, and ensure data security and privacy; Model Management Service: After completing the training task, the training service sends the trained model information to the Model Management Service, which then performs persistent storage and grouping operations. Registration center: In order to ensure high availability of services, the registration service of the distributed application coordination service software is introduced. When the service is started, the service information will be registered in the distributed application coordination service software. When a training request is initiated to the gateway service, the gateway service will pull available services from the distributed application coordination service software and complete the service call through the specified load balancing strategy; Distributed computing engine: By improving performance, protecting data privacy, supporting multi-party collaboration, optimizing resource utilization, improving computing efficiency, adapting to multiple scenarios, promoting real-time model updates, and reducing hardware requirements, it can efficiently process large-scale data sets and complex machine learning models while protecting user privacy, and realize cross-organizational collaborative model training and updates; Storage system: Save the model and process parameters. Through various distributed file storage systems, ensure that only model updates or gradients are shared, and the data is retained on the local device or system. At the same time, introducing multiple auxiliary nodes can reduce the impact of single point failures and maintain the stability of the overall learning framework.

2. A domain big model based on federated learning according to claim 1, characterized in that: The federated learning components include logistic regression, deep neural network, support vector machine, decision tree, and naive Bayes.

3. A method for training a domain large model based on federated learning as described in claim 1, characterized in that: S1 Model initialization: define the initial large model structure and parameters, create model building tasks, and determine the participants and their data set information; S2 Encrypted Gradient Transmission: Each participant uses the homomorphic encryption public key to encrypt the gradient obtained from local training, and sends the encrypted gradient to the aggregation node in the cloud data center. The online node transmits it through the network, and the offline node transmits it through the data carrier. S3 gradient aggregation and decryption: In the cloud data center, encrypted gradients are selected from the parameter queue in a pipeline parallel mode for aggregation to obtain aggregated gradients. Subsequently, the aggregated gradients are decrypted using the homomorphic encryption private key in the trusted execution environment to obtain decrypted gradients. S4 model update: Use the decrypted gradient to update the parameters of the current large model to obtain an updated model. Repeat this process until the pipeline parallel mode ends and multiple updated models are obtained. S5 Model aggregation and distribution: Aggregate the parameters of all updated models to obtain the final aggregated model, and send the aggregated model to each participant as the new current model to start the next round of training; S6 Model convergence judgment: judge whether the model has converged according to the preset convergence conditions. If not, return to S2 to continue training; S7 incremental iteration: Based on the data growth rate, each participant regularly adds the newly acquired data to the training and returns to S6 to continue iteration.

Images