Federated meta-learning method, server, and electronic equipment for training with small amounts of data

Abstract

The present disclosure relates to a federated meta-learning method, a server, and an electronic device for training with a small amount of data, and belongs to the field of federated learning technology. The method includes: sending a target task to nodes participating in federated learning, so that each node generates at least one similar subtask j according to the target task, and dividing the local data set based on the subtask j to obtain a data set D i,j ; The model to be trained g t Sent to the nodes participating in federated learning so that each node can use the dataset D i,j Treat the training model g t After training, a model is generated based on all training results. The model sent by each node is received and the model to be trained is generated based on the model g t+1 ; Based on the model to be trained g t+1 If the requirements of the target task are not met, the next round of training is carried out; when the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 The present invention enables nodes to participate in federated learning training even with only a small amount of data.

Description

Technical Field

[0001] The present invention provides a federated meta-learning method, a server, and an electronic device for training with a small amount of data, relating to the technical field of federated learning. Background Art

[0002] Federated meta-learning is a hot research topic. Meta-learning, or learning to learn, allows us to leverage meta-learned models to generate a robust model for a new task with only a few training steps, eliminating the need for extensive training on a single dataset, as is the case with classic machine learning.

[0003] During federated meta-learning, a server maintains an algorithm (meta-learner) and distributes it to clients for model training. In each meta-learning episode, a sample of clients receives the algorithm parameters and performs model training. The test results on the query set are then uploaded to the server for algorithm updates.

[0004] At the same time, when performing federated learning tasks, nodes may not have or only have a small amount of suitable data, making federated learning difficult to carry out. Summary of the Invention

[0005] In order to overcome the above-mentioned shortcomings of the existing technology, the present invention provides a federated meta-learning method for training with a small amount of data suitable for epidemic investigation business, so as to solve the problem that federated learning is difficult to carry out due to the small amount of data.

[0006] The technical solution of the present invention includes:

[0007] A federated meta-learning method for training with a small amount of data, applied to a federated learning server, includes:

[0008] The target task is sent to the nodes participating in federated learning, so that each node generates at least one similar subtask j according to the target task, and divides the local data set based on the subtask j to obtain the data set D i,j ; Where i represents the node number;

[0009] The model to be trained g t Sent to the nodes participating in federated learning so that each node can use the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Where t represents the number of training rounds;

[0010] Receive the model sent by each node Based on the model Generate the model to be trained gt+1 ;

[0011] In the model to be trained g t+1 If the requirements of the target task are not met, set t=t+1 and return to the training model g. t Sent to nodes participating in federated learning;

[0012] In the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 .

[0013] Furthermore, each node uses the data set D i,j For the model to be trained g t After training, generate a model based on all training results include:

[0014] On subtask j, use the corresponding training data subset to train the model g t Train and get the model

[0015] Get the model In the dataset D of subtask j i,j The training gradient on

[0016] Based on the training gradient Update model g t , get the model

[0017] Furthermore, the generative model Gradient update in, represents the unit vector of the hyperparameter, Represents model g t Hessian matrix of the loss function, I represents the model g t The corresponding unit vector, f i (·) represents the loss function.

[0018] Furthermore, the model based on Generate the model to be trained g t+1 ,include:

[0019] For the model Perform standardization operations to obtain the model

[0020] The model is tested using the validation dataset Perform the test and obtain the model Performance

[0021] The performance Arrange in descending order and select the top k models

[0022] For the first k models Weighted aggregation to obtain the model to be trained g t+1 .

[0023] A federated meta-learning server for small-data training, comprising:

[0024] The task distribution module is used to send the target task to the nodes participating in the federated learning, so that each node generates at least one similar subtask j according to the target task, and divides the local data set based on the subtask j to obtain the data set D i,j ; Where i represents the node number;

[0025] Model distribution module, used to distribute the model to be trained g t Sent to the nodes participating in federated learning so that each node can use the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Where t represents the number of training rounds;

[0026] Model aggregation module, used to receive the model sent by each node Based on the model Generate the model to be trained g t+1 ;

[0027] The model validation module is used to verify the model g t+1 If the requirements of the target task are not met, set t=t+1 and return to the training model g. t Sent to the nodes participating in federated learning; in the model to be trained g t +1 When the requirements of the target task are met, the model to be trained g is output t+1 .

[0028] An electronic device, comprising:

[0029] processor;

[0030] a memory for storing instructions executable by the processor;

[0031] The processor is used to read the executable instructions from the memory and execute the instructions to implement any of the above-mentioned longitudinal federated learning methods for balancing the differences in survey data among various parties.

[0032] A computer-readable storage medium having computer program instructions stored thereon, characterized in that when the program instructions are executed by a processor, they implement any of the above-mentioned longitudinal federated learning methods for balancing the differences in survey data among various parties.

[0033] A federated meta-learning method for training with a small amount of data, applied to a federated learning node, comprising:

[0034] Receive the target task sent by the server, generate at least one similar subtask j according to the target task, and divide the local data set based on the subtask j to obtain the data set D i,j ; Where i represents the node number;

[0035] Receive the model to be trained g sent by the server t , and using the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Where t represents the number of training rounds;

[0036] The model Send to the server so that the server can use the model of each node Generate the model to be trained g t+1 , and judge the model to be trained g t+1 Whether the requirements of the target tasks are met:

[0037] In the model to be trained g t+1 If the requirements of the target task are not met, the model to be trained g t+1 After distributing to each node, set t = t + 1 and return to the training model g sent by the receiving server. t ;

[0038] In the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 .

[0039] A federated meta-learning node for small-data training, comprising:

[0040] The task division module is used to receive the target task sent by the server, generate at least one similar subtask j according to the target task, and divide the local data set based on the subtask j to obtain the data set D i,j; Where i represents the node number;

[0041] Model training module, used to receive the model to be trained g sent by the server t , and using the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Where t represents the number of training rounds;

[0042] Model upload module, used to upload the model Send to the server so that the server can use the model of each node Generate the model to be trained g t+1 , and judge the model to be trained g t+1 Whether the requirements of the target tasks are met:

[0043] In the model to be trained g t+1 If the requirements of the target task are not met, the model to be trained g t+1 After distributing to each node, set t = t + 1 and return to the training model g sent by the receiving server. t ;

[0044] In the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 .

[0045] A federated meta-learning system for training with a small amount of data, the system comprising: a server for federated learning and a node for federated learning;

[0046] The server is used to send the target task to the nodes participating in the federated learning; t Send to nodes participating in federated learning; receive the model sent by each node Based on the model Generate the model to be trained g t+1 , where i represents the node number and t represents the number of training rounds; in the model to be trained g t+1 If the requirements of the target task are not met, set t=t+1 and return to the training model g. t Sent to the nodes participating in federated learning; in the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 ;

[0047] The node is used to generate at least one similar subtask j according to the target task, and divide the local data set based on the subtask j to obtain a data set D i,j ; Using the data set D i,j For the model to be trained g t After training, generate a model based on all training results

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] This method applies meta-learning technology to federated learning, enabling nodes to participate in training even with only five or even one piece of data. Furthermore, this method can filter locally trained models based on task requirements, eliminating the impact of deviating task requirements or even malicious models on federated learning. This significantly improves the effectiveness of federated learning and strengthens the application of federated learning technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Flowchart of the operation of the federated meta-learning system for training with small amounts of data. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only specific embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] The present invention is applied to a federated learning system, which may include several nodes for federated learning and at least one server for federated learning.

[0053] In one embodiment of the present invention, for a server for federated learning, the federated meta-learning method of the present invention includes:

[0054] Step 110: Send the target task to the nodes participating in federated learning, so that each node generates at least one similar subtask j according to the target task, and divides the local data set based on the subtask j to obtain the data set D i,j ; where i represents the node number.

[0055] The server first sends the target task to the nodes participating in federated learning. After receiving the target task, each node generates several similar subtasks based on the target task and divides the local data set based on each subtask.

[0056] Step 120: The model to be trained gt Sent to the nodes participating in federated learning so that each node can use the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Here, t represents the number of training rounds.

[0057] In the initial stage, the server will obtain a model to be trained g 1 After that, in each round of training, the nodes participating in the federated learning receive the model to be trained g t Afterwards, the model is trained on subtask j and get the model In the dataset D of subtask j i,j The training gradient on To update the model g t Finally, when training locally on the node, the model is updated as follows:

[0058]

[0059] Where m represents the number of subtasks on node i, and λ is a hyperparameter used to adjust the gradient value of each subtask. It is the local model generated by node i through local training.

[0060] Step 130: Receive the model sent by each node Based on the model Generate the model to be trained g t+1 .

[0061] The nodes participating in federated learning get the model Afterwards, the model Send to the server. The server receives the model sent by each node After that, the parameters of the local model are standardized. The specific process is as follows:

[0062]

[0063] Then, the server uses the verification dataset to Check and obtain Performance And follow Arrange them in descending order and select the top k models for aggregation to generate a new global model g t+1 The specific weighting process is as follows:

[0064]

[0065] Among them, the validation dataset is used When testing, you need to use the validation data set to Perform a one-step training instead of direct validation to allow models that learn knowledge from similar tasks Able to adapt to the data of the target task, thus making inferences that meet the task requirements. One-step training is to fine-tune the model parameters with a small amount of data, which can be as little as 5 or even 1 data point.

[0066] From the above, it can be seen that the local training of the present invention is very different from conventional federated learning. Conventional federated learning directly modifies the model parameters according to the training data to generate a local model during local training, while the local training in this method first updates the temporary model according to the local data. Then the temporary model The optimizable content is updated to the model parameters to generate a new local model. In other words, the local training of conventional federated learning only involves one layer of model, while the local model of this method involves two layers of model.

[0067] In addition, the local training involves the second-order gradient of the loss function. Let the function f i (·) represents the loss function, then the updated gradient of node i for local training is actually:

[0068]

[0069] in A set of unit vectors of hyperparameters, Represents model g t Gradient update of Represents model g t Hessian matrix of the loss function, I represents the model g t Therefore, this method requires the loss function of the node to be second-order continuous.

[0070] Step 140: In the to-be-trained model g t+1 If the requirements of the target task are not met, set t=t+1 and return to the training model g. t Sent to the nodes participating in federated learning; in the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 .

[0071] The server uses the validation data set to train the model g obtained in each round t+1 To verify, if g t+1 If the effect can meet the task requirements, then the federated learning training is terminated; if it cannot meet the task requirements, then jump to step 120 to continue training.

[0072] In summary, this invention applies meta-learning technology to federated learning, enabling nodes to participate in training even with only five or even one piece of data. Furthermore, this method can filter locally trained models based on task requirements, eliminating the impact of deviating task requirements or even malicious models on federated learning. This significantly improves the effectiveness of federated learning and enhances the application capabilities of federated learning technology.

[0073] The present invention also provides a federated meta-learning server for small-scale data training, comprising:

[0074] The task distribution module is used to send the target task to the nodes participating in the federated learning, so that each node generates at least one similar subtask j according to the target task, and divides the local data set based on the subtask j to obtain the data set D i,j ; Where i represents the node number;

[0075] Model distribution module, used to distribute the model to be trained g t Sent to the nodes participating in federated learning so that each node can use the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Where t represents the number of training rounds;

[0076] Model aggregation module, used to receive the model sent by each node Based on the model Generate the model to be trained g t+1 ;

[0077] The model validation module is used to verify the model g t+1 If the requirements of the target task are not met, set t=t+1 and return to the training model g. t Sent to the nodes participating in federated learning; in the model to be trained g t +1 When the requirements of the target task are met, the model to be trained g is output t+1 .

[0078] In one embodiment of the present invention, for a node used for federated learning, the federated meta-learning method of the present invention includes:

[0079] Step 210: Receive the target task sent by the server, generate at least one similar subtask j according to the target task, and divide the local data set based on the subtask j to obtain the data set D i,j ; Where i represents the node number;

[0080] Step 220: Receive the training model g sent by the server t , and using the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Where t represents the number of training rounds;

[0081] Step 230: The model Send to the server so that the server can use the model of each node Generate the model to be trained g t+1 , and judge the model to be trained g t+1 Whether the requirements of the target tasks are met:

[0082] In the model to be trained g t+1 If the requirements of the target task are not met, the model to be trained g t+1 After distributing to each node, set t = t + 1 and return to step 220 to continue training;

[0083] In the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 .

[0084] The present invention also provides a federated meta-learning node for small-scale data training, the node comprising:

[0085] The task division module is used to receive the target task sent by the server, generate at least one similar subtask j according to the target task, and divide the local data set based on the subtask j to obtain the data set D i,j ; Where i represents the node number;

[0086] Model training module, used to receive the model to be trained g sent by the server t , and using the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Where t represents the number of training rounds;

[0087] Model upload module, used to upload the model Send to the server so that the server can use the model of each node Generate the model to be trained g t+1 , and judge the model to be trained g t+1 Whether the requirements of the target tasks are met:

[0088] In the model to be trained g t+1If the requirements of the target task are not met, the model to be trained g t+1 After distributing to each node, set t = t + 1 and return to the training model g sent by the receiving server. t ;

[0089] In the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 .

[0090] The present invention also provides a federated meta-learning system for training with a small amount of data, the system comprising: a server end applied to federated learning and a node applied to federated learning. Figure 1 The figure is a flowchart of the overall operation of the system.

[0091] The server is used to send the target task to the nodes participating in the federated learning; t Send to nodes participating in federated learning; receive the model sent by each node Based on the model Generate the model to be trained g t+1 , where i represents the node number and t represents the number of training rounds; in the model to be trained g t+1 If the requirements of the target task are not met, set t=t+1 and return to the training model g. t Sent to the nodes participating in federated learning; in the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 ;

[0092] The node is used to generate at least one similar subtask j according to the target task, and divide the local data set based on the subtask j to obtain a data set D i,j ; Using the data set D i,j For the model to be trained g t After training, generate a model based on all training results

[0093] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered merely as exemplary, and the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof.

Claims

1. A federated meta-learning method for training with a small amount of data suitable for epidemic investigation services, applied to the server side of federated learning, characterized by: The method comprises: The target task is sent to the nodes participating in federated learning, so that each node generates at least one similar subtask j according to the target task, and divides the local data set based on the subtask j to obtain the data set D i,j ; Where i represents the node number; The model to be trained g t Sent to the nodes participating in federated learning so that each node can use the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Wherein, t represents the number of training rounds, and each node uses the data set D i,j For the model to be trained g t After training, generate a model based on all training results include: On subtask j, use the corresponding training data subset to train the model g t Train and get the model Get the model In the dataset D of subtask j i ,j The training gradient on Based on the training gradient Update model g t , get the model ; Receive the model sent by each node Based on the model Generate the model to be trained g t+1 ; In the model to be trained g t+1 If the requirements of the target task are not met, set t=t+1 and return to the training model g. t Sent to nodes participating in federated learning; In the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 .

2. The method according to claim 1, wherein The generative model Gradient update in, represents the unit vector of the hyperparameter, Represents model g t Hessian matrix of the loss function, I represents the model g t The corresponding unit vector, f i (·) represents the loss function.

3. The method according to claim 1, wherein Based on the model Generate the model to be trained g t+1 ,include: For the model Perform standardization operations to obtain the model The model is tested using the validation dataset Perform the test and obtain the model Performance The performance Arrange in descending order and select the top k models For the first k models Weighted aggregation to obtain the model to be trained g t+1 .

4. A federated meta-learning server for small-scale data training suitable for epidemic investigation business, characterized by: The server includes: The task distribution module is used to send the target task to the nodes participating in the federated learning, so that each node generates at least one similar subtask j according to the target task, and divides the local data set based on the subtask j to obtain the data set D i,j ; Where i represents the node number; Model distribution module, used to distribute the model to be trained g t Sent to the nodes participating in federated learning so that each node can use the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Wherein, t represents the number of training rounds, and each node uses the data set D i,j For the model to be trained g t After training, generate a model based on all training results include: On subtask j, use the corresponding training data subset to train the model g t Train and get the model Get the model In the dataset D of subtask j i,j The training gradient on Based on the training gradient Update model g t , get the model Model aggregation module, used to receive the model sent by each node Based on the model Generate the model to be trained g t+1 ; The model validation module is used to verify the model g t+1 If the requirements of the target task are not met, set t=t+1 and return to the training model g. t Sent to the nodes participating in federated learning; in the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 .

5. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the federated meta-learning method for small-scale data training applicable to flow investigation business as described in any one of claims 1-3.

6. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, the federated meta-learning method for small-scale data training applicable to flow investigation business as described in any one of claims 1-3 is implemented.

7. A federated meta-learning method for training with a small amount of data suitable for epidemic investigation business, applied to federated learning nodes, characterized by: The method comprises: Receive the target task sent by the server, generate at least one similar subtask j according to the target task, and divide the local data set based on the subtask j to obtain the data set D i,j ; Where i represents the node number; Receive the model to be trained g sent by the server t , and using the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Wherein, t represents the number of training rounds, and the dataset D is used i,j For the model to be trained g t After training, generate a model based on all training results include: On subtask j, use the corresponding training data subset to train the model g t Train and get the model Get the model In the dataset D of subtask j i,j The training gradient on Based on the training gradient Update model g t , get the model The model Send to the server so that the server can use the model of each node Generate the model to be trained g t+1 , and judge the model to be trained g t+1 Whether the requirements of the target tasks are met: In the model to be trained g t+1 If the requirements of the target task are not met, the model to be trained g t+1 After distributing to each node, set t = t + 1 and return to the training model g sent by the receiving server. t ; In the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 .

8. A federated meta-learning node suitable for small-scale data training of epidemic investigation business, characterized by: The nodes include: The task division module is used to receive the target task sent by the server, generate at least one similar subtask j according to the target task, and divide the local data set based on the subtask j to obtain the data set D i,j ; Where i represents the node number; Model training module, used to receive the model to be trained g sent by the server t , and using the dataset D i,j For the model to be trained g t After training, generate a model based on all training results Wherein, t represents the number of training rounds, and the dataset D is used i,j For the model to be trained g t After training, generate a model based on all training results include: On subtask j, use the corresponding training data subset to train the model g t Train and get the model Get the model In the dataset D of subtask j i,j The training gradient on Based on the training gradient Update model g t , get the model Model upload module, used to upload the model Send to the server so that the server can use the model of each node Generate the model to be trained g t+1 , and judge the model to be trained g t+1 Whether the requirements of the target tasks are met: In the model to be trained g t+1 If the requirements of the target task are not met, the model to be trained g t+1 After distributing to each node, set t = t + 1 and return to the training model g sent by the receiving server. t ; In the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 .

9. A federated meta-learning system for small-scale data training suitable for epidemic investigation business, characterized by: The system includes: a server applied to federated learning and a node applied to federated learning; The server is used to send the target task to the nodes participating in the federated learning; t Send to nodes participating in federated learning; receive the model sent by each node Based on the model Generate the model to be trained g t+1 , where i represents the node number and t represents the number of training rounds; in the model to be trained g t+1 If the requirements of the target task are not met, set t=t+1 and return to the training model g. t Sent to the nodes participating in federated learning; in the model to be trained g t+1 When the requirements of the target task are met, the model to be trained g is output t+1 ; The node is used to generate at least one similar subtask j according to the target task, and divide the local data set based on the subtask j to obtain a data set D i,j ; Using the data set D i,j For the model to be trained g t After training, generate a model based on all training results Wherein, the use of the data set D i,j For the model to be trained g t After training, generate a model based on all training results include: On subtask j, use the corresponding training data subset to train the model g t Train and get the model Get the model In the dataset D of subtask j i,j The training gradient on Based on the training gradient Update model g t , get the model

Images