Method and System for Triggering Execution of Federated Learning Tasks Based on Data Version Update

Abstract

The present invention discloses a method and system for triggering the execution of a federated learning task based on data version update. After the data in any participating party's execution end is updated, the participating party's execution end where the data is updated uploads the data to the control end; the control end parses the updated data and saves the newly obtained data element information into the metadata table of the control end; if there is corresponding historical version data and the historical federated learning task uses the historical version data, a new federated learning task is created; the control end generates the configuration parameters of the new federated learning task according to the configuration parameters of the historical federated learning task and the updated data; the control end selects the participating party's execution end where the data is updated and its data, and the control end sends the configuration parameters of the new federated learning task to the selected participating party's execution end; the control end triggers the new federated learning task, and the new federated learning task is executed in the selected participating party's execution end.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and particularly to a method and system for triggering the execution of a federated learning task based on data version update. Background Art

[0002] The statements in this part only mention the background art related to the present invention and do not necessarily constitute prior art.

[0003] In the field of federated learning, it is often necessary to create a new federated training task for a new round of federated training after data is updated. In the prior art, it can be achieved by manually creating a federated training task, but there are its disadvantages: when the data version is updated frequently, manually creating a task is time-consuming and laborious, and the task execution may not be timely; this participating party may not be able to perceive the update of the data version of other participating parties, resulting in the inability to create a new task. Summary of the Invention

[0004] To solve the deficiencies of the prior art, the present invention provides a method and system for triggering the execution of a federated learning task based on data version update; it can solve the problem of automatic execution of federated learning tasks, reduce the degree of user participation, reduce the user operation process, reduce the development cost, and improve the production efficiency.

[0005] In a first aspect, the present invention provides a method for triggering the execution of a federated learning task based on data version update;

[0006] The method for triggering the execution of a federated learning task based on data version update includes:

[0007] After the data in any participating party's execution end is updated, the participating party's execution end where the data is updated uploads the data to the management and control end;

[0008] The management and control end parses the updated data, saves the obtained new data element information into the meta-data table of the management and control end; determines whether there is corresponding historical version data for the new data element information, if there is corresponding historical version data, further determines whether there is a historical federated learning task that uses the historical version data, if there is a historical federated learning task that uses the historical version data, then creates a new federated learning task;

[0009] The management and control end generates the configuration parameters of the new federated learning task according to the configuration parameters of the historical federated learning task and the updated data; the management and control end selects the participating party's execution end where the data is updated and its data, and the management and control end sends the configuration parameters of the new federated learning task to the selected participating party's execution end; the management and control end triggers the new federated learning task, and the new federated learning task is executed within the selected participating party's execution end.

[0010] Second aspect, the present invention provides a system for triggering the execution of a federated learning task based on data version update;

[0011] The system for triggering the execution of a federated learning task based on data version update includes: at least one control terminal and several participating party execution terminals;

[0012] After the data in any participating party execution terminal is updated, the participating party execution terminal where the data is updated uploads the data to the control terminal;

[0013] The control terminal parses the updated data, saves the obtained new data element information into the metadata table of the control terminal; determines whether there is corresponding historical version data for the new data element information, if there is corresponding historical version data, further determines whether there is a historical federated learning task that uses the historical version data, if there is a historical federated learning task that uses the historical version data, then creates a new federated learning task;

[0014] The control terminal generates the configuration parameters of the new federated learning task according to the configuration parameters of the historical federated learning task and the updated data; the control terminal selects the participating party execution terminal where the data is updated and its data, and the control terminal sends the configuration parameters of the new federated learning task to the selected participating party execution terminal; the control terminal triggers the new federated learning task, and the new federated learning task is executed in the selected participating party execution terminal.

[0015] Compared with the prior art, the beneficial effects of the present invention are:

[0016] The participating execution party automatically uploads the updated data to the control terminal, and the control terminal makes judgments on the data itself and the corresponding historical federated learning tasks according to the updated data, and automatically creates a new federated learning task; solves the problem of automatic execution of federated learning tasks, can reduce the degree of user participation, reduce the user operation process, reduce the development cost, and improve the production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The specification drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0018] Figure 1 is a flowchart of a method for triggering the execution of a federated learning task based on data version update provided by an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of the system architecture for triggering the execution of a federated learning task based on data version update provided by an embodiment of the present invention;

[0020] Figure 3Schematic diagram of the system control end module for triggering the execution of federated learning tasks based on data version update provided by an embodiment of the present invention;

[0021] Figure 4 Schematic diagram of the system execution end module for triggering the execution of federated learning tasks based on data version update provided by an embodiment of the present invention;

[0022] Figure 5 Flowchart of the system operation for triggering the execution of federated learning tasks based on data version update provided by an embodiment of the present invention. Detailed implementation manners

[0023] It should be noted that the following detailed description is exemplary and is intended to provide further illustration of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0024] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those clearly listed steps or units, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0025] In the case of no conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other.

[0026] All data acquisition in this embodiment is a legal application of data on the basis of compliance with laws, regulations, and user consent.

[0027] Embodiment 1

[0028] This embodiment provides a method for triggering the execution of federated learning tasks based on data version update;

[0029] As Figure 1 shown, the method for triggering the execution of federated learning tasks based on data version update includes:

[0030] S101: After the data in any participating party's execution end is updated, the participating party's execution end where the data is updated uploads the data to the control end;

[0031] S102: The control end parses the updated data, saves the new data element information obtained from the parsing into the metadata table of the control end; determines whether there is corresponding historical version data for the new data element information, if there is corresponding historical version data, further determines whether there is a historical federated learning task that uses the historical version data, if there is a historical federated learning task that uses the historical version data, then creates a new federated learning task;

[0032] S103: The control end generates the configuration parameters of the new federated learning task according to the configuration parameters of the historical federated learning task and the updated data; the control end selects the participating party execution end for data update and its data, and the control end sends the configuration parameters of the new federated learning task to the selected participating party execution end; the control end triggers the new federated learning task, and the new federated learning task is executed within the selected participating party execution end.

[0033] Further, before the step of method S101, it further includes:

[0034] S100-1: The first participating party execution end uploads data to the control end for the first time;

[0035] S100-2: The control end parses the data uploaded for the first time, and saves the data element information obtained from the parsing into the metadata table of the control end;

[0036] S100-3: The control end creates a federated learning task, selects the participating party execution end and its data, triggers the execution of the task, and the federated learning task is executed within the first participating party execution end and the selected participating party execution end.

[0037] Further, the participating party execution end where the data is updated is the initiator or collaborator in the federated learning task.

[0038] Further, when the data in any participating party execution end is updated, the participating party execution end stores the data in a multi-version manner.

[0039] Further, the historical federated learning task is a federated training task that is manually created, has triggered execution enabled, and has been executed successfully.

[0040] The historical federated learning task is divided into manually created and automatically created ones, and the automatically created task is created based on the manually created task.

[0041] When generating a new federated learning task, it is submitted to the first participating party for execution, and the first participating party will jointly conduct federated training with the second participating party.

[0042] Further, after the step of S103, it further includes:

[0043] After a new federated learning task is executed in the execution end of the selected participating party, the control end stores the configuration parameters and data of the new federated learning task.

[0044] Exemplarily, the first participating party execution end and the second participating party execution end perform federated training using the already uploaded data. This training task information, data information, etc. are written into the database by means of manually calling a program, etc. When the first participating party execution end uploads data again with the same data name after the data is updated, it will detect that there is historical version data for this data, and at the same time, there is a manually created, trigger-execution-enabled, and successfully executed federated training task that uses the historical version data. At this time, the automatic creation of a new task will be triggered. When the new task is automatically created, it will generate the configuration parameters of the new version task according to the parameters of the historical task and the new data, and then submit them to the first participating party execution end for execution. At the same time, it will save the configuration parameters and data and other information of the new version task to the database.

[0045] Embodiment 2

[0046] This embodiment provides a system for triggering the execution of a federated learning task based on data version update;

[0047] A system for triggering the execution of a federated learning task based on data version update includes: at least one control end and several participating party execution ends;

[0048] After the data in any participating party execution end is updated, the participating party execution end where the data is updated uploads the data to the control end;

[0049] The control end parses the updated data, saves the obtained new data element information to the metadata table of the control end; determines whether there is corresponding historical version data for the new data element information. If there is corresponding historical version data, further determines whether there is a historical federated learning task that uses the historical version data. If there is a historical federated learning task that uses the historical version data, a new federated learning task is created;

[0050] The control end generates the configuration parameters of the new federated learning task according to the configuration parameters of the historical federated learning task and the updated data; the control end selects the participating party execution end where the data is updated and its data, and the control end sends the configuration parameters of the new federated learning task to the selected participating party execution end; the control end triggers the new federated learning task, and the new federated learning task is executed in the selected participating party execution end.

[0051] Figure 2It is a schematic diagram of a module of a system provided by an embodiment of the present invention. The system includes a control end and an execution end. The execution end is further divided into a first participant execution end and a second participant execution end according to the role of the federated learning participant. In a federated learning task, it is required to include one first participant and one or more second participants.

[0052] Furthermore, the participant execution end is deployed at each participant, and the number of participants shall not be less than two; the control end is independently deployed and does not belong to any participant.

[0053] Furthermore, as Figure 4 shown, the execution end includes: a data upload module, a data management module, a federated learning module, and a message queue module.

[0054] The data upload module is used to store data in the data management module, store metadata in the federated learning module, and send the metadata to the message queue module;

[0055] The data management module is used to store and manage multiple versions of data when the data names are the same.

[0056] The federated learning module is used to execute federated training tasks.

[0057] The message queue module is used to cache metadata information.

[0058] Furthermore, as Figure 3 shown, the control end includes: a message parsing module, a database module, and a task creation module.

[0059] The message parsing module is used to process the messages in each message queue module, parse the metadata and save it to the metadata table in the database module, and judge whether the triggering condition for task execution is satisfied. If the condition is satisfied, it triggers the task creation module to automatically create a task.

[0060] The database module stores a federated task table, a federated task participant table, and a metadata table.

[0061] The federated task table records the task id, name, creator, configuration parameters, etc.

[0062] The federated task participant table records the task id, participant id, data id, participant role, etc.

[0063] The metadata table records the data id, name, version id, path, fields, etc.

[0064] The task creation module is used to create a federated learning task, save task information, and submit the task to the federated learning module of the first participant execution end for execution.

[0065] Figure 5 It is an operation flow chart, and the specific operation steps include:

[0066] Step 1, the federated training participants perform the first data upload through the data upload module, store the data in multiple versions through the data management module, and add the data meta-information to the federated learning module and the message queue module.

[0067] Step 2, the message parsing module parses the data meta-information from the message queue module and saves it to the meta-data table of the database module. Since this is the first upload of the data, no historical version data of this data will be found in the meta-data table, so the execution of the task will not be triggered.

[0068] Step 3, manually create a federated learning task through the task creation module, select the participants and their data, and enable trigger execution. The task and parameter configuration information will be saved to the federated task table of the database module, and the participant and data information will be saved to the federated task participant table. At the same time, the task creation module submits the task to the federated learning module of the first participant for execution.

[0069] Step 4, after the data is updated, the first participant or the second participant performs data upload again through the data upload module. The data is saved to the data management module, and the metadata is added to the federated learning module and the message queue module.

[0070] Step 5, the message parsing module parses the data meta-information from the message queue module and saves it to the meta-data table of the database module. At the same time, it is found from the meta-data table that there is historical version data of this data, and there is a task that uses the historical version data for training. At this time, the task creation module will be triggered to automatically create and execute the task.

[0071] Step 6, the task creation module generates the parameter configuration information of the new task according to the historical task parameters and the new data, saves the parameter configuration information of the new task to the federated task table of the database module, and saves the participant and the new version data information to the federated task participant table. At the same time, submit the task to the federated learning module of the first participant for execution.

[0072] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for triggering the execution of a federated learning task based on data version update, characterized in that including: After the data in any participating party's execution end is updated, the participating party's execution end where the data is updated uploads the data to the management and control end; The management and control end parses the updated data and saves the metadata of the newly obtained data into the metadata table of the management and control end; Judge whether there is corresponding historical version data for the metadata of the new data. If there is corresponding historical version data, further judge whether there is a historical federated learning task that uses the historical version data. If there is a historical federated learning task that uses the historical version data, then create a new federated learning task; The management and control end generates the configuration parameters of the new federated learning task according to the configuration parameters of the historical federated learning task and the updated data; the management and control end selects the participating party's execution end where the data is updated and its data, and the management and control end sends the configuration parameters of the new federated learning task to the selected participating party's execution end; the management and control end triggers the new federated learning task, and the new federated learning task is executed in the selected participating party's execution end.

2. The method for triggering the execution of a federated learning task based on data version update according to claim 1, wherein Before the above-mentioned "After the data in any participating party's execution end is updated, the participating party's execution end where the data is updated uploads the data to the management and control end", it also includes: The first participating party's execution end uploads the data to the management and control end for the first time; The management and control end parses the data uploaded for the first time and saves the metadata of the parsed data into the metadata table of the management and control end; The management and control end creates a federated learning task, selects the participating party's execution end and its data, and triggers the execution of the task. The federated learning task is executed in the first participating party's execution end and the selected participating party's execution end.

3. The method for triggering the execution of a federated learning task based on data version update as claimed in claim 1, wherein The participating party's execution end where the data is updated is the initiator or collaborator in the federated learning task.

4. The method for triggering the execution of a federated learning task based on data version update as claimed in claim 1, wherein, When the data in any participating party's execution end is updated, the participating party's execution end stores the data in a multi-version manner.

5. The method for triggering the execution of a federated learning task based on data version update as claimed in claim 1, wherein, The historical federated learning task is a manually created, triggered and executed, and successfully executed federated training task.

6. The method for triggering the execution of a federated learning task based on data version update according to claim 1, wherein The historical federated learning task is divided into manually created and automatically created, and the automatically created task is created based on the manually created task.

7. The method for triggering the execution of a federated learning task based on data version update according to claim 1, characterized in that, After the management and control end triggers the new federated learning task and the new federated learning task is executed in the selected participating party's execution end, it also includes: after the new federated learning task is executed in the selected participating party's execution end, the management and control end stores the configuration parameters and data of the new federated learning task.

8. A system for triggering the execution of a federated learning task based on data version update, characterized in that including: At least one management and control end and several participating party's execution ends; After the data in any participating party's execution end is updated, the participating party's execution end where the data is updated uploads the data to the management and control end; The management and control end parses the updated data and saves the metadata of the newly obtained data into the metadata table of the management and control end; Judge whether there is corresponding historical version data for the metadata of the new data. If there is corresponding historical version data, further judge whether there is a historical federated learning task that uses the historical version data. If there is a historical federated learning task that uses the historical version data, then create a new federated learning task; The control terminal generates the configuration parameters of the new federated learning task based on the configuration parameters of the historical federated learning task and the updated data; the control terminal selects the participating execution terminals with data updates and their data, and the control terminal sends the configuration parameters of the new federated learning task to the selected participating execution terminals; the control terminal triggers the new federated learning task, and the new federated learning task is executed within the selected participating execution terminals.

9. The system for triggering the execution of a federated learning task based on data version update according to claim 8, characterized in that, The execution terminal includes: a data upload module, a data management module, a federated learning module, and a message queue module; The data upload module is used to store data in the data management module, store metadata in the federated learning module, and send the metadata to the message queue module; The data management module is used to store and manage multiple versions of data when the data names are the same; The federated learning module is used to execute the federated training task; The message queue module is used to cache metadata information.

10. The system for triggering the execution of a federated learning task based on data version update according to claim 8, wherein The control terminal includes: a message parsing module, a database module, and a task creation module; The message parsing module is used to process the messages in each message queue module, parse the metadata and save it to the metadata table in the database module, and judge whether the trigger condition for task execution is met. If the condition is met, it triggers the task creation module to automatically create a task; The database module stores a federated task table, a federated task participant table, and a metadata table; The federated task table records the task ID, name, creator, and configuration parameters; The federated task participant table records the task ID, participant ID, data ID, and participant role; The metadata table records the data ID, name, version ID, path, and fields; The task creation module is used to create a federated learning task, save task information, and submit the task to the federated learning module of the first participating execution terminal for execution.

Images