Horizontal federated learning system optimization method, device, equipment and readable storage medium

Abstract

The present invention discloses a method, device, equipment and readable storage medium for optimizing a horizontal federated learning system, which obtains the device resource information of each participating device participating in horizontal federated learning; configures the computing task parameters in the training process of the federated learning model corresponding to each participating device according to each of the device resource information, and the computing task parameters include the expected processing time step and / or the expected processing batch size; sends the computing task parameters to each of the participating devices, so that each of the participating devices can perform the federated learning task according to the respective computing task parameters. The present invention realizes that the contribution brought by the data of each data party is utilized in federated learning, and the overall efficiency of the federated learning system is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, device, equipment and readable storage medium for optimizing a horizontal federated learning system. Background Art

[0002] With the gradual development of federated learning technology, federated learning is increasingly being used in various fields to solve the problem of "data islands". As federated learning technology becomes more mature, the efficiency requirements for the federated learning process are becoming higher and higher in many fields, such as in the field of real-time data processing. However, in the actual federated learning process, the technical resource conditions of the devices of each data party are different. Some devices are relatively lacking in computing resources and cannot support efficient federated learning well, and may even reduce the overall efficiency of federated learning. If these devices are not allowed to participate in federated learning in order to prevent them from affecting the overall efficiency of federated learning, the contribution of the data in these devices cannot be utilized. Therefore, how to achieve the use of the contribution of the data of each data party in federated learning without affecting the overall efficiency of federated learning has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the present invention is to provide a method, device, equipment and readable storage medium for optimizing a horizontal federated learning system, aiming to solve the problem of how to utilize the contribution of data from each data party in federated learning without affecting the overall efficiency of federated learning.

[0004] To achieve the above object, the present invention provides a method for optimizing a horizontal federated learning system, which is applied to a coordination device participating in horizontal federated learning. The method for optimizing a horizontal federated learning system comprises the following steps:

[0005] Obtain device resource information of each participating device in horizontal federated learning;

[0006] According to the device resource information, respectively configure the computing task parameters in the federated learning model training process corresponding to each of the participating devices, wherein the computing task parameters include an estimated processing time step and / or an estimated processing batch size;

[0007] The computing task parameters are sent to each of the participating devices accordingly, so that each of the participating devices can perform the federated learning task according to its own computing task parameters.

[0008] Optionally, the step of respectively configuring the computing task parameters in the federated learning model training process corresponding to each participating device according to the device resource information includes:

[0009] Classifying each of the participating devices according to the resource information of each of the devices, and determining the resource category to which each of the participating devices belongs;

[0010] According to the resource category to which each of the participating devices belongs, the computing task parameters in the federated learning model training process corresponding to each of the participating devices are configured respectively.

[0011] Optionally, the step of respectively configuring the computing task parameters in the federated learning model training process corresponding to each of the participating devices according to the resource category to which each of the participating devices belongs includes:

[0012] Determine candidate task parameters corresponding to each of the participating devices according to the resource category to which each of the participating devices belongs;

[0013] Determine the estimated processing time corresponding to each of the participating devices based on the candidate task parameters, and detect whether the estimated processing times meet a preset time consistency condition;

[0014] If the estimated processing times satisfy the preset time consistency condition, the candidate task parameters of each participating device are used as the calculation task parameters of each participating device.

[0015] Optionally, when the model to be trained in the horizontal federated learning is a recurrent neural network model, and the computing task parameters include an estimated processing time step, after the step of sending the computing task parameters to each of the participating devices, the step further includes:

[0016] According to the estimated processing time steps corresponding to the participating devices, respectively, configuring the time step selection strategies corresponding to the participating devices respectively;

[0017] The time step selection strategy is sent to each of the participating devices accordingly, so that each of the participating devices selects sequence selected data from its own sequence data according to its own time step selection strategy, and performs the federated learning task according to the sequence selected data, wherein the time step of the sequence selected data is less than or equal to the expected processing time step of each of the participating devices.

[0018] Optionally, when the computing task parameters include an estimated processing batch size, after the step of sending the computing task parameters to each of the participating devices, the method further includes:

[0019] According to the estimated processing batch size corresponding to each of the participating devices, configure the learning rate corresponding to each of the participating devices;

[0020] The learning rate is sent to each participating device accordingly, so that each participating device performs the federated learning task according to the respective learning rate and the expected processing batch size received from the coordination device.

[0021] Optionally, the step of sending the computing task parameters to each of the participating devices so that each of the participating devices performs the federated learning task according to the respective computing task parameters includes:

[0022] The computing task parameters are sent to each of the participating devices accordingly, and the estimated duration of this round of global model update is sent to each of the participating devices, so that each of the participating devices can adjust the number of local model trainings according to the estimated duration when performing local model training based on the computing task parameters.

[0023] Optionally, the step of obtaining device resource information of each participating device participating in horizontal federated learning includes:

[0024] Receive device resource information sent by each participating device participating in horizontal federated learning, wherein the device resource information includes at least one or more of power resource information, computing resource information, and communication resource information.

[0025] To achieve the above object, the present invention provides a horizontal federated learning system optimization device, which is deployed on a coordination device participating in horizontal federated learning, and includes:

[0026] An acquisition module is used to obtain device resource information of each participating device participating in horizontal federated learning;

[0027] A configuration module, used to configure the computing task parameters in the federated learning model training process corresponding to each of the participating devices according to the device resource information, wherein the computing task parameters include an estimated processing time step and / or an estimated processing batch size;

[0028] The sending module is used to send the computing task parameters to each of the participating devices so that each of the participating devices can perform the federated learning task according to its own computing task parameters.

[0029] To achieve the above-mentioned objectives, the present invention also provides a horizontal federated learning system optimization device, which includes: a memory, a processor, and a horizontal federated learning system optimization program stored in the memory and executable on the processor, and when the horizontal federated learning system optimization program is executed by the processor, the steps of the horizontal federated learning system optimization method as described above are implemented.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also proposes a computer-readable storage medium, on which a horizontal federated learning system optimization program is stored. When the horizontal federated learning system optimization program is executed by a processor, the steps of the horizontal federated learning system optimization method as described above are implemented.

[0031] In the present invention, the device resource information of each participating device is obtained through the coordinating device; the computing task parameters are configured for each participating device according to the device resource information of each participating device, and the computing task parameters include the expected processing time step and / or the expected processing batch size; the computing task parameters of each participating device are sent to each participating device accordingly, so that each participating device can perform the federated learning task according to the computing task parameters. In the present invention, the computing tasks to be processed locally by each participating device are coordinated by configuring computing task parameters for each participating device through a coordination device, wherein the computing task parameters include an estimated processing time step and / or an estimated processing batch size, that is, by configuring the estimated processing time step and / or the estimated processing batch size for the participating device, the amount of computing tasks to be processed locally by each participating device is coordinated; and by configuring different computing task parameters for each participating device according to the device resource information of each participating device, the differences in device resource conditions of each participating device are taken into account; for participating devices with relatively abundant device resources, more computing tasks are assigned to them, and for participating devices with relatively scarce device resources, fewer computing tasks are assigned to them, so that participating devices with relatively scarce device resources can also quickly complete the update of local model parameters without the need for participating devices with relatively abundant computing resources to spend time waiting, thereby improving the overall efficiency of horizontal federated learning by each participating device, and at the same time, the contribution of local data of each participating device to model training can be used, including the contribution brought by participating devices with relatively scarce resources, further increasing the stability of the model, thereby achieving the optimization of the efficiency and model performance of the horizontal federated learning system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention;

[0033] Figure 2 This is a flow chart of the first embodiment of the horizontal federated learning system optimization method of the present invention;

[0034] Figure 3 This is a functional schematic module diagram of a preferred embodiment of the horizontal federated learning system optimization device of the present invention.

[0035] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0036] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0037] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0038] It should be noted that the horizontal federated learning system optimization device in the embodiment of the present invention can be a smart phone, a personal computer, a server and other devices, and no specific limitation is made here.

[0039] like Figure 1 As shown, the horizontal federated learning system optimization device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0040] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation on the horizontal federated learning system optimization device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0041] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a horizontal federated learning system optimization program. Among them, the operating system is a program that manages and controls the hardware and software resources of the device, and supports the operation of the horizontal federated learning system optimization program and other software or programs.

[0042] When the horizontal federated learning system optimization device is the coordination device participating in horizontal federated learning, Figure 1 In the device shown, the user interface 1003 is mainly used to communicate data with the client; the network interface 1004 is mainly used to establish a communication connection with the participating devices participating in the horizontal federated learning; the processor 1001 can be used to call the horizontal federated learning system optimization program stored in the memory 1005, and perform the following operations:

[0043] Obtain device resource information of each participating device in horizontal federated learning;

[0044] According to the device resource information, respectively configure the computing task parameters in the federated learning model training process corresponding to each of the participating devices, wherein the computing task parameters include an estimated processing time step and / or an estimated processing batch size;

[0045] The computing task parameters are sent to each of the participating devices accordingly, so that each of the participating devices can perform the federated learning task according to its own computing task parameters.

[0046] Furthermore, the step of respectively configuring the computing task parameters in the federated learning model training process corresponding to each of the participating devices according to the device resource information includes:

[0047] Classifying each of the participating devices according to the resource information of each of the devices, and determining the resource category to which each of the participating devices belongs;

[0048] According to the resource category to which each of the participating devices belongs, the computing task parameters in the federated learning model training process corresponding to each of the participating devices are configured respectively.

[0049] Furthermore, the step of respectively configuring the computing task parameters in the federated learning model training process corresponding to each of the participating devices according to the resource category to which each of the participating devices belongs includes:

[0050] Determine candidate task parameters corresponding to each of the participating devices according to the resource category to which each of the participating devices belongs;

[0051] Determine the estimated processing time corresponding to each of the participating devices based on the candidate task parameters, and detect whether the estimated processing times meet a preset time consistency condition;

[0052] If the estimated processing times satisfy the preset time consistency condition, the candidate task parameters of each participating device are used as the calculation task parameters of each participating device.

[0053] Further, when the model to be trained in the horizontal federated learning is a recurrent neural network model, and the computing task parameters include an estimated processing time step, after the step of sending the computing task parameters to each of the participating devices, the processor 1001 can be used to call the horizontal federated learning system optimization program stored in the memory 1005, and further perform the following operations:

[0054] According to the estimated processing time steps corresponding to the participating devices, respectively, configuring the time step selection strategies corresponding to the participating devices respectively;

[0055] The time step selection strategy is sent to each of the participating devices accordingly, so that each of the participating devices selects sequence selected data from its own sequence data according to its own time step selection strategy, and performs the federated learning task according to the sequence selected data, wherein the time step of the sequence selected data is less than or equal to the expected processing time step of each of the participating devices.

[0056] Further, when the computing task parameters include an estimated processing batch size, after the step of sending the computing task parameters to each of the participating devices, the processor 1001 may be used to call a horizontal federated learning system optimization program stored in the memory 1005, and further perform the following operations:

[0057] According to the estimated processing batch size corresponding to each of the participating devices, configure the learning rate corresponding to each of the participating devices;

[0058] The learning rate is sent to each participating device accordingly, so that each participating device performs the federated learning task according to the respective learning rate and the expected processing batch size received from the coordination device.

[0059] Furthermore, the step of sending the computing task parameters to each of the participating devices so that each of the participating devices performs the federated learning task according to the computing task parameters includes:

[0060] The computing task parameters are sent to each of the participating devices accordingly, and the estimated duration of this round of global model update is sent to each of the participating devices, so that each of the participating devices can adjust the number of local model trainings according to the estimated duration when performing local model training based on the computing task parameters.

[0061] Furthermore, the step of obtaining device resource information of each participating device participating in horizontal federated learning includes:

[0062] Receive device resource information sent by each participating device participating in horizontal federated learning, wherein the device resource information includes at least one or more of power resource information, computing resource information, and communication resource information.

[0063] Based on the above structure, various embodiments of the horizontal federated learning system optimization method are proposed.

[0064] Reference Figure 2 , Figure 2 This is a flowchart of the first embodiment of the horizontal federated learning system optimization method of the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here.

[0065] In this embodiment, the horizontal federated learning system optimization method is applied to the coordination device participating in the horizontal federated learning, and the coordination device and each participating device participating in the horizontal federated learning can be a smart phone, a personal computer, a server and other devices. In this embodiment, the horizontal federated learning system optimization method includes:

[0066] Step S10, obtaining device resource information of each participating device participating in horizontal federated learning;

[0067] In this embodiment, the coordination device and each participating device can establish a communication connection in advance by asking for handshake authentication and identity authentication, and determine the model to be trained for this federated learning, such as a neural network model or other machine learning model. Each participating device can locally build the model to be trained with the same or similar structure, or the coordination device can build the model to be trained and send it to each participating device. Each participating device has training data for training the model to be trained locally.

[0068] In horizontal federated learning, the coordinating device and the participating devices cooperate with each other to perform multiple global model updates on the model to be trained. Model update refers to updating the model parameters of the model to be trained, such as the connection weight values ​​between neurons in the neural network, and finally obtaining a model that meets the quality requirements. In a global model update, each participating device uses its own local training data to locally train the local model to be trained to obtain a local model parameter update. The local model parameter update can be the gradient information used to update the model parameters, or the model parameters after local update; each participating device sends its own local model parameter update to the coordinating device; the coordinating device fuses the local model parameter updates, such as weighted averaging, to obtain a global model parameter update, and sends it to each participating device; each participating device uses the global model parameter update to update the model parameters of the local model to be trained, that is, to perform a model update on the local model to be trained, thus completing a global model update. After each global model update, the model parameters of the local models to be trained of each participating device are synchronized.

[0069] During the federated learning process, the coordinating device can obtain the device resource information of each participating device. The device resource information can be the resource information related to computing efficiency and communication efficiency in the participating devices, such as computing resource information, power resource information, and communication resource information. The computing resources can be represented by the number of CPUs and GPUs owned by the participating devices, the power resources can be represented by the time that the participating devices can continue to work, and the communication resources can be represented by the communication rate of the participating devices. The coordinating device can send a device resource query request to each participating device, and after receiving the request, each participating device uploads its current device resource information to the coordinating device.

[0070] Step S20, respectively configuring the computing task parameters in the federated learning model training process corresponding to each of the participating devices according to the device resource information, wherein the computing task parameters include an estimated processing time step and / or an estimated processing batch size;

[0071] The coordination device can obtain the device resource information of each participating device before the first global model update to configure the computing task parameters in the subsequent global model update for each participating device; or, before entering a certain or each global model update, obtain the device resource information of each participating device to configure the computing task parameters in the current global model update for each participating device. That is, the coordination device can configure the computing task parameters for each participating device to participate in the federated learning model training by obtaining the device resource information of each participating device during the federated learning model training process.

[0072] Among them, the computing task parameters include the expected processing time step and / or the expected processing batch size, that is, the computing task parameters include the expected processing time step, or include the expected processing batch size, or include the expected processing time step and the expected processing batch size. The time step refers to the number of time steps. The time step is a concept in the recurrent neural network model and is for sequence data. The expected processing time step refers to the number of time steps that the participating device is expected to process. The batch size (mini-batch size) refers to the size of the data batch used for model update each time. The expected processing batch size refers to the data batch size that the participating device is expected to use when performing local model updates.

[0073] After obtaining the device resource information of each participating device, the coordination device configures the computing task parameters for each participating device according to the device resource information. Specifically, the richer the computing resources of the participating device, the higher the computing efficiency; the richer the power resources, the longer the time that can continuously participate in federated learning; the richer the communication resources (large communication bandwidth, short communication delay), the higher the efficiency of data transmission. Then, for the same computing task, the device with richer device resource information takes less time. The principle of configuring computing task parameters is to configure more computing tasks for participating devices with richer device resources, and configure fewer computing tasks for participating devices with fewer device resources, so as to make the processing time of each participating device the same as much as possible. The number of computing tasks can be quantified by the expected processing time step or the expected processing batch size; the more expected processing time steps or the larger the expected processing batch size, the more data the participating device has to process, and thus, the more expected processing time steps, the more computing tasks, and the larger the expected processing batch size, the more computing tasks. It should be noted that since the participating devices upload gradient information or model parameter information locally, the amount of uploaded data will not vary due to different batch sizes or time step sizes. If the participating devices have abundant communication resources, it means that the participating devices have fast data transmission speeds and need to spend less time uploading data, so they can spend more time on local model training. Therefore, the coordinating device can assign more computing tasks to participating devices with abundant communication resources, that is, configure more estimated processing time steps or estimated processing batch sizes.

[0074] There are many ways for the coordination device to configure the computing task parameters according to the device resource information of the participating devices. For example, the corresponding relationship between the device resource information and the computing task parameters can be pre-set; the device resource information can be divided into several segments according to the numerical value, and each segment represents a different resource richness. For example, when the device resource information is the number of CPUs, the number of CPUs can be divided into several segments, and the more CPUs a segment has, the richer the technical resources it represents; the computing task parameters corresponding to each segment are pre-set, that is, the corresponding relationship, and the more resource richness a segment represents, the more computing tasks are limited by the corresponding set computing task parameters; the coordination device determines which segment the device resource information of the participating device falls into, and configures the computing task parameters corresponding to that segment for the participating device.

[0075] Step S30, sending the computing task parameters to each of the participating devices accordingly, so that each of the participating devices performs the federated learning task according to its own computing task parameters.

[0076] The coordination device sends the computing task parameters of each participating device to each participating device. After receiving the computing task parameters, the participating device uses the received computing task parameters to perform the federated learning task. Specifically, when the coordination device sends the computing task parameters to be used in each subsequent global model update, the participating device participates in each subsequent global model update based on the computing task parameters, thereby completing the federated learning task; when the coordination device sends the computing task parameters to be used in a global model update, the participating device participates in the global model update based on the computing task parameters.

[0077] Take the example of a participating device participating in a global model update based on the calculation task parameters:

[0078] When the model to be trained is a recurrent neural network and the calculation task parameter is the expected processing time step, the participating device selects the data of the expected processing time step from the local sequence data used for model training (the selected data is called sequence selection data); specifically, for each sequence data composed of multiple time step data, the participating device selects a part of the time step data from the sequence data as the sequence selection data, and the time step of this part of the time step data can be less than or equal to the expected processing time step, wherein, when the time step of the sequence data is less than or equal to the expected processing time step, the sequence data is not selected, and the sequence data is directly used as the sequence selection data; for example, for a sequence data with 32 time step data, the expected processing time step is 15, then the participating device can select 15 time step data from the sequence data as the sequence selection data; there are many ways to select. , for example, selecting one at an interval, or randomly selecting 15, etc. Different selection methods can be set according to the specific model application scenario; in addition, the selection methods used for each global model parameter update can be different, so as to ensure that most of the local data can be used to participate in model training; based on the selected time steps that are less than or equal to the expected processing time step, the selected data is selected, and the model to be trained is trained once or multiple times to obtain a local model parameter update. The process of local model training is consistent with the general training process of the recurrent neural network and is not described in detail here; the local model parameter update is sent to the coordination device, and the coordination device integrates the local model parameter updates of each participating device to obtain a global model parameter update, and sends it to each participating device; after receiving the global model parameter update, the participating device uses the global model parameter update to update the model parameters of the local model to be trained.

[0079] When the model to be trained is a neural network model or other machine learning model, and the computing task parameter is the expected processing batch size, the participating device can divide the multiple local training data into multiple batches, and the size of each batch, that is, the number of training data contained in each batch, is less than or equal to the expected processing batch size received from the coordinating device; for example, the expected processing batch size received by the participating device is 100, and the participating device has 1,000 training data locally, then the participating device can divide the local training data into 10 batches; after the participating device divides the local training data into batches according to the expected processing batch size, it uses one batch of data to update the local model during the process of participating in a global model update.

[0080] When the model to be trained is a recurrent neural network and the computing task parameters are the expected processing time step and the expected processing batch size, the participating devices combine the operations in the above two cases, select the time step for each training data, and batch the selected sequence selected data, and use a batch of sequence selected data each time to participate in the global model update.

[0081] It can be understood that since the neural network nodes corresponding to different time steps in the recurrent neural network share weights, the input data of the recurrent neural network can be variable in length, that is, the time step of the input data can be different, thereby supporting each participating device to perform local model training based on different expected processing time steps.

[0082] In this embodiment, the device resource information of each participating device is obtained through the coordinating device; the computing task parameters are configured for each participating device according to the device resource information of each participating device, and the computing task parameters include the expected processing time step and / or the expected processing batch size; the computing task parameters of each participating device are sent to each participating device accordingly, so that each participating device can perform the federated learning task according to the computing task parameters. In this embodiment, the computing tasks to be processed locally by each participating device are coordinated by configuring computing task parameters for each participating device through a coordination device, wherein the computing task parameters include an estimated processing time step and / or an estimated processing batch size, that is, by configuring the estimated processing time step and / or the estimated processing batch size for the participating device, the amount of computing tasks to be processed locally by each participating device is coordinated; and by configuring different computing task parameters for each participating device according to the device resource information of each participating device, the differences in device resource conditions of each participating device are taken into account; for participating devices with relatively abundant device resources, more computing tasks are assigned to them, and for participating devices with relatively scarce device resources, fewer computing tasks are assigned to them, so that participating devices with relatively scarce device resources can also quickly complete the local model parameter update without the need for participating devices with relatively abundant computing resources to spend time waiting, thereby improving the overall efficiency of each participating device in horizontal federated learning, and at the same time, it can also use the contribution of local data of each participating device to model training, including being able to utilize the contribution brought by participating devices with relatively scarce resources, further increasing the stability of the model, thereby achieving the optimization of the efficiency and model performance of the horizontal federated learning system.

[0083] Further, based on the above first embodiment, a second embodiment of the horizontal federated learning system optimization method of the present invention is proposed. In this embodiment, step S20 includes:

[0084] Step S201, classifying each of the participating devices according to the device resource information, and determining the resource category to which each of the participating devices belongs;

[0085] In this embodiment, a feasible method is proposed for the coordination device to configure the computing task parameters according to the device resource information of the participating devices. Specifically, the coordination device classifies each participating device according to the device resource information of each participating device, and determines the resource category described by each participating device. The coordination device can arrange the resource information of each device according to the numerical value, and can set the number of categories to be classified in advance, and then divide the interval formed by the minimum and maximum values ​​in the sorting into several division segments with a preset number of categories, then each division segment is a category, and it is determined in which division segment the numerical value corresponding to the resource information of each participating device belongs to which category. For example, when the device resource information includes computing resource information, and the computing resource information is represented by the number of CPUs of the participating devices, the number of CPUs of each participating device can be arranged. It can be understood that compared with the method of presetting each resource category, the resource category is determined according to the resource information of each device in this embodiment, so that the division of resource categories is more in line with the actual resource situation of the participating devices, and can adapt to the resource situation of the participating devices that is not fixed in the federated learning process.

[0086] When the device resource information includes data of multiple types of device resources, the data of each type of device resources can be normalized so that the data of each type of resources can be calculated and compared. The normalization method can adopt a commonly used normalization method, which will not be described in detail here. For example, when the device resources include computing resources, power resources and communication resources, by normalizing the data of computing resources, power resources and communication resources, the computing resources, power resources and communication resources can be calculated and compared; for the device resource information of each participating device, the normalized data of each type of device resources of the participating device is obtained, and the weight value of each type of resource can be set in advance according to the degree of influence of each type of resource on the local computing efficiency of the participating device, and then the normalized data of each type of device resources is weighted and averaged to obtain a value that can evaluate the overall resource richness of the participating device; the coordination device performs the above-mentioned sorting, division and classification operations based on the calculated values ​​of each participating device. By normalizing the device resource information of each participating device and calculating an overall resource richness value, the complex device resource information is quantified, so that the computing task parameters can be configured for each participating device more conveniently and accurately; and by quantifying the complex device resource information, the resource categories of each participating device can be divided, so that computing tasks can be configured for the participating devices more quickly.

[0087] Step S202: According to the resource category to which each of the participating devices belongs, respectively configure the computing task parameters in the federated learning model training process corresponding to each of the participating devices.

[0088] After determining the resource category to which each participating device belongs, the coordination device configures the computing task parameters corresponding to each participating device according to the resource category to which each participating device belongs. Specifically, a maximum expected processing time step can be set in advance, and each resource category can be numbered from low to high according to the resource richness: 1, 2, 3..., and then the expected processing time step corresponding to each category is calculated according to the maximum expected processing time step. Specifically, the maximum expected processing time step can be divided by the number of resource categories to obtain a minimum time step, and then the number of each resource category is multiplied by the minimum time step to obtain the expected processing time step corresponding to each resource category. For example, the maximum expected processing time step is 32, and the number of resource categories is 4. Then, from the low to high resource category richness, the expected processing time steps corresponding to the four resource categories are: 8, 16, 24 and 32 respectively. Similarly, a maximum expected processing batch size can also be set in advance, and then the expected processing batch size corresponding to each category is calculated according to the maximum expected processing batch size.

[0089] Furthermore, the step S202 includes:

[0090] Step S2021, determining candidate task parameters corresponding to each of the participating devices according to the resource category to which each of the participating devices belongs;

[0091] Furthermore, the coordination device can determine the candidate task parameters corresponding to each participating device according to the resource category to which each participating device belongs. Specifically, the computing task parameters corresponding to each resource category can be calculated in a similar manner as described above, in which the estimated processing time step corresponding to each resource category is calculated according to the maximum estimated processing time step, and then the computing task parameters corresponding to each participating device can be determined according to the resource category to which each participating device belongs, and the computing task parameters can be used as candidate task parameters.

[0092] Step S2022, determining the estimated processing time corresponding to each of the participating devices based on the candidate task parameters, and detecting whether the estimated processing time satisfies a preset time consistency condition;

[0093] The coordinating device can determine the estimated processing time corresponding to each participating device based on the candidate task parameters corresponding to each participating device, that is, determine the time required for each participating device to perform the federated learning task according to its own candidate task parameters. Specifically, it may be the time required for the participating device to perform local model training and upload model parameter updates when the participating device participates in the next global model update according to the candidate task parameters.

[0094] Specifically, the coordination device can estimate the time required for each participating device to perform local model training and upload model parameter updates according to the candidate task parameters based on the device resource information of each participating device. Among them, the unit time for processing a unit time step or a unit batch size using unit resources can be set in advance based on testing or experience. Then, the estimated processing time of the participating device can be calculated based on this unit time and the actual resources owned by the participating device, as well as the estimated processing time step or estimated processing batch size in the candidate task parameters. The whole process can be similar to the principle of multiplying the unit by the quantity to get the total amount. For example, it is set in advance based on experience that the participating device has 1 CPU. When processing data of 1 time step, it takes x time to train the local model and y time to upload the model parameter update. If a participating device has 3 CPUs and needs to process 10 time steps, then the estimated processing time of the participating device is (10x+10y) / 3.

[0095] After calculating the estimated processing time corresponding to each participating device, the coordination device can detect whether the estimated processing time of each participating device meets the preset time consistency condition. Among them, the preset time consistency condition can be pre-set according to the principle of ensuring that the estimated processing time of each participating device is consistent as much as possible. For example, the preset time consistency condition can be set to that the difference between the maximum value and the minimum value in each estimated processing time needs to be less than a set threshold value. When the coordination device detects that the difference between the maximum value and the minimum value in each estimated processing time is less than the threshold value, it means that the estimated processing time of each participating device meets the preset time consistency condition, that is, the estimated processing time of each participating device is roughly the same. When the coordination device detects that the difference between the maximum value and the minimum value in each estimated processing time is not less than the threshold value, it means that the estimated processing time of each participating device does not meet the preset time consistency condition, that is, the difference between the estimated processing times of each participating device is large.

[0096] Step S2023: If the estimated processing times satisfy the preset time consistency condition, the candidate task parameters of each participating device are used as the calculation task parameters of each participating device.

[0097] If the coordinating device detects that each estimated processing time satisfies a preset time consistency condition, the candidate task parameters of each participating device may be used as the final calculation task parameters of each participating device.

[0098] If the coordination device detects that each estimated processing time does not meet the preset time consistency condition, it means that the estimated processing time of each participating device is quite different. At this time, if each participating device performs subsequent federated learning tasks based on the corresponding candidate task parameters, the processing time will be quite different. Some participating devices may need to wait for other participating devices. For example, in a global model update process, the time spent on local model update and model update parameter upload of some participating devices is relatively short, while the time spent on other participating devices is relatively long, so the coordination device and the participating devices with relatively short time need to wait until the participating devices with relatively long time upload the model parameter update. The coordination device can then merge and obtain the global model parameter update, and then complete the global model update. Therefore, the coordination device can make adjustments based on the parameters of each candidate task when it is detected that each estimated processing time does not meet the preset time consistency condition. For example, the candidate task parameters of the participating device with the largest estimated processing time can be reduced. The reduction can be to reduce the estimated processing time step, or to reduce the estimated processing batch size, or to reduce both the estimated processing time step and the estimated processing batch size. After adjustments are made based on the parameters of each candidate task, the coordinating device again estimates the estimated processing time of each participating device based on the adjusted candidate task parameters, and detects whether each estimated processing time meets the preset time consistency condition, and repeats this cycle until it is detected that the preset time consistency condition is met.

[0099] In this embodiment, by configuring computing task parameters for each participating device so that the estimated processing time of each participating device meets the preset time consistency condition, the estimated processing time of each participating device is made as consistent as possible, so that the participating devices do not need to wait for each other for too long, or even do not need to wait at all, so that even participating devices with poor device resources can keep up with participating devices with rich device resources, so that each participating device can participate in horizontal federated learning, thereby improving the overall efficiency of horizontal federated learning while also being able to utilize the contribution of the data of each participating device. Even participating devices with poor device resources can utilize the contribution of their training data.

[0100] Furthermore, based on the first and second embodiments above, a third embodiment of the horizontal federated learning system optimization method of the present invention is proposed. In this embodiment, after step S30, it also includes:

[0101] Step S40, configuring a time step selection strategy corresponding to each of the participating devices according to the estimated processing time step corresponding to each of the participating devices;

[0102] Furthermore, when the model to be trained in horizontal federated learning is a recurrent neural network model, and the calculation task parameters include an estimated processing time step, the coordination device can configure the time step selection strategy corresponding to each participating device according to the estimated processing time step corresponding to each participating device. Among them, the recurrent neural network (RNN) model involved in each embodiment of the present invention can be an ordinary RNN, or a deep RNN, LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit) and IndRNN (Independently Recurrent Neural Network), etc.

[0103] The time step selection strategy is a strategy for selecting part of the time step data from the sequence data. There are many ways for the coordination device to configure the time step selection strategy for each participating device. The principle is to make the time step data selected from the sequence data by each participating device complement each other as much as possible. For example, if the estimated processing time step of participating device A and participating device B are both 15, the time step selection strategy configured by the coordination device for participating device A can be to select the time step data with an odd sequence number in the sequence data, and the time step selection strategy configured for participating device B can be to select the time step data with an even sequence number in the sequence data. By configuring different time step selection strategies for each participating device, or even complementary time step selection strategies, the sequence data used by each participating device can be distributed differently in time steps when training the local model, so that the model to be trained can learn features from more different sequence data, thereby improving the generalization ability of the model, that is, it can have better prediction ability for new samples of various forms.

[0104] Step S50, sending the time step selection strategy to each of the participating devices accordingly, so that each of the participating devices selects sequence selected data from its own sequence data according to its own time step selection strategy, and performs the federated learning task according to the sequence selected data, wherein the time step of the sequence selected data is less than or equal to the expected processing time step of each of the participating devices.

[0105] The coordination device sends the time step selection strategy of each participating device to each participating device. It should be noted that the coordination device can send the time step selection strategy of each participating device and the expected processing time step to the participating device together, or send them separately. After receiving the expected processing time step and the time step selection strategy, the participating device selects sequence selection data from the local sequence data according to the time step selection strategy, and the time step of the sequence selection data is less than or equal to the expected processing time step.

[0106] Further, in one embodiment, when the calculation task parameters include the estimated processing batch size, after step S30, the method further includes:

[0107] Step S60 configures the learning rate corresponding to each of the participating devices according to the estimated processing batch size corresponding to each of the participating devices;

[0108] Furthermore, when the computing task parameters include an estimated processing batch size, the coordination device can configure the learning rate corresponding to each participating device according to the estimated processing batch size corresponding to each participating device. Among them, the learning rate is a hyperparameter in the model training process. There are many ways for the coordination device to configure the learning rate for each participating device. The principle based on this can be that the estimated processing batch size is proportional to the learning rate. For example, the coordination device can set a baseline learning rate. If the estimated processing batch size of the participating device is smaller than the batch size corresponding to the baseline learning rate, the learning rate smaller than the baseline learning rate is configured for the participating device. If the estimated processing batch size of the participating device is larger than the batch size corresponding to the baseline learning rate, the learning rate larger than the baseline learning rate is configured for the participating device.

[0109] Step S70: Send the learning rate to each participating device accordingly, so that each participating device performs the federated learning task according to the respective learning rate and the estimated processing batch size received from the coordination device.

[0110] After configuring the learning rate for each participating device, the coordinating device sends the learning rate of each participating device to each participating device. It should be noted that the coordinating device can send the learning rate of each participating device and the expected processing batch size to the participating device together, or separately. After receiving the learning rate and the expected processing batch size, the participating device performs the federated learning task according to the learning rate and the expected processing batch size. For example, when training the local model, the data batch of the expected processing batch size is used for model training, and the learning rate is used to update the model parameters when updating the model parameters.

[0111] In this embodiment, the coordinating device configures a learning rate for each participating device based on the expected processing batch size of each participating device, so that the coordinating device can control the model convergence speed of each participating device as a whole, thereby setting a different learning rate for each participating device to make the model convergence speed of each participating device tend to be consistent, so that the model to be trained can converge better during the federated learning process.

[0112] It should be noted that when the calculation task parameters include the expected processing time step and the expected processing batch size, the above-mentioned coordination device configures the time step selection strategy and learning rate of each participating device can also be implemented in combination. Specifically, the coordination device configures the time step selection strategy of each participating device according to the expected processing time step of each participating device, and configures the learning rate of each participating device according to the expected processing batch size of each participating device; the coordination device sends the time step selection strategy and learning rate of each participating device, as well as the calculation task parameters, to each participating device; the participating device selects sequence selection data from the local sequence data according to the time step selection strategy, uses the sequence selection data of the expected processing batch size for local model training, and uses the received learning rate to update the model during the training process.

[0113] Furthermore, the step S20 includes:

[0114] Step S203, sending the computing task parameters to each of the participating devices accordingly, and sending the estimated duration of this round of global model update to each of the participating devices, so that each of the participating devices can adjust the number of local model training times according to the estimated duration when performing local model training based on the computing task parameters.

[0115] The coordination device sends the calculation task parameters of each participating device to each participating device, and at the same time sends the estimated duration of this round of global model update to each participating device. The estimated duration of this round of global model update can be determined based on the estimated processing duration of each participating device, such as taking the maximum value of the estimated processing duration of each participating device as the estimated duration of this round of global model update. After receiving the calculation task parameters and the estimated duration, the participating device performs local model training according to the calculation task parameters, and can adjust the number of local model training according to the estimated duration. Specifically, the participating device can calculate the duration 1 spent on a local model training after performing a local model training, and then determine whether the result obtained by subtracting the estimated duration from the duration 1 is greater than the duration 1. If it is greater, then perform another local model training, that is, increase the number of local model trainings; then calculate the duration 2 spent on the local model training, and then determine whether the result obtained by subtracting the estimated duration from the duration 1 and the duration 2 is greater than the duration 2. If it is greater, then perform another local model training; in this way, until it is detected that the remaining duration is less than the duration spent on the most recent local model training, the local model training will no longer be performed, and the local model parameters obtained from the local model training will be updated and uploaded to the coordination device. That is, when the participating device determines that the remaining duration is sufficient for a local model training based on the estimated duration, it will increase the local model training.

[0116] In this embodiment, the coordinating device sends the estimated duration of this round of global model update to each participating device, so that when each participating device actually performs local model training faster, it can avoid wasting time waiting for other participating devices by increasing the number of local model trainings.

[0117] Furthermore, the step S10 includes:

[0118] Step S101, receiving device resource information sent by each participating device participating in horizontal federated learning, wherein the device resource information at least includes one or more of power resource information, computing resource information and communication resource information.

[0119] Furthermore, each participating device may actively upload its own device resource information to the coordination device. The coordination device receives the device resource information uploaded by each participating device. The device resource information may include at least one or more of power resource information, computing resource information, and communication resource information. Specifically, computing resources may be represented by the number of CPUs and GPUs owned by the participating device, power resources may be represented by the time that the participating device can continue to work, and communication resources may be represented by the communication rate of the participating device.

[0120] Furthermore, in one embodiment, each participating device may be a remote sensing satellite with different sequence image data, and each remote sensing satellite uses its own image data to perform horizontal federated learning to train RNN to complete the weather forecasting task. The coordination device may be one of the remote sensing satellites or a base station located on the ground. The coordination device obtains the device resource information of each remote sensing satellite, and then configures the computing task parameters for each remote sensing satellite according to the device resource information of each remote sensing satellite. The computing task parameters include the expected processing time step and / or the expected processing batch size; the computing task parameters of each remote sensing satellite are sent to each remote sensing satellite accordingly, so that each remote sensing satellite can perform the federated learning task according to the computing task parameters to complete the training of the RNN. After obtaining the trained RNN, each remote sensing satellite can input the most recently captured sequence remote sensing image data into the RNN, and predict the next weather conditions through the RNN. During the RNN training process, the coordination device coordinates the computing tasks of each remote sensing satellite based on the device resource information of each remote sensing satellite, so that remote sensing satellites with more abundant computing resources do not need to spend time waiting during the training process, thereby improving the overall efficiency of horizontal federated learning of each remote sensing satellite, thereby accelerating the deployment of the weather forecast RNN. In addition, during the training process, the contribution of data from each remote sensing satellite to model training can be used, including the contribution of remote sensing satellites with less resources, which further increases the stability of the model, thereby making the credibility of the weather conditions and results predicted by RNN higher.

[0121] In addition, an embodiment of the present invention further provides a horizontal federated learning system optimization device, wherein the device is deployed on a coordination device participating in horizontal federated learning, Figure 3 , the device comprises:

[0122] An acquisition module 10 is used to acquire device resource information of each participating device participating in horizontal federated learning;

[0123] A configuration module 20, configured to configure computing task parameters in the federated learning model training process corresponding to each of the participating devices according to the device resource information, wherein the computing task parameters include an estimated processing time step and / or an estimated processing batch size;

[0124] The sending module 30 is used to send the computing task parameters to each of the participating devices so that each of the participating devices performs the federated learning task according to its own computing task parameters.

[0125] Furthermore, the configuration module 20 includes:

[0126] A classification unit, used to classify each of the participating devices according to the resource information of each of the devices, and determine the resource category to which each of the participating devices belongs;

[0127] A configuration unit is used to configure the computing task parameters in the federated learning model training process corresponding to each participating device according to the resource category to which each participating device belongs.

[0128] Furthermore, the configuration unit includes:

[0129] A first determining subunit is used to determine candidate task parameters corresponding to each of the participating devices according to the resource category to which each of the participating devices belongs;

[0130] A detection subunit, configured to determine the estimated processing time corresponding to each of the participating devices based on the candidate task parameters, and detect whether the estimated processing time satisfies a preset time consistency condition;

[0131] The second determining subunit is used to correspond the candidate task parameters of each participating device as the calculation task parameters of each participating device if the estimated processing times satisfy the preset time consistency condition.

[0132] Further, when the model to be trained in the horizontal federated learning is a recurrent neural network model, and the computing task parameters include an estimated processing time step, the configuration module 20 is further used to configure the time step selection strategy corresponding to each of the participating devices according to the estimated processing time step corresponding to each of the participating devices;

[0133] The sending module 30 is also used to send the time step selection strategy to each of the participating devices, so that each of the participating devices can select sequence selection data from its own sequence data according to its own time step selection strategy, and perform federated learning tasks according to the sequence selection data, wherein the time step of the sequence selection data is less than or equal to the expected processing time step of each of the participating devices.

[0134] Further, when the computing task parameters include an estimated processing batch size, the configuration module 20 is further used to configure the learning rate corresponding to each of the participating devices according to the estimated processing batch size corresponding to each of the participating devices;

[0135] The sending module 30 is also used to send the learning rate to each participating device accordingly, so that each participating device performs the federated learning task according to its own learning rate and the estimated processing batch size received from the coordination device.

[0136] Furthermore, the sending module 30 is also used to send the computing task parameters to each of the participating devices, and send the estimated duration of this round of global model update to each of the participating devices, so that each of the participating devices can adjust the number of local model training according to the estimated duration when performing local model training based on the computing task parameters.

[0137] Furthermore, the acquisition module 10 is also used to receive device resource information sent by each participating device participating in horizontal federated learning, wherein the device resource information includes at least one or more of power resource information, computing resource information and communication resource information.

[0138] The expanded content of the specific implementation method of the horizontal federated learning system optimization device of the present invention is basically the same as the various embodiments of the above-mentioned horizontal federated learning system optimization method, and will not be repeated here.

[0139] In addition, an embodiment of the present invention also proposes a computer-readable storage medium, on which a horizontal federated learning system optimization program is stored. When the horizontal federated learning system optimization program is executed by a processor, the steps of the horizontal federated learning system optimization method described below are implemented.

[0140] The various embodiments of the horizontal federated learning system optimization device and computer-readable storage medium of the present invention can all refer to the various embodiments of the horizontal federated learning system optimization method of the present invention, and will not be repeated here.

[0141] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0142] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the embodiments of the present invention.

[0144] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for optimizing a horizontal federated learning system, characterized in that: The method is applied to a coordination device participating in horizontal federated learning, and the horizontal federated learning system optimization method comprises the following steps: Obtain device resource information of each participating device in horizontal federated learning; According to the device resource information, respectively configure the computing task parameters in the federated learning model training process corresponding to each participating device, wherein the computing task parameters include an estimated processing time step and / or an estimated processing batch size, wherein participating devices with richer device resources are configured with more computing tasks, and participating devices with less device resources are configured with fewer computing tasks, so that the estimated processing time of each participating device meets the preset time consistency condition; The computing task parameters are sent to each of the participating devices accordingly, so that each of the participating devices can perform the federated learning task according to its own computing task parameters.

2. The method for optimizing a horizontal federated learning system according to claim 1, wherein: The step of respectively configuring the computing task parameters corresponding to each participating device in the federated learning model training process according to the device resource information includes: Classifying each of the participating devices according to the resource information of each of the devices, and determining the resource category to which each of the participating devices belongs; According to the resource category to which each of the participating devices belongs, the computing task parameters in the federated learning model training process corresponding to each of the participating devices are configured respectively.

3. The method for optimizing a horizontal federated learning system according to claim 2, wherein: The step of respectively configuring the computing task parameters in the federated learning model training process corresponding to each participating device according to the resource category to which each participating device belongs includes: Determine candidate task parameters corresponding to each of the participating devices according to the resource category to which each of the participating devices belongs; Determine the estimated processing time corresponding to each of the participating devices based on the candidate task parameters, and detect whether the estimated processing times meet a preset time consistency condition; If the estimated processing times satisfy the preset time consistency condition, the candidate task parameters of each participating device are used as the calculation task parameters of each participating device.

4. The method for optimizing a horizontal federated learning system according to claim 1, wherein: When the model to be trained in horizontal federated learning is a recurrent neural network model, and the computing task parameters include an estimated processing time step, after the step of sending the computing task parameters to each of the participating devices, the step further includes: According to the estimated processing time steps corresponding to the participating devices, respectively, configuring the time step selection strategies corresponding to the participating devices respectively; The time step selection strategy is sent to each of the participating devices accordingly, so that each of the participating devices selects sequence selected data from its own sequence data according to its own time step selection strategy, and performs the federated learning task according to the sequence selected data, wherein the time step of the sequence selected data is less than or equal to the expected processing time step of each of the participating devices.

5. The method for optimizing a horizontal federated learning system according to claim 1, wherein: When the computing task parameters include an estimated processing batch size, after the step of sending the computing task parameters to each of the participating devices, the step further includes: According to the estimated processing batch size corresponding to each of the participating devices, configure the learning rate corresponding to each of the participating devices; The learning rate is sent to each participating device accordingly, so that each participating device performs the federated learning task according to the respective learning rate and the estimated processing batch size received from the coordination device.

6. The method for optimizing a horizontal federated learning system according to claim 1, wherein: The step of sending the computing task parameters to each of the participating devices so that each of the participating devices performs the federated learning task according to the computing task parameters comprises: The computing task parameters are sent to each of the participating devices accordingly, and the estimated duration of this round of global model update is sent to each of the participating devices, so that each of the participating devices can adjust the number of local model trainings according to the estimated duration when performing local model training based on the computing task parameters.

7. The method for optimizing a horizontal federated learning system according to any one of claims 1 to 6, wherein: The step of obtaining device resource information of each participating device participating in horizontal federated learning includes: Receive device resource information sent by each participating device participating in horizontal federated learning, wherein the device resource information includes at least one or more of power resource information, computing resource information, and communication resource information.

8. A horizontal federated learning system optimization device, characterized in that: The device is deployed on a coordination device participating in horizontal federated learning, and the device includes: An acquisition module is used to obtain device resource information of each participating device participating in horizontal federated learning; A configuration module, used to configure the computing task parameters in the federated learning model training process corresponding to each of the participating devices according to the device resource information, wherein the computing task parameters include an estimated processing time step and / or an estimated processing batch size, wherein participating devices with richer device resources are configured with more computing tasks, and participating devices with less device resources are configured with fewer computing tasks, so that the estimated processing time of each participating device meets a preset time consistency condition; The sending module is used to send the computing task parameters to each of the participating devices so that each of the participating devices can perform the federated learning task according to its own computing task parameters.

9. A horizontal federated learning system optimization device, characterized in that: The horizontal federated learning system optimization device includes: a memory, a processor, and a horizontal federated learning system optimization program stored in the memory and executable on the processor. When the horizontal federated learning system optimization program is executed by the processor, the steps of the horizontal federated learning system optimization method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a horizontal federated learning system optimization program, which, when executed by a processor, implements the steps of the horizontal federated learning system optimization method according to any one of claims 1 to 7.

Images