Model self-adaptive issuing method and system under heterogeneous federated learning

Abstract

The invention discloses a model self-adaptive issuing method and system under heterogeneous federated learning, and the method comprises the steps: constructing a dynamic comprehensive evaluation strategy of a multi-dimensional sub-model scale, comprehensively evaluating the sub-model scale from multiple dimensions of calculation, storage, communication, data resources, model scale, calculation amount and the like of a client, obtaining a more precise model cutting rate, and improving the model cutting efficiency. The server is guided to adaptively issue a sub-model of an adaptive resource to the client; for a scene that a server has a public data set, issuing a sub-model to a client with heterogeneous resources through a dynamic sub-model issuing method designed based on technologies such as rolling training; for a scene that a server does not have a public data set, issuing a sub-model for a client with heterogeneous resources through a sub-model issuing strategy designed according to network element importance; according to the method, the global model is high in convergence speed, high in accuracy and good in fluency and grammar correctness.

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning, and specifically relates to a method and system for adaptively issuing models under heterogeneous federated learning. Background Art

[0002] With the deep integration of federated learning technology with application scenarios such as the Internet of Things and satellite computing, clients have evolved from a single type of device to devices with heterogeneous resources, such as sensors, mobile phones, and vehicle-mounted equipment.

[0003] At present, the ability of clients to participate in model training and run models is very different. Resource-constrained clients cannot run global-scale models, cannot participate in full-process training, and are limited in the size of transmitted models. They can only send sub-models derived from the global model from the server. The "equal" sub-model delivery and training method cannot meet the needs of heterogeneous clients. How to adaptively send models that meet different computing, storage, and communication resource conditions to resource-heterogeneous clients in federated learning has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the purpose of the present invention is to propose a method and system for adaptively issuing models under heterogeneous federated learning, so as to solve or partially solve the above-mentioned technical problems.

[0005] Based on the above objectives, the present invention provides a method for adaptively issuing models in heterogeneous federated learning, including:

[0006] Construct a dynamic comprehensive evaluation strategy for multi-dimensional client sub-models, and perform dynamic comprehensive quantitative calculation of the sub-model scale based on the client's computing, storage, communication, data resources, model training time, and number of rounds;

[0007] Construct a dynamic sub-model adaptive delivery strategy in the scenario where the server has no public data set, and deliver the sub-model of the adapted resource to the client based on the rolling training technology; the server sets a rolling window according to the size of the client sub-model, and the rolling window moves forward in the global model, loops through the global model in turn in different rounds, and iterates to make the global model parameters balanced and trained until convergence;

[0008] The server is built with a model pruning strategy based on the importance of network elements in a public data set scenario, and the invalid parameters of the sub-model are reduced based on the model network backbone node identification technology; the contribution of each neuron or channel is calculated according to the public data set, the global model is pruned according to the importance of the network elements and the pruning rate dynamically set by the sub-model scale, and the corresponding sub-model is sent to the client.

[0009] As the preferred solution for the adaptive model delivery method under heterogeneous federated learning, in the process of constructing a dynamic comprehensive evaluation strategy for multi-dimensional client sub-models, the server regularly obtains the resource information owned by the client based on client reports, calling client APIs, remote evaluation systems, and historical data inference, and quantitatively evaluates the scale of the client sub-model from specified dimensions based on the client's resource information.

[0010] As the preferred solution for the model adaptive delivery method under heterogeneous federated learning, the channels through which the server obtains the resource information owned by the client include:

[0011] The client reports computing power, communication capabilities, storage capabilities, device type, and battery status information;

[0012] Call the client API, which includes the system monitoring API and hardware information API provided by the operating system; regularly collect client device resource status according to the remote evaluation system;

[0013] The server collects and stores the client's historical resource data, including model training performance records and task execution status, and infers the client's resource status by analyzing the model training performance records and task execution status.

[0014] As the optimal solution for model adaptive delivery in heterogeneous federated learning, the time t for each round of local calculation of client u is train for:

[0015] t train =(D+1)·c·R·N·FLOPs / FLOPS

[0016] In the formula, FLOPS is the number of floating-point operations per second of the device, D is the ratio of the computing time of the model's back propagation to forward propagation; c is a constant, indicating the tolerance of client u to each round of computing time; R is the number of local training rounds of client u; N is the amount of client data;

[0017] The time t of each round of transmission transmit for:

[0018] t transmit =M s / V comm

[0019] Where M s is the model size, V comm is the communication rate;

[0020] The storage memory stor is:

[0021] storage = RAM u

[0022] In the formula, RAMu is the used storage memory;

[0023] The time that client u participates in each round of training satisfies the constraints:

[0024] t t ra i n +t t ransm it ≤t t o t a l , and M s <stor

[0025] In the formula, t total The maximum time to complete one round of training for the global model;

[0026] The server calculates the sub-model size based on the constraints.

[0027] As a preferred solution for the model adaptive delivery method under heterogeneous federated learning, the rolling window moves forward in the global model, and in the process of traversing the global model in turn in different rounds:

[0028] In the jth round, the high-order sub-model pruned from the global model is {a, b, c, d}, and the low-order sub-model is {c, d, e};

[0029] In the j+1th round, the rolling window moves forward one step, and the high-order sub-model and low-order sub-model become {b, c, d, e} and {d, e, a} respectively;

[0030] In the j+2th round, the rolling window moves forward one step, and the high-order sub-model and the low-order sub-model become {c, d, e, a} and {e, a, b} respectively.

[0031] As a preferred solution for the model adaptive delivery method under heterogeneous federated learning, the contribution of each neuron or channel is calculated based on the public data set, and the global model is pruned according to the importance of network elements and the pruning rate dynamically set by the sub-model scale:

[0032] Use the public dataset to train the global model and run the network on the validation dataset to obtain the activation value of each neuron; prune neurons with zero activation values; match the size of the pruned sub-model with the client resources. If they match, send the current sub-model to the client; if they do not match, retrain the network using the weights before pruning as the initialization values. By alternating pruning and retraining, reduce the zero activation values ​​in the network to obtain a sub-model that meets the model size requirements.

[0033] The present invention also provides a model adaptive delivery system under heterogeneous federated learning, including:

[0034] The sub-model dynamic comprehensive evaluation module is used to build a dynamic comprehensive evaluation strategy for multi-dimensional client sub-models, and dynamically and comprehensively quantify the sub-model scale based on the client's computing, storage, communication, data resources, model training time, and number of rounds;

[0035] The sub-model adaptive delivery module is used to build a dynamic sub-model adaptive delivery strategy in the scenario where the server has no public data set, and delivers the sub-model of the adapted resource to the client based on the rolling training technology; the server sets a rolling window according to the size of the client sub-model, and the rolling window moves forward in the global model, loops through the global model in turn in different rounds, and iterates to make the global model parameters balanced and trained until convergence;

[0036] The model pruning processing module is used to build a model pruning strategy based on the importance of network elements in scenarios where the server has a public data set, and to reduce invalid parameters of sub-models based on the model network backbone node identification technology; the contribution of each neuron or channel is calculated according to the public data set, and the global model is pruned according to the importance of the network elements and the pruning rate dynamically set by the sub-model scale, and the corresponding sub-model is sent to the client.

[0037] As a preferred solution for the model adaptive delivery system under heterogeneous federated learning, in the sub-model dynamic comprehensive evaluation module, the server regularly obtains the resource information owned by the client based on the client report, calling the client API, remote evaluation system and historical data inference, and quantitatively evaluates the client sub-model scale from the specified dimension according to the client's resource information;

[0038] In the sub-model dynamic comprehensive evaluation module, the channels through which the server obtains the resource information owned by the client include:

[0039] The client reports computing power, communication capabilities, storage capabilities, device type, and battery status information;

[0040] Call the client API, which includes the system monitoring API and hardware information API provided by the operating system; regularly collect client device resource status according to the remote evaluation system;

[0041] The server collects and stores the client's historical resource data, including model training performance records and task execution status, and infers the client's resource status by analyzing the model training performance records and task execution status.

[0042] As a preferred solution for the model adaptive delivery system under heterogeneous federated learning, in the sub-model dynamic comprehensive evaluation module, the time t for each round of local calculation of client u istrain for:

[0043] t train =(D+1)·c·R·N·FLOPs / FLOPS

[0044] In the formula, FLOPS is the number of floating-point operations per second of the device, D is the ratio of the computing time of the model's back propagation to forward propagation; c is a constant, indicating the tolerance of client u to each round of computing time; R is the number of local training rounds of client u; N is the amount of client data;

[0045] The time t of each round of transmission transmit for:

[0046] t transmit =M s / V comm

[0047] Where M s is the model size, V comm is the communication rate;

[0048] The storage memory stor is:

[0049] storage = RAM u

[0050] In the formula, RAM u is the used storage memory;

[0051] The time that client u participates in each round of training satisfies the constraints:

[0052] t t ra i n +t t ransm it ≤t t o t a l , and M s <stor

[0053] Where, t total The maximum time to complete one round of training for the global model;

[0054] The server calculates the sub-model size based on the constraints.

[0055] As a preferred solution for the model adaptive delivery system under heterogeneous federated learning, in the sub-model adaptive delivery module:

[0056] In the jth round, the high-order sub-model pruned from the global model is {a, b, c, d}, and the low-order sub-model is {c, d, e};

[0057] In the j+1th round, the rolling window moves forward one step, and the high-order sub-model and low-order sub-model become {b, c, d, e} and {d, e, a} respectively;

[0058] In the j+2th round, the rolling window moves forward one step, and the high-order sub-model and low-order sub-model become {c, d, e, a} and {e, a, b} respectively;

[0059] In the model cutting processing module:

[0060] Use the public dataset to train the global model, and run the network on the validation dataset to obtain the activation value of each neuron; prune neurons with zero activation values; match the size of the pruned sub-model with the client resources. If they match, send the current sub-model to the client; if they do not match, retrain the network using the weights before pruning as the initialization values, and reduce the zero activation values ​​in the network by alternating pruning and retraining to obtain a sub-model that meets the model size requirements.

[0061] From the above, it can be seen that the technical solution provided by the present invention, by constructing a dynamic comprehensive evaluation strategy for the scale of multi-dimensional sub-models, comprehensively evaluates the scale of sub-models from multiple dimensions such as client computing, storage, communication, data resources and model scale, and computational complexity, obtains a more accurate model clipping rate, and guides the server to adaptively send sub-models that adapt to resources to the client; for scenarios where the server has a public data set, a dynamic sub-model sending method designed based on rolling training and other technologies is used to send sub-models to clients with heterogeneous resources; for scenarios where the server has no public data set, a sub-model sending strategy designed based on the importance of network elements is used to send sub-models to clients with heterogeneous resources; the global model in the present invention has fast convergence speed and high accuracy, and has good fluency and grammatical correctness. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 A schematic diagram of a method for adaptively issuing models under heterogeneous federated learning provided by an embodiment of the present invention;

[0064] Figure 2A schematic diagram of a dynamic comprehensive evaluation strategy for a multi-dimensional client sub-model in a method for adaptively issuing a model under heterogeneous federated learning provided in an embodiment of the present invention;

[0065] Figure 3 A schematic diagram of a dynamic sub-model adaptive delivery strategy in a model adaptive delivery method under heterogeneous federated learning provided in an embodiment of the present invention;

[0066] Figure 4 A schematic diagram of a model adaptive delivery strategy based on neuron importance in a model adaptive delivery method under heterogeneous federated learning provided in an embodiment of the present invention;

[0067] Figure 5 A schematic diagram of a model adaptive delivery strategy based on channel importance in a model adaptive delivery method under heterogeneous federated learning provided in an embodiment of the present invention;

[0068] Figure 6 A diagram of the architecture of a model adaptive delivery system for heterogeneous federated learning provided by an embodiment of the present invention;

[0069] Figure 7 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0071] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The words "include" or "comprise" and the like used in the embodiments of the present invention mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, but do not exclude other elements or objects.

[0072] With the deep integration of federated learning technology with application scenarios such as the Internet of Things and satellite computing, clients have evolved from single-type devices to heterogeneous devices, such as sensors, mobile phones, and vehicle-mounted devices. The ability of these clients to participate in model training and run models is very different. Resource-constrained clients cannot run global-scale models, cannot participate in full-process training, and are limited in the size of transmitted models. They can only send sub-models derived from the global model from the server. The "equal treatment" sub-model delivery and training method cannot meet the needs of heterogeneous clients.

[0073] At present, it is difficult for the server to adaptively send a model that is adapted to the client's resources. In the related art, in federated learning with heterogeneous resources, the server prunes the global model along the depth or width of the neural network according to a predefined pruning rate to obtain different sub-models with similar architectures. The predefined sub-model sending method has the following problems: there is a lack of a dynamic comprehensive quantitative evaluation method for the sub-model scale, which cannot effectively guide the server to send a sub-model that is adapted to the current client resources; the parameters of the global model are not evenly trained, and the differences between sub-models at all levels are large, which makes the global model easily affected by client drift; the potential data value of the server is not fully utilized, and the sub-models have many invalid network parameters, resulting in slow convergence and low accuracy of the global model.

[0074] In view of this, in order to realize adaptive delivery of heterogeneous resources to clients in federated learning to meet models with different computing, storage and communication resource conditions, an embodiment of the present invention provides a method and system for adaptive delivery of models in heterogeneous federated learning. The following is the specific content of the embodiment of the present invention.

[0075] See also Figure 1 The embodiment of the present invention provides a method for adaptively issuing a model in heterogeneous federated learning, comprising the following steps:

[0076] Construct a dynamic comprehensive evaluation strategy for multi-dimensional client sub-models, and perform dynamic comprehensive quantitative calculation of the sub-model scale based on the client's computing, storage, communication, data resources, model training time, and number of rounds;

[0077] Construct a dynamic sub-model adaptive delivery strategy in the scenario where the server has no public data set, and deliver the sub-model of the adapted resource to the client based on the rolling training technology; the server sets a rolling window according to the size of the client sub-model, and the rolling window moves forward in the global model, loops through the global model in turn in different rounds, and iterates to make the global model parameters balanced and trained until convergence;

[0078] The server is built with a model pruning strategy based on the importance of network elements in a public data set scenario, and the invalid parameters of the sub-model are reduced based on the model network backbone node identification technology; the contribution of each neuron or channel is calculated according to the public data set, the global model is pruned according to the importance of the network elements and the pruning rate dynamically set by the sub-model scale, and the corresponding sub-model is sent to the client.

[0079] In this embodiment, by constructing a dynamic comprehensive evaluation strategy for the scale of multi-dimensional sub-models, a more accurate model pruning rate is obtained, guiding the server to adaptively send sub-models of adapted resources to the client to make full use of client resources; according to the pruning rate and the condition that the server has / has no data set, a dynamic sub-model sending strategy based on rolling training and other technologies and a sub-model sending strategy based on the importance of network elements such as nodes, channels, filters, and network layers are designed respectively, which not only fully utilizes server data resources but also improves model training efficiency and performance.

[0080] See also Figure 2 ,In this embodiment, in the process of constructing a dynamic ,comprehensive evaluation strategy for a multi-dimensional client sub-model, the server ,regularly obtains the resource information owned by the client based on the ,client report, calling the client API, the remote evaluation system and ,historical data inference, and quantitatively evaluate

[0081] Among them, the channels for the server to obtain the resource information owned by the client include: the client reports computing power, communication capability, storage capability, device type, and battery status information; calling the client API, which includes the system monitoring API and hardware information API provided by the operating system; regularly collecting client device resource status according to the remote evaluation system; the server collects and stores the client's historical resource data, including model training performance records and task execution status, and infers the client's resource status by analyzing model training performance records and task execution status.

[0082] In this embodiment, the time t for each round of local calculation by client u is train for:

[0083] t train =(D+1)·c·R·N·FLOPs / FLOPS

[0084] In the formula, FLOPS is the number of floating-point operations per second of the device, D is the ratio of the computing time of the model's back propagation to forward propagation; c is a constant, indicating the tolerance of client u to each round of computing time; R is the number of local training rounds of client u; N is the amount of client data;

[0085] The time t of each round of transmission transmit for:

[0086] t transmit =M s / V comm

[0087] Where M s is the model size, V comm is the communication rate;

[0088] The storage memory stor is:

[0089] stor = RAM u

[0090] where RAM u is the used storage memory;

[0091] The time for the client u to participate in each round of training satisfies the constraint:

[0092] t train + t transmit ≤ t total and M s < stor

[0093] where t total is the maximum time for the global model to complete one round of training;

[0094] The server calculates the sub - model scale according to the constraint.

[0095] Specifically, the sub - model scale of the client is quantitatively evaluated from multiple dimensions according to the resource information of the client. The client resource metrics include: the number of floating - point operations per second FLOPS of the device, the communication rate V comm , the used memory RAM u , the client data volume N, which represent the client's computing, communication, storage, and data resources respectively. The model information includes the maximum time t total for the global model to complete one round of training, the amount of computation FLOPs and the model size M s required for the sub - model sent to the client u, and the number of local training rounds R of the client u. c is a constant representing the tolerance of the client u for the computing time of each round, and the ratio D of the computing time of the model's backpropagation to forward propagation. The time for the client u to perform local computation in each round is t train =(D + 1)·c·R·N·FLOPs / FLOPS, and the time for each round of transmission is t transmit = M s / V comm and the storage stor = RAM u . The time for the client u to participate in each round of training should satisfy t train + t transmit ≤ t total and M s < stor. The server calculates the sub - model scale according to the above constraints. The corresponding sub - model is obtained using the model pruning method, and this sub - model is sent to the client u.

[0096] In this embodiment, for scenarios where the server does not maintain a public data set, the sub-model is dynamically issued based on rolling training so that each client can evenly train the global model parameters during the entire federated learning training process, alleviating the client drift problem and making the model converge faster. Therefore, for scenarios where the server does not have a public data set, a rolling window is used to issue different sub-models to the same client in different training rounds, thereby solving the problem of slow global model convergence caused by client drift.

[0097] Specifically, the server sets a rolling window according to the size of the client sub-model, and the rolling window moves forward in the global model and loops through all parts of the global model in turn in different rounds.

[0098] See also Figure 3 , the rolling window moves forward in the global model, and traverses the global model in turn in different rounds:

[0099] In the jth round, the high-order sub-model pruned from the global model is {a, b, c, d}, and the low-order sub-model is {c, d, e};

[0100] In the j+1th round, the rolling window moves forward one step, and the high-order sub-model and low-order sub-model become {b, c, d, e} and {d, e, a} respectively;

[0101] In the j+2th round, the rolling window moves forward one step, and the high-order sub-model and the low-order sub-model become {c, d, e, a} and {e, a, b} respectively.

[0102] In this embodiment, for the scenario where the server has a public data set, the contribution of each neuron or channel is calculated based on the public data set, and then the global model is pruned according to the importance of these network elements and the pruning rate dynamically set by the sub-model scale, and a more effective sub-model is issued.

[0103] See also Figure 4 The model pruning strategy based on neuron importance is mainly to delete unnecessary nodes according to the importance of nodes. Among them, the contribution of each neuron or channel is calculated according to the public data set, and the global model is pruned according to the importance of network elements and the pruning rate dynamically set by the sub-model scale:

[0104] Use the public dataset to train the global model, and run the network on the validation dataset to obtain the activation value of each neuron; prune neurons with zero activation values; match the size of the pruned sub-model with the client resources. If they match, send the current sub-model to the client; if they do not match, retrain the network using the weights before pruning as the initialization values, and reduce the zero activation values ​​in the network by alternating pruning and retraining to obtain a sub-model that meets the model size requirements.

[0105] from Figure 4 It can be concluded that for scenarios where the server can maintain a public data set, the pruning strategy based on neuron importance prunes nodes that contribute little to the model, effectively simplifying the model structure of the fully connected neural network, making the scale of the sub-model based on the fully connected neural network fit the client resources to the maximum extent. This method retains important nodes as much as possible in the client sub-model obtained, and the model performance is better.

[0106] See also Figure 5 The model pruning strategy based on channel importance mainly prunes the global model according to the importance of the convolutional layer channels. The main steps are as follows: use a public dataset to train the global model; calculate the contribution of each channel in the convolutional layer, that is, calculate its L1 norm or L2 norm; calculate the number of channels to be pruned based on the model pruning rate and contribution; reconstruct the convolutional layer of the network. Based on the above steps, a sub-model that meets the scale requirements is obtained.

[0107] from Figure 5 It can be concluded that for scenarios where the server can maintain a public data set, the pruning strategy based on channel importance prunes channels that contribute little to the model, effectively simplifying the model structure of the convolutional neural network, making the scale of the sub-model based on the convolutional neural network fit the client resources to the maximum extent. This method retains important channels as much as possible in the client sub-model obtained, and the model performance is better.

[0108] In summary, the embodiments of the present invention construct a dynamic comprehensive evaluation strategy for a multi-dimensional client sub-model, and perform dynamic comprehensive quantitative calculation on the sub-model scale based on the client's computing, storage, communication, data resources, model training time, and number of rounds; construct a dynamic sub-model adaptive delivery strategy in a scenario where the server has no public data set, and deliver a sub-model of adapted resources to the client based on a rolling training technology; the server sets a rolling window according to the scale of the client sub-model, and the rolling window moves forward in the global model, loops through the global model in turn in different rounds, and iterates so that the global model parameters are evenly trained until convergence; construct a model pruning strategy based on the importance of network elements in a scenario where the server has a public data set, and reduce the invalid parameters of the sub-model based on the model network backbone node identification technology; calculate the contribution of each neuron or each channel according to the public data set, prune the global model according to the importance of the network element and the pruning rate dynamically set by the sub-model scale, and deliver the corresponding sub-model to the client. The present invention constructs a dynamic comprehensive evaluation strategy for the scale of multi-dimensional sub-models, comprehensively evaluates the scale of sub-models from multiple dimensions such as client computing, storage, communication, data resources and model scale, and computational complexity, obtains a more accurate model clipping rate, and guides the server to adaptively send sub-models that adapt to resources to the client; for scenarios where the server has a public data set, a dynamic sub-model sending method designed based on rolling training and other technologies is used to send sub-models to clients with heterogeneous resources; for scenarios where the server has no public data set, a sub-model sending strategy designed based on the importance of network elements is used to send sub-models to clients with heterogeneous resources; the global model in the present invention has fast convergence speed, high accuracy, and good fluency and grammatical correctness.

[0109] It should be noted that the method of the embodiment of the present invention can be performed by a single device, such as a computer or a server. The method of this embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present invention, and the multiple devices will interact with each other to complete the described method.

[0110] It should be noted that some embodiments of the present invention are described above. In some cases, the actions or steps recorded can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0111] See also Figure 6Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present invention further provides a model adaptive delivery system under heterogeneous federated learning, including:

[0112] The sub-model dynamic comprehensive evaluation module 100 is used to construct a dynamic comprehensive evaluation strategy for the multi-dimensional client sub-model, and to perform dynamic comprehensive quantitative calculation of the sub-model scale based on the client's computing, storage, communication, data resources, model training time, and number of rounds;

[0113] The sub-model adaptive delivery module 200 is used to construct a dynamic sub-model adaptive delivery strategy in the scenario where the server has no public data set, and deliver the sub-model of the adaptation resource to the client based on the rolling training technology; the server sets a rolling window according to the size of the client sub-model, and the rolling window moves forward in the global model, loops through the global model in turn in different rounds, and iterates to make the global model parameters balanced training until convergence;

[0114] The model pruning processing module 300 is used to construct a model pruning strategy based on the importance of network elements in a scenario where the server has a public data set, and to reduce invalid parameters of the sub-model based on the model network backbone node identification technology; calculate the contribution of each neuron or each channel according to the public data set, prune the global model according to the importance of the network elements and the pruning rate dynamically set by the sub-model scale, and send the corresponding sub-model to the client.

[0115] In this embodiment, in the sub-model dynamic comprehensive evaluation module 100, the server periodically obtains the resource information owned by the client based on the client report, calling the client API, the remote evaluation system and the historical data inference method, and quantitatively evaluates the client sub-model scale from the specified dimension according to the client resource information;

[0116] In the sub-model dynamic comprehensive evaluation module 100, the channels through which the server obtains the resource information owned by the client include:

[0117] The client reports computing power, communication capabilities, storage capabilities, device type, and battery status information;

[0118] Call the client API, which includes the system monitoring API and hardware information API provided by the operating system; regularly collect client device resource status according to the remote evaluation system;

[0119] The server collects and stores the client's historical resource data, including model training performance records and task execution status, and infers the client's resource status by analyzing the model training performance records and task execution status.

[0120] In this embodiment, in the sub-model dynamic comprehensive evaluation module 100, the time t for each round of local calculation by the client u istrain for:

[0121] t train =(D+1)·c·R·N·FLOPs / FLOPS

[0122] In the formula, FLOPS is the number of floating-point operations per second of the device, D is the ratio of the computing time of the model's back propagation to forward propagation; c is a constant, indicating the tolerance of client u to each round of computing time; R is the number of local training rounds of client u; N is the amount of client data;

[0123] The time t of each round of transmission transmit for:

[0124] t transmit =M s / V comm

[0125] Where M s is the model size, V comm is the communication rate;

[0126] The storage memory stor is:

[0127] storage = RAM u

[0128] In the formula, RAM u is the used storage memory;

[0129] The time that client u participates in each round of training satisfies the constraints:

[0130] t train +t transmit ≤t total , and M s <stor

[0131] Where, t total The maximum time to complete one round of training for the global model;

[0132] The server calculates the sub-model size based on the constraints.

[0133] In this embodiment, in the sub-model adaptive sending module 200:

[0134] In the jth round, the high-order sub-model pruned from the global model is {a, b, c, d}, and the low-order sub-model is {c, d, e};

[0135] In the j+1th round, the rolling window moves forward one step, and the high-order sub-model and low-order sub-model become {b, c, d, e} and {d, e, a} respectively;

[0136] In the j+2th round, the rolling window moves forward one step, and the high-order sub-model and low-order sub-model become {c, d, e, a} and {e, a, b} respectively;

[0137] In the model cutting processing module 300:

[0138] Use the public dataset to train the global model, and run the network on the validation dataset to obtain the activation value of each neuron; prune neurons with zero activation values; match the size of the pruned sub-model with the client resources. If they match, send the current sub-model to the client; if they do not match, retrain the network using the weights before pruning as the initialization values, and reduce the zero activation values ​​in the network by alternating pruning and retraining to obtain a sub-model that meets the model size requirements.

[0139] The system of the above embodiment is used to implement a corresponding model adaptive delivery method under heterogeneous federated learning in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0140] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for adaptively delivering the model under heterogeneous federated learning described in any of the above embodiments is implemented.

[0141] Figure 7 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, the memory 420, the input / output interface 430, and the communication interface 440 are connected to each other through the bus 450 in the device.

[0142] The processor 410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0143] The memory 420 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 420 and are called and executed by the processor 410.

[0144] The input / output interface 430 is used to connect the input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0145] The communication interface 440 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0146] The bus 450 comprises a pathway for transmitting information between the various components of the device (eg, the processor 410, the memory 420, the input / output interface 430, and the communication interface 440).

[0147] It should be noted that, although the above device only shows the processor 410, the memory 420, the input / output interface 430, the communication interface 440 and the bus 450, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0148] The electronic device of the above embodiment is used to implement a corresponding model adaptive delivery method under heterogeneous federated learning in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0149] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present invention also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute a model adaptive delivery method under heterogeneous federated learning as described in any of the above embodiments.

[0150] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0151] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute a model adaptive delivery method under heterogeneous federated learning as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0152] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention is limited to these examples. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of the embodiments of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0153] In addition, to simplify the description and discussion, and in order not to obscure the embodiments of the present invention, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the system may be shown in the form of a block diagram so as to avoid obscuring the embodiments of the present invention, and this also takes into account the fact that the details of the implementation of these block diagram systems are highly dependent on the platform on which the embodiments of the present invention will be implemented (i.e., these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present invention, it will be apparent to those skilled in the art that embodiments of the present invention may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0154] Although the present invention has been described in conjunction with specific embodiments of the present invention, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0155] The embodiments of the present invention are intended to cover all such substitutions, modifications and variations that fall within the scope of the protection claimed. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the present invention.

Claims

1. A model adaptive delivery method for heterogeneous federated learning, wherein: include: Construct a dynamic comprehensive evaluation strategy for multi-dimensional client sub-models, and perform dynamic comprehensive quantitative calculation of the sub-model scale based on the client's computing, storage, communication, data resources, model training time, and number of rounds; Construct a dynamic sub-model adaptive delivery strategy in scenarios where the server has no public data set, and deliver resource-adaptive sub-models to the client based on rolling training technology; The server sets a rolling window according to the size of the client sub-model, and the rolling window moves forward in the global model, loops through the global model in turn in different rounds, and iterates so that the global model parameters are balanced and trained until convergence; The server is built with a model pruning strategy based on the importance of network elements in a public data set scenario, and the invalid parameters of the sub-model are reduced based on the model network backbone node identification technology; the contribution of each neuron or channel is calculated according to the public data set, the global model is pruned according to the importance of the network elements and the pruning rate dynamically set by the sub-model scale, and the corresponding sub-model is sent to the client.

2. According to the method for adaptively issuing models in heterogeneous federated learning according to claim 1, In the process of building a dynamic comprehensive evaluation strategy for a multi-dimensional client sub-model, the server regularly obtains the resource information owned by the client based on client reports, calling the client API, the remote evaluation system, and historical data inference, and quantitatively evaluates the size of the client sub-model from the specified dimension based on the client's resource information.

3. According to the method for adaptively issuing models in heterogeneous federated learning according to claim 2, wherein: The channels through which the server obtains resource information owned by the client include: The client reports computing power, communication capabilities, storage capabilities, device type, and battery status information; Call the client API, which includes the system monitoring API and hardware information API provided by the operating system; regularly collect client device resource status according to the remote evaluation system; The server collects and stores the client's historical resource data, including model training performance records and task execution status, and infers the client's resource status by analyzing the model training performance records and task execution status.

4. According to the method for adaptively issuing models in heterogeneous federated learning according to claim 2, The time t of each round of local calculation of client u train for: t train =(D+1)·c·R·N·FLOPs / FLOPS In the formula, FLOPS is the number of floating-point operations per second of the device, and D is the ratio of the computing time of the model's back propagation to forward propagation; c is a constant, which indicates the tolerance of client u to the computation time of each round; R is the number of local training rounds of client u; N is the amount of client data; The time t of each round of transmission transmit for: t transmit =M s / V comm Where M s is the model size, V comm is the communication rate; The storage memory stor is: large=RAM u In the formula, RAM u is the used storage memory; The time that client u participates in each round of training satisfies the constraints: t train +t transmit ≤t total , and M s <stor Where, t total The maximum time to complete one round of training for the global model; The server calculates the sub-model size based on the constraints.

5. According to the method for adaptively issuing models in heterogeneous federated learning according to claim 2, wherein: The rolling window moves forward in the global model, and traverses the global model in turn in different rounds: In the jth round, the high-order sub-model pruned from the global model is {a, b, c, d}, and the low-order sub-model is {c, d, e}; In the j+1th round, the rolling window moves forward one step, and the high-order sub-model and low-order sub-model become {b, c, d, e} and {d, e, a} respectively; In the j+2th round, the rolling window moves forward one step, and the high-order sub-model and the low-order sub-model become {c, d, e, a} and {e, a, b} respectively.

6. According to the method for adaptively issuing models in heterogeneous federated learning according to claim 1, The contribution of each neuron or channel is calculated based on the public data set, and the global model is pruned according to the importance of the network elements and the pruning rate dynamically set by the size of the sub-model: Use the public dataset to train the global model and run the network on the validation dataset to obtain the activation value of each neuron; prune neurons with zero activation value; match the size of the pruned sub-model with the client resources, and if they match, send the current sub-model to the client; If there is a mismatch, the network is retrained using the weights before pruning as initialization values. By alternating pruning and retraining, the zero activation values ​​in the network are reduced to obtain a sub-model that meets the model size requirements.

7. A model adaptive delivery system under heterogeneous federated learning, wherein: include: The sub-model dynamic comprehensive evaluation module is used to build a dynamic comprehensive evaluation strategy for multi-dimensional client sub-models, and dynamically and comprehensively quantify the sub-model scale based on the client's computing, storage, communication, data resources, model training time, and number of rounds; The sub-model adaptive delivery module is used to build a dynamic sub-model adaptive delivery strategy in scenarios where the server has no public data set, and delivers the sub-model of the adapted resource to the client based on the rolling training technology; The server sets a rolling window according to the size of the client sub-model, and the rolling window moves forward in the global model, loops through the global model in turn in different rounds, and iterates so that the global model parameters are balanced and trained until convergence; The model pruning processing module is used to build a model pruning strategy based on the importance of network elements in scenarios where the server has a public data set, and to reduce invalid parameters of sub-models based on the model network backbone node identification technology; the contribution of each neuron or channel is calculated according to the public data set, and the global model is pruned according to the importance of the network elements and the pruning rate dynamically set by the sub-model scale, and the corresponding sub-model is sent to the client.

8. The model adaptive delivery system for heterogeneous federated learning according to claim 7, wherein: In the sub-model dynamic comprehensive evaluation module, the server regularly obtains the resource information owned by the client based on the client report, calling the client API, remote evaluation system and historical data inference, and quantitatively evaluates the client sub-model scale from the specified dimension according to the client resource information; In the sub-model dynamic comprehensive evaluation module, the channels through which the server obtains the resource information owned by the client include: The client reports computing power, communication capabilities, storage capabilities, device type, and battery status information; Call the client API, which includes the system monitoring API and hardware information API provided by the operating system; regularly collect client device resource status according to the remote evaluation system; The server collects and stores the client's historical resource data, including model training performance records and task execution status, and infers the client's resource status by analyzing the model training performance records and task execution status.

9. The model adaptive delivery system under heterogeneous federated learning according to claim 7, wherein: In the sub-model dynamic comprehensive evaluation module, the time t for each round of local calculation by client u is train for: t train =(D+1)·c·R·N·FLOPs / FLOPS In the formula, FLOPS is the number of floating-point operations per second of the device, and D is the ratio of the computing time of the model's back propagation to forward propagation; c is a constant, which indicates the tolerance of client u to the computation time of each round; R is the number of local training rounds of client u; N is the amount of client data; The time t of each round of transmission transmit for: t transmit =M s / V comm Where M s is the model size, V comm is the communication rate; The storage memory stor is: large=RAM u In the formula, RAM u is the used storage memory; The time that client u participates in each round of training satisfies the constraints: t train +t transmit ≤t total , and M s <stor Where, t total The maximum time to complete one round of training for the global model; The server calculates the sub-model size based on the constraints.

10. The model adaptive delivery system under heterogeneous federated learning according to claim 7, wherein: In the sub-model adaptive delivery module: In the jth round, the high-order sub-model pruned from the global model is {a, b, c, d}, and the low-order sub-model is {c, d, e}; In the j+1th round, the rolling window moves forward one step, and the high-order sub-model and low-order sub-model become {b, c, d, e} and {d, e, a} respectively; In the j+2th round, the rolling window moves forward one step, and the high-order sub-model and low-order sub-model become {c, d, e, a} and {e, a, b} respectively; In the model cutting processing module: Use the public dataset to train the global model and run the network on the validation dataset to obtain the activation value of each neuron; prune neurons with zero activation value; match the size of the pruned sub-model with the client resources, and if they match, send the current sub-model to the client; If there is a mismatch, the network is retrained using the weights before pruning as initialization values. By alternating pruning and retraining, the zero activation values ​​in the network are reduced to obtain a sub-model that meets the model size requirements.

Images