Construction method and device of local model based on federated learning, and related equipment

Abstract

The application provides a method and device for constructing a local model based on federated learning and related equipment, the method comprising iteratively training a teacher model using each client until the teacher model converges; training a second student model using each client to obtain a converged second student model and sending the parameters of the converged second student model to a server; performing parameter replacement on an aggregated model using the server, iteratively training the parameter-replaced aggregated model until the parameter-replaced aggregated model converges; training a first student model using the server to obtain a converged first student model and sending the first student model to each client; and adjusting the parameters of the converged first student model using each client, taking the adjusted first student model as a local model of each client, thereby solving the technical problem of excessively high communication overhead between the server and the clients in the federated learning process in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of federated learning, and particularly relates to a method and device for constructing a local model based on federated learning and related equipment. BACKGROUND

[0002] The basic learning process of federated learning is as follows: each client trains a local model according to a local data set, uploads parameters of the local model to a server end, the server end trains a global model using the parameters of each local model, and sends parameters of the global model to each client, and the above process is iterated until the global model converges. In the above learning process, the client and the server end need to communicate for multiple rounds to obtain a converged model. There are many parameters in the global model and each local model, and in the process of communication between the client and the server end, the communication overhead is high due to the transmission of all parameters of the global model and all parameters of each local model. Frequent communication between the client and the server end also causes the problem of high communication overhead. SUMMARY

[0003] Therefore, the present application aims to provide a method and device for constructing a local model based on federated learning and related equipment to overcome all or part of the deficiencies in the prior art.

[0004] To achieve the above object, the present application provides a method for constructing a local model based on a federated learning, applied to a federated learning system, wherein the federated learning system comprises a server end and a plurality of client ends, the server end is deployed with an aggregated model and a first student model, and a teacher model and a second student model are respectively deployed in each client end, the aggregated model is obtained by aggregating all the second student models; the method comprises the following steps: based on a local data set of each client end, each client end is used to iteratively train the teacher model until the teacher model converges; based on the converged teacher model of each client end, each client end is used to train the second student model to obtain a converged second student model, and the parameters of the converged second student model are sent to the server end; based on the parameters of all the converged second student models, the server end is used to replace the parameters of the aggregated model, based on a data set of the server end, the server end is used to iteratively train the aggregated model with the replaced parameters until the aggregated model with the replaced parameters converges; based on the converged aggregated model, the server end is used to train the first student model to obtain a converged first student model, and the converged first student model is sent to each client end; based on the converged first student model and the local data set of each client end, each client end is used to adjust the parameters of the converged first student model, and the adjusted first student model is used as a local model of each client end.

[0005] Optionally, the number of neural network layers in each second student model is the same; and the method for deploying the aggregated model in the server end comprises the following steps: the server end is used to obtain construction information of all the second student models; based on the construction information of all the second student models, the server end is used to construct all the second student models; and the server end is used to sequentially splice the corresponding neural network layers in each second student model to obtain the aggregated model and deploy the aggregated model in the server end.

[0006] Optionally, based on the converged teacher model of each client end, each client end is used to train the second student model to obtain a converged second student model, which comprises the following steps: the converged teacher model is used to output probability distribution information; based on the probability distribution information of the converged teacher model, each client end is used to train the second student model; and in response to determining that the difference between the probability distribution information of the converged teacher model and the probability distribution information output by the second student model is less than a first preset value, the converged second student model is determined to be obtained.

[0007] Optionally, each second student model comprises a feature extractor, the number of channels of a corresponding neural network layer in the feature extractor of each second student model is the same, the channels comprise input channels and output channels, the number of output channels in each neural network layer in the feature extractor of the aggregated model is equal to the sum of the number of output channels of the corresponding neural network layers of all second student models; the iterative training of the aggregated model with the parameter replacement on the server side based on the server-side dataset comprises: for each round of training in the iterative training, performing the following operations: for each neural network layer in the feature extractor of the aggregated model, performing the following operations: calculating the input data received by the input channels of the current neural network layer to obtain the output data of the output channels of the current neural network layer; determining the target output data of the current neural network layer based on the output data and the number of output channels of the neural network layers of the second student model spliced with the current neural network layer, and inputting the target output data into the input channels of the next neural network layer of the current neural network layer.

[0008] Optionally, the determining the target output data of the current neural network layer based on the output data and the number of output channels of the neural network layers of the second student model spliced with the current neural network layer comprises: determining the attention weights of all output data in the current neural network layer based on the output data; sorting all the attention weights in descending order, and taking the output data corresponding to the first N attention weights as the target output data, wherein the value of N is equal to the number of output channels of the neural network layers of the second student model spliced with the current neural network layer.

[0009] Optionally, the training of the first student model on the server side based on the converged aggregated model comprises: obtaining the probability distribution information output by the converged aggregated model using the server side; training the first student model on the server side based on the probability distribution information of the converged aggregated model; and determining that the converged first student model is obtained in response to determining that the difference between the probability distribution information of the converged aggregated model and the probability distribution information output by the first student model is less than a second preset value.

[0010] Optionally, the parameter adjustment on the converged first student model by each client comprises: obtaining a local data set of each client; performing a preprocessing operation on the local data set of each client; dividing the local data set of each client after the preprocessing operation into a training set and a test set; performing parameter adjustment on the converged first student model by using the training set; testing the first student model after the parameter adjustment based on the test set to obtain a test evaluation parameter, and when the test evaluation parameter is greater than a preset evaluation threshold, obtaining the first student model after the parameter adjustment is completed.

[0011] Based on the same inventive concept, the application further provides a local model construction device based on federated learning, applied to a federated learning system, wherein the federated learning system comprises a server end and a plurality of clients, an aggregation model and a first student model are deployed on the server end, a teacher model and a second student model are respectively deployed in each client, and the aggregation model is obtained by aggregating all second student models; the device comprises: a first training module configured to perform iterative training on the teacher model by each client based on a local data set of each client until the teacher model converges; a second training module configured to train the second student model by each client based on the converged teacher model of each client to obtain a converged second student model, and send parameters of the converged second student model to the server end; a parameter replacement module configured to replace parameters of the aggregation model by the server end based on parameters of all converged second student models, perform iterative training on the aggregation model by the server end based on a data set of the server end until the aggregation model after the parameter replacement converges; a third training module configured to train the first student model by the server end based on the converged aggregation model to obtain a converged first student model, and send the converged first student model to each client; and an adjustment module configured to perform parameter adjustment on the converged first student model by each client based on the converged first student model and the local data set of each client, and take the first student model after the adjustment as a local model of each client.

[0012] Based on the same inventive concept, the application further provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0013] Based on the same inventive concept, the application further provides a non-transitory computer readable storage medium storing computer instructions for causing a computer to execute the method described above.

[0014] From the above, it can be seen that the method, device and related equipment for constructing a local model based on federated learning provided by the present application, by using each client to iteratively train the teacher model based on the local data set of each client until the teacher model converges, retains the data feature information of the local data set, and obtains a teacher model with optimal prediction accuracy. Based on the converged teacher model of each client, each client trains the second student model to obtain a converged second student model, and sends the parameters of the converged second student model to the server end, reducing the number of parameters transmitted by the client to the server end and reducing the communication overhead between the server end and the client. Based on the parameters of all converged second student models, the server end replaces the parameters of the aggregated model, and based on the data set of the server end, the server end iteratively trains the aggregated model after parameter replacement until the aggregated model after parameter replacement converges, to a certain extent. The heterogeneous problem of the local data set of different clients is solved, and the aggregated model achieves the expected prediction effect. Based on the converged aggregated model, the server end trains the first student model to obtain a converged first student model, and sends the converged first student model to each client, reducing the number of parameters returned by the server end to the client and further reducing the communication overhead between the server end and the client. Based on the converged first student model and the local data set of each client, each client adjusts the parameters of the converged first student model, and the adjusted first student model is used as the local model of each client. The server end and the client only communicate once, avoiding iterative parameter transfer between the server end and the client in federated learning, improving the communication efficiency between the client and the server end, and reducing the communication overhead between the server end and the client. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present application or related art, the drawings needed to be used in the embodiments or related art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 The flowchart of the method for constructing a local model based on federated learning according to an embodiment of the present application is shown in the figure.

[0017] Figure 2 The structure diagram of the device for constructing a local model based on federated learning according to an embodiment of the present application is shown in the figure.

[0018] Figure 3 Figure 1 is a schematic diagram of an electronic device hardware structure according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to specific embodiments and drawings.

[0020] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the present application should be understood as their common meanings to those skilled in the art to which the present application pertains. The terms "first", "second", and similar terms used in the embodiments of the present application do not denote any order, quantity, or importance, but are merely used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" and "connected" and similar terms do not mean physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "upper", "lower", "left", "right", and the like are merely used to indicate relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships can also change accordingly.

[0021] As described in the background section, in the learning process of federated learning, the client sends the local model parameters to the server, the global model of the server is trained based on the local model parameters of the client, and the parameters of the global model are sent to each client. The above process is iterated until the model converges. In the above process, the client and the server need to communicate for multiple rounds to obtain a converged model. However, in the communication process between the client and the server, a large amount of parameters is transmitted between the client and the server, and multiple rounds of communication are performed between the client and the server, resulting in high communication overhead of the client and the server.

[0022] Therefore, an embodiment of the present application proposes a method for constructing a local model based on federated learning, which is applied to a federated learning system. The federated learning system includes a server and a plurality of clients. An aggregation model and a first student model are deployed on the server. A teacher model and a second student model are deployed in each client. The aggregation model is obtained by aggregating all second student models. Referring to Figure 1 , the method includes the following steps:

[0023] Step 101: Based on the local data set of each client, each client iteratively trains the teacher model until the teacher model converges.

[0024] In this step, the local data set of each client is different, and each client does not need to share the local data set with other clients, thereby protecting the privacy of the local data set of each client, wherein the source of the local data set can be a public data set or a private data set of the client. According to the data characteristics and data amount size of the local data set of each client, a suitable teacher model is deployed at each client. The teacher model is an abstract representation of the data characteristics of the local data set, and in the case that the data amount of the local data set is large, a relatively complex teacher model can be deployed at each client to prevent the teacher model from underfitting; in the case that the data amount of the local data set is small, a relatively simple teacher model can be deployed at each client to prevent the teacher model from overfitting, thereby improving the generalization ability of the teacher model of each client. The teacher model of different clients is related to the local data set of each client, and different teacher models can be deployed on different clients. Based on the local data set of each client, each client trains the teacher model for multiple rounds until it converges, thereby retaining the data characteristic information of the local data set and obtaining a teacher model with optimal prediction accuracy.

[0025] In step 102, based on the converged teacher model of each client, each client trains the second student model to obtain a converged second student model, and sends the parameters of the converged second student model to the server end.

[0026] In this step, the converged teacher model of each client has the optimal prediction accuracy. The knowledge learned by the teacher model to determine the optimal prediction accuracy is migrated to the second student model by each client, the training of the second student model is realized to obtain the converged second student model. That is, the converged second student model is obtained by distilling the converged teacher model, and the compression of the teacher model is completed. The converged second student model has the advantage of convenient deployment, also retains the prediction performance of the converged teacher model, and the parameter quantity of the second student model of each client is less than that of the teacher model. Wherein, distillation (Knowledge Distillation) is to transfer the knowledge learned by a complex model to a simple model, the training process of the simple model is supervised by the complex model, so that the prediction accuracy of the simple model approaches that of the complex model, the complex model is called the teacher model, and the simple model is called the student model. It should be noted that the second student model of each client needs to meet a specific structure, including but not limited to: the number of layers of the model between the second student models of each client is consistent, the input dimension and the output dimension of each layer corresponding between the second student models of each client are consistent, etc. The parameters of the converged second student model of each client are sent to the server end by each client, and the above sending process can be sent by using a network or a fixed communication line, etc. By sending the parameters of the converged second student model to the server end, the number of parameters transmitted from the client to the server end is reduced, and the communication overhead of the server end and the client is reduced.

[0027] In step 103, based on the parameters of all converged second student models, the server end replaces the parameters of the aggregation model, and based on the data set of the server end, the server end iteratively trains the aggregation model after the parameter replacement until the aggregation model after the parameter replacement converges.

[0028] In this step, the aggregation model is composed of all the second student models, and each layer in the aggregation model contains the model architecture, parameters, etc. of the corresponding layer of all the second student models, and the second student models have no direct relationship with each other and do not affect each other. After receiving all the converged second student model parameters, the parameters of the corresponding second student model in the aggregation model are replaced. It should be noted that each layer of different second student models can be regarded as a block encapsulated into the aggregation model on the server side. After feature extraction in different blocks at each layer in the aggregation model on the server side, the output with the same dimension but different values is obtained. During the training of the aggregation model, the parameter values of the convolutional layers in all encapsulated blocks are unchanged, so that different blocks can still focus on the data feature information of the training task when facing different data inputs, to a certain extent, solving the heterogeneous problem of different local data sets of different clients. By retaining all the convolutional layer parameter values of the second student model of the client, the aggregation model can better obtain more data information of the client, and has better aggregation effect. The data set on the server side can be a public data set, a server private data set, or a data set generated by the client through the generation model. The aggregation model with parameter replacement is iteratively trained by the server side using the data set on the server side until the aggregation model converges. The parameters of the aggregation model are trained and adjusted to make the aggregation model achieve the expected prediction effect.

[0029] In step 104, based on the converged aggregation model, the first student model is trained by the server side to obtain a converged first student model, and the converged first student model is sent to each client.

[0030] In this step, the aggregation model has the expected prediction effect, and the knowledge learned by the aggregation model to achieve the expected prediction effect is migrated to the first student model by the server side to obtain a converged first student model, that is, the converged first student model is obtained by distilling the converged aggregation model, and the compression of the converged aggregation model is completed. The converged first student model has the advantage of convenient deployment and also retains the prediction performance of the converged aggregation model, and the number of parameters of the first student model is less than that of the aggregation model. The selection of the first student model can be adjusted according to the size of the data set on the server side, the communication compression demand, etc. The distillation effect and the parameter reduction amount should be considered. The process of sending the first learning model can use the network for sending or use the fixed communication line for sending, etc. By sending the converged first student model to the client, the number of parameters returned by the server side to the client is reduced, and the communication overhead between the server side and the client is reduced.

[0031] In step 105, based on the converged first student model and the local data set of each client, each client adjusts the parameters of the converged first student model, and the adjusted first student model is used as the local model of each client.

[0032] In this step, after the server-side converged first student model is transmitted to each client, each client needs to retrain the converged first student model based on the local data set. The parameters of the converged first student model are adjusted based on the local data set of each client until the converged first student model converges again. The problem of heterogeneity of local data sets of different clients and the problem of heterogeneity of local models of different clients are solved. Since the local data sets of different clients are different, the converged first student model after parameter adjustment is also different. The server and the client only communicate for one round, avoiding iterative parameter transmission between the server and the client in federated learning, improving the communication efficiency between the client and the server, and reducing the communication overhead between the server and the client.

[0033] Through the above scheme, based on the local data set of each client, each client iteratively trains the teacher model until the teacher model converges, preserving the data feature information of the local data set and obtaining a teacher model with optimal prediction accuracy. Based on the converged teacher model of each client, each client trains the second student model to obtain a converged second student model, and sends the parameters of the converged second student model to the server, reducing the number of parameters transmitted by the client to the server and reducing the communication overhead between the server and the client. Based on the parameters of all converged second student models, the server replaces the parameters of the aggregated model, and based on the data set of the server, the server iteratively trains the parameter-replaced aggregated model until the parameter-replaced aggregated model converges, to some extent solving the problem of heterogeneity of local data sets of different clients and making the aggregated model achieve the expected prediction effect. Based on the converged aggregated model, the server trains the first student model to obtain a converged first student model, and sends the converged first student model to each client, reducing the number of parameters returned by the server to the client and further reducing the communication overhead between the server and the client. Based on the converged first student model and the local data set of each client, each client adjusts the parameters of the converged first student model, and the adjusted first student model is used as the local model of each client. The server and the client only communicate for one round, avoiding iterative parameter transmission between the server and the client in federated learning, improving the communication efficiency between the client and the server, and reducing the communication overhead between the server and the client.

[0034] In some embodiments, the number of neural network layers in each second student model is the same; the method for deploying the aggregated model on the server side comprises: obtaining, by the server side, construction information of all second student models; constructing, by the server side, all second student models based on the construction information of all second student models; and sequentially splicing, by the server side, the corresponding neural network layers in each second student model to obtain the aggregated model and deploy the aggregated model on the server side.

[0035] In this embodiment, the server side constructs all second models after obtaining the construction information of all second models. Each neural network layer of the aggregated model is formed by splicing the corresponding neural network layers of all second student models, constituting an aggregated network. The aggregated model on the server side is formed by stacking the neural network layers of all second student models. It should be noted that the neural network layers in the aggregated model on the server side can be spliced by the corresponding neural network layers of all second student models, and the neural network layers of the aggregated model can also be increased according to the specific needs of the user. For example, in order to better adapt to the specific needs of the user, a neural network layer for realizing the specific needs of the user can be added to the aggregated model. In the model deployment stage on the server side, the construction information of the second student model is obtained by the server side, and the aggregated model is deployed on the server side. During the communication process between the client and the server side, the architecture and other information of the second student model do not need to be transmitted, only the parameters of the converged second student model need to be transmitted, thereby reducing the data transmission amount during the communication process between the client and the server side, and further reducing the communication overhead.

[0036] In some embodiments, based on the converged teacher model of each client, the second student model is trained by each client to obtain a converged second student model, comprising: obtaining, by each client, probability distribution information output by the converged teacher model; training, by each client, the second student model based on the probability distribution information of the converged teacher model; and in response to determining that the difference between the probability distribution information of the converged teacher model and the probability distribution information output by the second student model is less than a first preset value, determining that the converged second student model is obtained.

[0037] In the embodiment, the converged teacher model has the optimal prediction accuracy and can output the optimal probability distribution information. The parameters of the converged teacher model remain unchanged in the training process of the second student model, have a supervision effect on the second student model, and achieve the purpose of training the second student model. The first preset value is determined according to historical experimental values. In the case where the difference between the probability distribution information of the converged teacher model and the probability distribution information output by the second student model is less than the first preset value, it is indicated that the prediction accuracy of the second student model approaches the prediction accuracy of the converged teacher model, and the converged second student model is determined. The converged teacher model is compressed, so that the number of parameters sent to the server end is reduced in the case where the second student model has the prediction accuracy of the converged teacher model, and the communication consumption between the client and the server end is reduced.

[0038] In some embodiments, each second student model includes a feature extractor, the number of channels of a corresponding neural network layer in the feature extractor of each second student model is the same, the channels include input channels and output channels, the number of output channels in each neural network layer of the feature extractor of the aggregated model is equal to the sum of the output channel numbers of the corresponding neural network layers of all second student models; the iterative training of the aggregated model with the parameter replacement on the server end based on the data set of the server end includes: for each round of training in the iterative training, performing the following operations: for each neural network layer in the feature extractor of the aggregated model, performing the following operations: calculating the input data received by the input channels of the current neural network layer to obtain the output data of the output channels of the current neural network layer; determining the target output data of the current neural network layer based on the output data and the number of output channels of the neural network layer of the second student model spliced with the current neural network layer, and inputting the target output data into the input channels of the next neural network layer of the current neural network layer.

[0039] In the embodiment, the feature extractor of the aggregated model is composed of the feature extractors of all the second student models, and the number of output channels in each neural network layer of the feature extractor of the aggregated model is equal to the sum of the output channel numbers of the corresponding neural network layers of the feature extractors of all the second student models. However, in the case that the output channels in each neural network layer of the feature extractor of the aggregated model all output data, the amount of data output by the output channels of the neural network layer is too large, thereby affecting the training efficiency of the aggregated model. The input data received by the input channels of the current neural network layer is calculated to obtain the output data of the output channels. The output data is filtered to obtain target output data, wherein the number of the target output data is the same as the number of the output channels of the neural network layer of the second student model used for splicing the current neural network layer. Through the filtering of the output data of each neural network layer, the number of the output data in the current neural network layer is consistent with the number of the input channels of the next neural network layer, thereby realizing the reduction of the amount of data and improving the training efficiency of the aggregated model.

[0040] In some embodiments, the determining of the target output data of the current neural network layer based on the output data and the number of the output channels of the neural network layer of the second student model used for splicing the current neural network layer comprises: determining the attention weights of all the output data in the current neural network layer based on the output data; and sorting all the attention weights in descending order, and taking the output data corresponding to the first N attention weights as the target output data, wherein the value of N is equal to the number of the output channels of the neural network layer of the second student model used for splicing the current neural network layer.

[0041] In the embodiment, each neural network layer of the feature extractor of the aggregated model includes the channels of the neural network layers corresponding to the feature extractors of all the second student models. In the case that all the output channels output data to the input channels of the next neural network layer, the amount of data is too large, thereby affecting the convergence speed of the aggregated model. The output data needs to be filtered according to the attention mechanism of each neural network layer, so that the number of the output data in each neural network layer is consistent with the number of the input channels of the next neural network layer. The channels corresponding to the output data with larger attention weights are selected as the output data channels, and the channels corresponding to the output data with smaller attention weights are discarded, which is beneficial to improve the training efficiency of the aggregated model. It should be noted that the method for filtering the output data in the present application is not limited to the attention mechanism method, and other methods for filtering the output data in the neural network model can also be used.

[0042] In some embodiments, the convergent aggregation model is used to train the first student model on the server side to obtain a convergent first student model, including: obtaining probability distribution information output by the convergent aggregation model using the server side; training the first student model using the server side based on the probability distribution information of the convergent aggregation model; and determining that the convergent first student model is obtained in response to determining that a difference between the probability distribution information of the convergent aggregation model and probability distribution information output by the first student model is less than a second preset value.

[0043] In this embodiment, the convergent aggregation model has the optimal prediction effect and can output the optimal probability distribution information. The convergent aggregation model parameters remain unchanged during training and have a supervisory effect on the first student model, achieving the purpose of training the first student model. The second preset value is determined according to historical experimental values. In the case where the difference between the probability distribution information of the convergent aggregation model and the probability distribution information output by the first student model is less than the second preset value, it is indicated that the prediction accuracy of the first student model approaches the prediction accuracy of the convergent aggregation model, and the convergent first student model is determined. The convergent aggregation model is compressed, so that the number of parameters sent to each client is reduced when the first student model has the prediction effect of the convergent aggregation model, and the communication consumption between the server side and the client is reduced.

[0044] In some embodiments, the convergent first student model is adjusted in parameters using each client, including: obtaining a local data set of each client; performing a preprocessing operation on the local data set of each client; dividing the local data set of each client after the preprocessing operation into a training set and a test set; adjusting the parameters of the convergent first student model using the training set; testing the convergent first student model after the parameter adjustment based on the test set to obtain a test evaluation parameter, and obtaining the first student model after the parameter adjustment when the test evaluation parameter is greater than a preset evaluation threshold.

[0045] In this embodiment, the local data set of each client is obtained. The local data set of each client is processed to meet the input requirements of the convergent first student model. The local data set of each client after processing can be divided into a 80% training set and a 20% test set. The training set is used to adjust the parameters of the convergent first student model, and the test set is used to verify the convergent first student model after the parameter adjustment. The test evaluation parameter is obtained by adjusting the parameters of the convergent first student model using the training set and testing the convergent first student model after the parameter adjustment using the test set. When the test evaluation parameter is greater than the preset evaluation threshold, the first student model after the parameter adjustment is finally obtained, and the detection accuracy of the first student model is improved.

[0046] It should be noted that the method of the embodiments of the present application can be executed by a single device, such as a computer or a server, etc. The method of the embodiments can also be applied to a distributed scenario, and be completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present application, and the multiple devices can interact with each other to complete the method.

[0047] It should be noted that some embodiments of the present application have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0048] Based on the same inventive concept, the present application also provides a local model construction device based on federated learning, corresponding to the method of any of the above embodiments.

[0049] Reference Figure 2 , the local model construction device based on federated learning is applied to a federated learning system, the federated learning system includes a server end and a plurality of client ends, an aggregation model and a first student model are deployed on the server end, a teacher model and a second student model are respectively deployed in each client end, the aggregation model is obtained by aggregating all second student models, and includes:

[0050] The first training module 10 is configured to perform iterative training on the teacher model by each client end based on the local data set of each client end until the teacher model converges.

[0051] The second training module 20 is configured to train the second student model by each client end based on the converged teacher model of each client end to obtain a converged second student model, and send the parameters of the converged second student model to the server end.

[0052] The parameter replacement module 30 is configured to replace the parameters of the aggregation model by the server end based on the parameters of all converged second student models, and perform iterative training on the aggregation model by the server end based on the data set of the server end until the aggregation model after the parameter replacement converges.

[0053] The third training module 40 is configured to train the first student model on the server side based on the converged aggregation model to obtain a converged first student model, and send the converged first student model to each client.

[0054] The adjustment module 50 is configured to adjust the parameters of the converged first student model on each client based on the converged first student model and the local data set of each client, and take the adjusted first student model as the local model of each client.

[0055] Through the above device, the teacher model is iteratively trained on each client based on the local data set of each client until the teacher model converges, the data feature information of the local data set is retained, and the teacher model with optimal prediction accuracy is obtained. The second student model is trained on each client based on the converged teacher model of each client to obtain a converged second student model, and the parameters of the converged second student model are sent to the server side, which reduces the number of parameters transmitted by the client to the server side and reduces the communication overhead between the server side and the client. The parameters of all converged second student models are replaced by the server side, the aggregation model after parameter replacement is iteratively trained on the server side based on the data set of the server side until the aggregation model after parameter replacement converges, which to some extent solves the heterogeneous problem of local data sets of different clients and makes the aggregation model achieve the expected prediction effect. The first student model is trained on the server side based on the converged aggregation model to obtain a converged first student model, and the converged first student model is sent to each client, which reduces the number of parameters returned by the server side to the client and further reduces the communication overhead between the server side and the client. The parameters of the converged first student model are adjusted on each client based on the converged first student model and the local data set of each client, and the adjusted first student model is taken as the local model of each client. The server side and the client only communicate once, avoiding iterative parameter transmission between the server side and the client in federated learning, improving the communication efficiency between the client and the server side, and reducing the communication overhead between the server side and the client.

[0056] In some embodiments, the deployment module 50 is further configured to: the number of neural network layers in each second student model is the same; acquire, by the server side, construction information of all second student models; construct, by the server side, all second student models based on the construction information of all second student models; and sequentially splice, by the server side, the corresponding neural network layers in each second student model to obtain the aggregated model and deploy the aggregated model on the server side.

[0057] In some embodiments, the second training module 20 is further configured to: acquire, by each client, probability distribution information of the converged teacher model output; train, by each client, the second student model based on the probability distribution information of the converged teacher model; and determine that the converged second student model is obtained in response to determining that a difference between the probability distribution information of the converged teacher model and the probability distribution information of the second student model output is less than a first preset value.

[0058] In some embodiments, the parameter replacement module 30 is further configured to: each second student model includes a feature extractor, the number of channels of the corresponding neural network layer in the feature extractor of each second student model is the same, the channels include input channels and output channels, and the number of output channels in each neural network layer of the feature extractor of the aggregated model is equal to the sum of the numbers of output channels of the corresponding neural network layers of all second student models; and for each round of training in the iterative training, perform the following operations: for each neural network layer in the feature extractor of the aggregated model, perform the following operations: calculate the input data received by the input channels of the current neural network layer to obtain the output data of the output channels of the current neural network layer; determine target output data of the current neural network layer based on the output data and the number of output channels of the neural network layer of the second student model used for splicing the current neural network layer, and input the target output data into the input channels of the next neural network layer of the current neural network layer.

[0059] In some embodiments, the parameter replacement module 30 is further configured to: determine attention weights of all output data in the current neural network layer based on the output data; sort all the attention weights in descending order, and take the output data corresponding to the first N attention weights as the target output data, where the value of N is equal to the number of output channels of the neural network layer of the second student model used for splicing the current neural network layer.

[0060] In some embodiments, the third training module 40 is further configured to acquire, by using the server side, probability distribution information of the converged aggregated model output; train, by using the server side, the first student model based on the probability distribution information of the converged aggregated model; and determine the converged first student model in response to determining that a difference between the probability distribution information of the converged aggregated model and probability distribution information output by the first student model is less than a second preset value.

[0061] In some embodiments, the adjustment module 50 is further configured to acquire a local data set of each client; perform a preprocessing operation on the local data set of each client; divide the local data set of each client after the preprocessing operation into a training set and a test set; perform parameter adjustment on the converged first student model by using the training set; test the converged first student model after the parameter adjustment based on the test set to obtain a test evaluation parameter, and obtain the first student model after the parameter adjustment when the test evaluation parameter is greater than a preset evaluation threshold.

[0062] For the convenience of description, the above apparatus is described in various modules in terms of functions. Of course, the functions of the modules can be implemented in one or more software and / or hardware in the implementation of the present application.

[0063] The apparatus of the above embodiments is used to implement the corresponding method of constructing a local model based on federated learning in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here.

[0064] Based on the same inventive concept, the present application also provides an electronic device corresponding to the method of any of the above embodiments, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of constructing a local model based on federated learning according to any of the above embodiments when executing the program.

[0065] Figure 3 A more specific hardware structure schematic diagram of an electronic device provided by the present embodiment is shown, which can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for communication within the device.

[0066] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing relevant programs to implement the technical solutions provided by the embodiments of the present specification.

[0067] The memory 1020 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the relevant program codes are saved in the memory 1020 and called and executed by the processor 1010.

[0068] The input / output interface 1030 is configured to connect input / output modules to implement information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input devices can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices can include a display, a speaker, a vibrator, an indicator light, etc.

[0069] The communication interface 1040 is configured to connect a communication module (not shown in the figure) to implement the communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as USB, network cable, etc.) or through a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0070] The bus 1050 includes a channel for transmitting information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0071] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only include the components necessary to implement the solutions of the embodiments of the present specification, and does not have to include all the components shown in the figure.

[0072] The electronic device of the above embodiment is used to implement the corresponding method for constructing a local model based on federated learning in any of the preceding embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0073] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method for constructing a local model based on federated learning according to any of the above embodiments.

[0074] The computer-readable medium of the present embodiment includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0075] The storage medium of the above embodiment stores computer instructions for causing the computer to perform the method for constructing a local model based on federated learning according to any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0076] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope (including claims) of the present application is limited to these examples; under the idea of the present application, the above embodiments or technical features in different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the embodiments of the present application as described above. In order to be brief, they are not provided in detail.

[0077] Additionally, to simplify the description and discussion, and so as not to obscure the embodiments of the application being presented, the well-known functions or constructions of integrated circuit (IC) chips and other components can or can not be shown in the figures and will be omitted as not to unnecessarily obscure the embodiments of the application being presented. Moreover, the devices can be shown in block diagram form in order to avoid obscuring the embodiments of the application, and this also acknowledges the fact that the details in regard to how the devices are implemented in regard to the embodiments of the application are highly dependent on the platform within which the embodiments of the application are being implemented (i.e., these details should be readily apparent to those of ordinary skill in the art). Where specific details are set forth in order to describe an illustrative embodiment of the application, it will be apparent to one of ordinary skill in the art that the embodiments of the application can be practiced without, or with variation of, these specific details. Thus, the description is to be considered as illustrative and not restrictive, and the scope of the application should be determined not with reference to the above description, but should be given to the appended claims.

[0078] While the application has been described in connection with specific embodiments thereof, many alternatives, modifications and variations will be apparent to those of ordinary skill in the art. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0079] The embodiments of the application are intended to cover all such alternatives, modifications and variations as falling within the broad scope of the appended claims. Accordingly, any and all such modifications, variations or equivalents that fall within the spirit and scope of the embodiments of the application are intended to be included within the scope of the application.

Claims

1. A method for constructing a local model based on federated learning, characterized in that, The method is applied to a federated learning system, the federated learning system comprising a server end and a plurality of client ends, an aggregation model and a first student model being deployed on the server end, a teacher model and a second student model being respectively deployed on each client end, the aggregation model being obtained by aggregating all the second student models; the method comprising: training the teacher model on each client end based on a local data set of each client end until the teacher model converges; training the second student model on each client end based on the converged teacher model of each client end to obtain a converged second student model, and sending parameters of the converged second student model to the server end; replacing parameters of the aggregation model on the server end based on parameters of all the converged second student models, and iteratively training the aggregation model with the replaced parameters on the server end based on a data set of the server end until the aggregation model with the replaced parameters converges; training the first student model on the server end based on the converged aggregation model to obtain a converged first student model, and sending the converged first student model to each client end; adjusting the converged first student model on each client end based on the converged first student model and the local data set of each client end, and taking the adjusted first student model as a local model of each client end.

2. The method of claim 1, wherein, The number of neural network layers in each second student model is the same; a method for deploying the aggregation model on the server end, comprising: obtaining construction information of all the second student models on the server end; constructing all the second student models on the server end based on the construction information of all the second student models; sequentially splicing the neural network layers corresponding to each second student model on the server end to obtain the aggregation model and deploy the aggregation model on the server end.

3. The method of claim 1, wherein, The method for training the second student model on each client end based on the converged teacher model of each client end to obtain a converged second student model, comprising: obtaining probability distribution information output by the converged teacher model on each client end; training the second student model on each client end based on the probability distribution information of the converged teacher model; in response to determining that a difference between the probability distribution information of the converged teacher model and probability distribution information output by the second student model is less than a first preset value, determining the converged second student model.

4. The method of claim 1, wherein, Each second student model comprises a feature extractor, the number of channels of the corresponding neural network layer in the feature extractor of each second student model is the same, the channels comprising input channels and output channels, the number of output channels in each neural network layer of the feature extractor of the aggregation model being equal to the sum of the numbers of output channels of the corresponding neural network layers of all the second student models; The method for iteratively training the aggregation model with the replaced parameters on the server end based on a data set of the server end, comprising: For each round of training in the iterative training, the following operations are performed: for each neural network layer in a feature extractor of the aggregated model, the following operations are performed: computing input data received by an input channel of a current neural network layer to obtain output data of an output channel of the current neural network layer; determining target output data of the current neural network layer based on the output data and a number of output channels of a neural network layer of a second student model spliced with the current neural network layer, and inputting the target output data into an input channel of a next neural network layer of the current neural network layer.

5. The method of claim 4, wherein, The determining of the target output data of the current neural network layer based on the output data and the number of output channels of the neural network layer of the second student model spliced with the current neural network layer includes: determining attention weights of all the output data in the current neural network layer based on the output data; sorting all the attention weights in descending order, and taking output data corresponding to the first N attention weights as the target output data, where N is equal to the number of output channels of the neural network layer of the second student model spliced with the current neural network layer.

6. The method of claim 1, wherein, The training of the first student model by the server based on the converged aggregated model includes: obtaining probability distribution information output by the converged aggregated model by the server; training the first student model by the server based on the probability distribution information of the converged aggregated model; determining the converged first student model in response to determining that a difference between the probability distribution information of the converged aggregated model and probability distribution information output by the first student model is less than a second preset value.

7. The method of claim 1, wherein, The parameter adjustment of the converged first student model by each client includes: obtaining a local data set of each client; performing a preprocessing operation on the local data set of each client; dividing the local data set of each client after the preprocessing operation into a training set and a test set; adjusting the parameters of the converged first student model by using the training set; testing the converged first student model after the parameter adjustment based on the test set to obtain a test evaluation parameter, and obtaining the first student model after the parameter adjustment when the test evaluation parameter is greater than a preset evaluation threshold. 8.A device for constructing a local model based on federated learning, characterized in that, The application is applied to a federated learning system, the federated learning system includes a server and a plurality of clients, an aggregated model and a first student model are deployed on the server, a teacher model and a second student model are respectively deployed in each client, the aggregated model is obtained based on all the second student models; the device includes: a first training module configured to train the teacher model by each client based on a local data set of each client until the teacher model converges. a second training module configured to train, based on the converged teacher model of each client, the second student model of each client to obtain a converged second student model, and send parameters of the converged second student model to the server end; a parameter replacement module configured to replace, based on parameters of all converged second student models, parameters of the aggregated model, and iteratively train, based on a data set of the server end, the aggregated model until the aggregated model after the parameter replacement converges; a third training module configured to train, based on the converged aggregated model, the first student model of the server end to obtain a converged first student model, and send the converged first student model to each client; an adjustment module configured to adjust, based on the converged first student model and the local data set of each client, parameters of the converged first student model of each client, and use the adjusted first student model as a local model of each client.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 7 when executing the program.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to make the computer execute the method of any one of claims 1 to 7.

Images