A federated learning method and related apparatus

Abstract

A federated learning method applied to the field of artificial intelligence technology. In the method, when the aggregation node obtains models of different network structures, the models of the same network structure are divided into the same group, and the parameters of the models in the same group are aggregated to obtain multiple aggregation models with different network structures; and for each aggregation model, knowledge distillation training is performed on each aggregation model based on the multiple models obtained originally, to realize experience transmission between models of different network structures, thereby integrating the knowledge and experience of models of various network structures, combining the advantages of parameter aggregation and knowledge distillation in integrating model experience, realizing aggregation of models of different network structures, and ensuring the prediction accuracy of the model obtained after aggregation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and more particularly to a federated learning method and related apparatus. Background Technology

[0002] As users become increasingly concerned about protecting their personal privacy, the inability of data owners to share user data has created numerous "data silos." These "data silos" pose a new challenge to artificial intelligence (AI) based on massive amounts of data: how to train machine learning models without access to sufficient training data.

[0003] To address the challenges posed by "data silos," federated learning emerged. Federated learning effectively helps clients perform joint training and build shared machine learning models without sharing data resources (i.e., training data does not leave their local machines). During the local training phase, each client trains its local model based on the training data. In the model aggregation phase, each client uploads its local model to the cloud server, which aggregates all local models to obtain a global model and distributes it. Clients then update their local models using the training data based on the global model, repeating this process until the global model converges.

[0004] Current federated learning methods require that each client trains a model with the same structure so that the central node can aggregate models uploaded by multiple clients through parameter averaging. However, in some cases, models trained on different clients may have different structures. Therefore, existing federated learning methods cannot be applied to scenarios with heterogeneous models and struggle to achieve model aggregation. Summary of the Invention

[0005] This application provides a federated learning method that can aggregate models with different network structures and ensure the prediction accuracy of the aggregated model.

[0006] The first aspect of this application provides a federated learning method applied in the field of artificial intelligence technology. The method includes: an aggregation node acquiring multiple models, which are obtained from multiple nodes. Furthermore, the multiple models acquired by the aggregation node are used to perform the same processing task, such as image processing or natural language processing.

[0007] Then, based on the network structure of multiple models, the aggregation node divides the multiple models into multiple groups, where each group includes at least one model, and the models in the same group have the same network structure;

[0008] Secondly, the aggregation node aggregates the parameters of the models in each of the multiple groups, resulting in multiple aggregated models. Models in the same group are aggregated into one aggregated model, and multiple aggregated models correspond to multiple groups. Specifically, the network structure of each aggregated model is the same as the network structure of the models in its corresponding group, and the weight parameters in each aggregated model can be obtained by weighted summation of the weight parameters of one or more models in the same group.

[0009] After obtaining multiple aggregate models, the aggregation node uses each of these models as a teacher model and performs knowledge distillation training on each of the multiple aggregate models to obtain multiple trained models.

[0010] Finally, the aggregation node sends multiple trained models to multiple nodes.

[0011] In this scheme, when the aggregation node obtains models with different network structures, models with the same network structure are grouped into the same group, and the parameters of the models in the same group are aggregated to obtain multiple aggregated models with different network structures. Furthermore, for each aggregated model, knowledge distillation training is performed on each aggregated model based on the original multiple models, so as to realize the transfer of experience between models with different network structures, thereby integrating the knowledge and experience of models with various network structures and improving the prediction accuracy of the model.

[0012] By aggregating parameters of models with the same network structure and performing knowledge distillation on the aggregated models based on models with various network structures, the advantages of parameter aggregation and knowledge distillation in integrating model experience can be combined to achieve the aggregation of models with different network structures and ensure the prediction accuracy of the resulting models.

[0013] In one possible implementation, the process of the aggregation node performing knowledge distillation training on each of the multiple aggregation models may specifically include: the aggregation node inputting training samples into a first aggregation model and multiple models respectively, to obtain a target prediction result and multiple prediction results, wherein the first aggregation model is any one of the multiple aggregation models, the target prediction result corresponds to the first aggregation model, and the multiple prediction results correspond to the multiple models; the aggregation node trains the first aggregation model based on a loss function to obtain the trained first aggregation model, wherein the loss function is obtained based on the difference between the target prediction result and the multiple prediction results.

[0014] In this scheme, a distillation loss function is constructed based on the difference between the prediction results of the original multiple models and the prediction results of the aggregated model. This enables the knowledge distillation method to support the transfer of experience between models with different network structures, allowing the aggregated model to further aggregate knowledge and experience under various network structures and improve the prediction accuracy of the aggregated model.

[0015] In one possible implementation, the loss function used to train the first aggregation model is obtained based on the difference between the target prediction and each of the multiple predictions; or, the loss function is obtained based on the difference between the target prediction and the weighted average of the multiple predictions.

[0016] In one possible implementation, the loss function used to train the first aggregation model includes a first sub-loss function and a second sub-loss function. The first sub-loss function is obtained based on the difference between the target prediction result and multiple prediction results, and the second sub-loss function is obtained based on the difference between the true label of the training sample and the target prediction result.

[0017] In this scheme, a distillation loss function is constructed based on the difference between the prediction results of the original multiple models and the prediction results of the aggregated model, and a label loss function is constructed based on the real labels of the training samples. Under the constraint of the label loss function, knowledge distillation is used to support the transfer of experience between models with different network structures, so that the aggregated model can effectively aggregate knowledge and experience under various network structures and improve the prediction accuracy of the aggregated model.

[0018] In one possible implementation, the method further includes: based on training data, the aggregation node determines a target model from multiple trained models; using the multiple trained models as teacher models, the aggregation node performs knowledge distillation training on the target model to obtain an updated target model, and the updated target model is used to perform inference tasks based on local data.

[0019] In this scheme, after the aggregation node obtains the target model based on the local training data, it uses the original received model as the teacher model to perform knowledge distillation training on the target model. This ensures that the target model does not forget its original experience while taking into account the local data of the first node. It effectively avoids the drastic fluctuation of the target model's prediction accuracy due to the poor quality of the local data of the first node (i.e., avoids the phenomenon of overfitting of the target model).

[0020] In one possible implementation, the aggregation node determines the target model among multiple trained models based on the training data. Specifically, the aggregation node inputs the training data into each of the multiple models and determines the prediction accuracy of each model based on the prediction results output by each model. The aggregation node determines the target model based on the prediction accuracy of each model, and the target model includes one or more models with the highest prediction accuracy among the multiple trained models.

[0021] In this scheme, the prediction accuracy of each model is determined based on the local data of the node, thereby measuring the degree of adaptation of each model to the local data and ensuring that the node can select the target model with the best performance when processing the local data of the node from multiple models.

[0022] In one possible implementation, the aggregation node determines the target model from multiple trained models based on the training data. Specifically, the aggregation node inputs the training data into a gate network, obtaining multiple weight values ​​output by the gate network. These weight values ​​correspond one-to-one with the multiple trained models, and each weight value represents a weight in the weighted summation process of the outputs of the multiple trained models. The gate network is a pre-trained network. In other words, the multiple trained models work in conjunction with the gate network to process the training data. The gate network outputs the weight values ​​corresponding to each of the multiple trained models based on the training data. Thus, after each trained model outputs its corresponding training data output, the multiple outputs are weighted and summed according to their respective weight values ​​to obtain the final output.

[0023] The aggregation node determines the target model based on multiple weight values. The target model includes one or more models with the highest corresponding weight values ​​among multiple trained models. In other words, the first node can measure the degree of fit of each trained model to the local data of the first node by the weight values ​​corresponding to each trained model, and then select one or more models with the highest degree of fit (i.e., the target model) from among multiple trained models.

[0024] In this scheme, the weight values ​​corresponding to each model are output by training a gate network, thereby measuring the degree of adaptation of each model to local data and ensuring that the node can select the target model with the best performance when processing the node's local data from multiple models.

[0025] In one possible implementation, the aggregation node sends multiple trained models to multiple nodes. Specifically, the aggregation node determines a subset of trained models from among the multiple trained models based on the target data, where the target data is the data obtained from the first node, and the subset of trained models is the model with the best performance in processing the target data among the multiple trained models; the aggregation node then sends the subset of trained models to the first node.

[0026] In one possible implementation, the parameters of the models in each of the multiple groups are aggregated to obtain multiple aggregated models, including: weighted summation of the weight parameters of the multiple models in the first group to obtain a first aggregated model; wherein the first group belongs to multiple groups and the first aggregated model belongs to multiple aggregated models.

[0027] The second aspect of this application provides a federalized learning approach, including:

[0028] The aggregation node receives multiple models with different network structures, and these models are obtained by aggregating models from different nodes.

[0029] Based on the training data, the target model is determined among multiple models;

[0030] Using multiple models as teacher models, knowledge distillation training is performed on the target model to obtain the trained target model.

[0031] In one possible implementation, the target model is determined from multiple models based on training data, including:

[0032] The training data is input into each of the multiple models, and the prediction accuracy of each model is determined based on the prediction results output by each model.

[0033] The target model is determined based on the prediction accuracy of each model. The target model is one or more models with the highest prediction accuracy among multiple models.

[0034] In one possible implementation, the target model is determined from multiple models based on training data, including:

[0035] The training data is input into the gate network to obtain multiple weight values ​​output by the gate network. Each weight value corresponds to one of the multiple trained models. The multiple weight values ​​are used to represent the weights of the output results of the multiple trained models in the weighted summation process. The gate network is a pre-trained network.

[0036] The target model is determined based on multiple weight values. The target model includes one or more models with the highest corresponding weight values ​​among the multiple models.

[0037] In one possible implementation, multiple models are obtained by the aggregation node receiving models from different nodes, aggregating the parameters of models with the same network structure, and then training the aggregated model by knowledge distillation based on the models from different nodes.

[0038] A third aspect of this application provides a federal learning device, comprising:

[0039] The acquisition module is used to acquire multiple models, which are obtained from multiple nodes;

[0040] The processing module is also used to divide multiple models into multiple groups based on the network structure of multiple models, wherein each group includes at least one model and the models in the same group have the same network structure.

[0041] The processing module is also used to aggregate the parameters of the models in each of the multiple groups to obtain multiple aggregated models. The models in the same group are aggregated to obtain one aggregated model, and the multiple aggregated models correspond to multiple groups.

[0042] The processing module is also used to perform knowledge distillation training on each of the multiple aggregate models, using multiple models as teacher models, to obtain multiple trained models.

[0043] The sending module is used to send multiple trained models to multiple nodes.

[0044] In one possible implementation, the processing module is also used for:

[0045] The training samples are input into the first aggregation model and multiple models respectively to obtain the target prediction result and multiple prediction results. The first aggregation model is any one of the multiple aggregation models. The target prediction result corresponds to the first aggregation model, and the multiple prediction results correspond to the multiple models.

[0046] The first aggregate model is trained based on the loss function, and the trained first aggregate model is obtained. The loss function is obtained based on the difference between the target prediction result and multiple prediction results.

[0047] In one possible implementation, the loss function is derived based on the difference between the target prediction and each of the multiple predictions.

[0048] Alternatively, the loss function can be derived from the difference between the target prediction and the weighted average of multiple predictions.

[0049] In one possible implementation, the loss function includes a first sub-loss function and a second sub-loss function. The first sub-loss function is obtained based on the difference between the target prediction result and multiple prediction results, and the second sub-loss function is obtained based on the difference between the true label of the training sample and the target prediction result.

[0050] In one possible implementation, the processing module is also used for:

[0051] Based on the training data, the target model is determined from multiple trained models;

[0052] Using multiple trained models as teacher models, knowledge distillation training is performed on the target model to obtain an updated target model. The updated target model is then used to perform reasoning tasks based on local data.

[0053] In one possible implementation, the processing module is also used for:

[0054] The training data is input into each of the multiple models, and the prediction accuracy of each model is determined based on the prediction results output by each model.

[0055] The target model is determined based on the prediction accuracy of each model. The target model includes one or more models with the highest prediction accuracy among multiple trained models.

[0056] In one possible implementation, the processing module is also used for:

[0057] The training data is input into the gate network to obtain multiple weight values ​​output by the gate network. Each weight value corresponds to one of the multiple trained models. The multiple weight values ​​are used to represent the weights of the output results of the multiple trained models in the weighted summation process. The gate network is a pre-trained network.

[0058] The target model is determined based on multiple weight values. The target model includes one or more models with the highest corresponding weight values ​​among multiple trained models.

[0059] In one possible implementation,

[0060] The processing module is also used to determine a partial trained model from multiple trained models based on the target data. The target data is the data obtained from the first node, and the partial trained model is the model with the best performance in processing the target data among multiple trained models.

[0061] The sending module is also used to send part of the trained model to the first node.

[0062] In one possible implementation, the processing module is also used to perform a weighted summation of the weight parameters of multiple models in the first group to obtain a first aggregated model;

[0063] Wherein, the first group belongs to the plurality of groups, and the first aggregation model belongs to the plurality of aggregation models.

[0064] The fourth aspect of this application provides a federal learning device, comprising:

[0065] The receiving module is used to receive multiple models from the aggregation node. The multiple models have different network structures, and the multiple models are obtained by the aggregation node aggregating the models on different nodes.

[0066] The processing module is used to determine the target model from multiple models based on the training data;

[0067] The processing module is also used to perform knowledge distillation training on the target model using multiple models as teacher models, to obtain the trained target model.

[0068] In one possible implementation, the processing module is also used for:

[0069] The training data is input into each of the multiple models, and the prediction accuracy of each model is determined based on the prediction results output by each model.

[0070] The target model is determined based on the prediction accuracy of each model, and the target model is one or more models with the highest prediction accuracy among the plurality of models.

[0071] In one possible implementation, the processing module is also used for:

[0072] The training data is input into the gate network to obtain multiple weight values ​​output by the gate network. Each weight value corresponds to one of the multiple trained models. Each weight value is used to represent the weight of the output of the multiple trained models in the weighted summation process. The gate network is a pre-trained network.

[0073] The target model is determined based on the plurality of weight values, wherein the target model includes one or more models with the highest corresponding weight values ​​among the plurality of models.

[0074] In one possible implementation, the multiple models are obtained by the aggregation node after receiving models from different nodes, performing parameter aggregation on models with the same network structure, and then training the aggregated model by knowledge distillation based on the models from the different nodes.

[0075] The fifth aspect of this application provides a federated learning apparatus, which may include a processor and a memory coupled together. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method of any implementation of the first or second aspect described above is implemented. For details regarding the steps of the various possible implementations of the first or second aspect executed by the processor, please refer to the first or second aspect; further details will not be provided here.

[0076] The sixth aspect of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method of any implementation of the first or second aspect described above.

[0077] The seventh aspect of this application provides a circuit system including a processing circuit configured to perform the method of any implementation of the first or second aspect described above.

[0078] The eighth aspect of this application provides a computer program product that, when run on a computer, causes the computer to perform any implementation of the first or second aspect described above.

[0079] A ninth aspect of this application provides a chip system including a processor for supporting a server or threshold value acquisition device in implementing the functions involved in any implementation of the first or second aspect described above, such as transmitting or processing data and / or information involved in the methods described above. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the server or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0080] The beneficial effects of aspects two through nine mentioned above can be referred to the introduction of aspect one above, and will not be repeated here. Attached Figure Description

[0081] Figure 1 A schematic diagram illustrating a scenario of federated learning provided for an embodiment of this application;

[0082] Figure 2 This is a schematic diagram illustrating the execution process of federated learning in related technologies.

[0083] Figure 3 This is a schematic diagram of the structure of a convolutional neural network provided in an embodiment of this application;

[0084] Figure 4 This is a schematic diagram of another convolutional neural network structure provided in an embodiment of this application;

[0085] Figure 5A schematic diagram of a system architecture 500 provided for an embodiment of this application;

[0086] Figure 6 A schematic diagram of a system architecture 600 provided for an embodiment of this application;

[0087] Figure 7 A flowchart illustrating a federated learning method provided in an embodiment of this application;

[0088] Figure 8 A schematic diagram illustrating an aggregation node receiving multiple models, provided in an embodiment of this application;

[0089] Figure 9 This is a schematic diagram illustrating grouping and parameter aggregation of multiple models, provided as an embodiment of this application.

[0090] Figure 10 This is a schematic diagram illustrating knowledge distillation of multiple aggregation models, provided as an embodiment of this application.

[0091] Figure 11 This is a schematic diagram illustrating how an aggregation node sends a trained model to each node, as provided in an embodiment of this application.

[0092] Figure 12A A schematic diagram of a target screening model provided in an embodiment of this application;

[0093] Figure 12B A schematic diagram of a target screening model provided in an embodiment of this application;

[0094] Figure 13 A schematic diagram of another target screening model provided in an embodiment of this application;

[0095] Figure 14 A flowchart illustrating another federated learning method provided in this application embodiment;

[0096] Figure 15 A schematic diagram of the structure of a federated learning device provided in an embodiment of this application;

[0097] Figure 16 A schematic diagram of the structure of a federated learning device provided in an embodiment of this application;

[0098] Figure 17 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0099] Figure 18 A schematic diagram of the structure of a chip provided in an embodiment of this application;

[0100] Figure 19 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0101] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will understand that, with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0102] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0103] For ease of understanding, some technical terms involved in the embodiments of this application will be introduced below.

[0104] (1) Federated Learning

[0105] Federated learning is essentially a model training method that enables data sharing and collaborative model building while ensuring data privacy, security, and legal compliance. The core idea of ​​federated learning is that when multiple data sources participate in model training, the original data does not need to be transferred; instead, the model is trained jointly only by exchanging intermediate parameters. The original data can remain locally. This approach achieves a balance between data privacy protection and data sharing analysis, resulting in a data application model that is "usable but not visible."

[0106] For example, please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating a federated learning scenario provided in an embodiment of this application. The federated learning scenario may include multiple client nodes and a central node, where the client nodes and central node can be any node supporting data transmission (such as a network node). For example, a client node can be a mobile terminal or a personal computer. The central node can be a server or a cluster server. In some embodiments, the client nodes may be referred to as the owners of the training data, and the central node may be referred to as the coordinator of the federated learning process.

[0107] The central node is used to maintain the federated model. Client nodes can obtain the federated model from the central node and train it locally using their local training data to obtain a local model. After training the local model, the client node can send it to the central node so that the central node can update or optimize the federated model. This process is repeated multiple times until the federated model converges or reaches a preset iteration stopping condition.

[0108] The following will combine Figure 2 This section introduces the general process of federated learning. For example... Figure 2 As shown, the execution process of federated learning includes the following steps 201-205.

[0109] Step 201: The central node constructs the federated model.

[0110] The central node can build general machine learning models or specific machine learning models according to needs. Taking image recognition tasks as an example, the central node can build a convolutional neural network (CNN) as a federated model.

[0111] Step 202: Multiple client nodes obtain or receive the federated model from the central node.

[0112] In this system, multiple client nodes can obtain or receive the same federated model from the central node. For example, in one implementation, a client node can proactively request the central node to distribute the federated model. Alternatively, in another implementation, the central node proactively distributes the federated model to the client nodes. Taking a personal computer as the client node and a server as the central node, the personal computer can download the federated model from the server.

[0113] Step 203: Multiple client nodes train the federated model using local training data to obtain a local model.

[0114] Specifically, multiple client nodes can use the federated model as the initial model for their local model, and then use the local training data to train the initial model in one or more steps to obtain the local model.

[0115] Because different client nodes have different local training data, different client nodes can train different local models based on the same federated model.

[0116] Step 204: Multiple client nodes upload the trained local models to the central node.

[0117] Step 205: The central node aggregates multiple local models to obtain the updated federated model.

[0118] For example, in one implementation, the central node can perform a weighted summation of the parameters of the local models of multiple client nodes and use the result of the weighted summation as the updated federated model.

[0119] It should be noted that the process described in steps 201-205 can be viewed as an iteration in the federated learning process. The central node and client nodes can repeat steps 201-205 until the federated model converges or achieves the preset effect.

[0120] Generally, federated learning can be used to train machine learning models. The most common machine learning model is the neural network. The relevant concepts of neural networks will be explained below.

[0121] (2) Neural Network

[0122] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0123]

[0124] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0125] (3) Deep Neural Networks

[0126] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows:

[0127] in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation.

[0128] Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0129] (4) Convolutional Neural Network (CNN)

[0130] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or feature map. A convolutional layer refers to the layer of neurons in a CNN that performs convolution processing on the input signal (e.g., the first and second convolutional layers in this embodiment). In a convolutional layer of a CNN, a neuron can be connected to only some of the neurons in neighboring layers. A convolutional layer typically contains several feature planes, each of which can be composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of the image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, for all locations on the image, we can use the same learned image information. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0131] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0132] Specifically, such as Figure 3 As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130.

[0133] The structure consisting of the convolutional layer / pooling layer 120 and the neural network layer 130 can be the first convolutional layer and the second convolutional layer described in this application. The input layer 110 is connected to the convolutional layer / pooling layer 120, and the convolutional layer / pooling layer 120 is connected to the neural network layer 130. The output of the neural network layer 130 can be input to the activation layer, and the activation layer can perform non-linear processing on the output of the neural network layer 130.

[0134] Convolutional / pooling layers: 120. Convolutional layers: such as... Figure 3 The convolutional / pooling layer 120 shown may include layers 121-126 as in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0135] Taking convolutional layer 121 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors from the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices with the same dimension also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0136] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0137] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0138] Pooling layers: Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 3 In the example of 120, each layer 121-126 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers.

[0139] Neural Network Layer 130: After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required output information. As before, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layer 130 to generate one or a set of required class numbers of output. Therefore, neural network layer 130 may include multiple hidden layers (such as...). Figure 3 As shown in layers 131, 132 to 13n) and output layer 140, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0140] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 4 The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 4 The propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0141] It should be noted that, as Figure 3 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 4 The multiple convolutional / pooling layers shown are run in parallel, and the extracted features are all input into the full neural network layer 130 for processing.

[0142] (5) Loss Function

[0143] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the neural network becomes a process of minimizing this loss.

[0144] (6) Backpropagation algorithm

[0145] Neural networks can use the back propagation (BP) algorithm to correct the initial parameters of the model during training, thereby reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates error loss. By backpropagating this error loss information, the initial model parameters are updated, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining optimal model parameters, such as the weight matrix.

[0146] (7) Non-independent and identically distributed (Non-IID)

[0147] In probability theory and statistics, independent and identically distributed (IID) means that each variable in a set of random variables has the same probability distribution, and these random variables are independent of each other. However, an IID set of random variables does not mean that every event in their sample space has the same probability of occurring. For example, the sequence of results from rolling a non-fair die is IID, but the probability of rolling each face up is not the same.

[0148] So, Non-IID means that the variables are not independent, or not identically distributed. In federated learning, Non-IID generally refers to situations where the data does not conform to the same distribution, because the data distributions are certainly independent, but they do not necessarily follow the same sampling method. For example, if the entire dataset contains 100 types of images, one device contains only landscape images, while another device contains only images of people and plants. The former follows one distribution (1 / 100), and the latter follows another distribution (2 / 100). Conversely, if one device contains all 100 types of images, and other devices also contain all 100 types of images, then they are identically distributed.

[0149] (8) Modality

[0150] From a data perspective, modality refers to data types such as images or text, and different modalities of data refer to different types of data.

[0151] From a model perspective, modality refers to the type of model. Different modal models are different models trained on various types of data.

[0152] (9) Knowledge Transfer

[0153] Knowledge transfer refers to improving the modeling capabilities of a target domain by transferring an existing set of knowledge to another target domain. Model-based knowledge transfer can be simply understood as transfer learning based on model parameters, which aims to enable the constructed model to learn general knowledge across domains.

[0154] (10) Knowledge Distillation

[0155] Knowledge distillation is a method in transfer learning. It involves using the output of a pre-trained model as a supervisory signal to train another network. Simply put, knowledge distillation uses a pre-trained network as the teacher network and the network to be trained as the student network. Guided by the teacher network, the student network is trained to acquire similar or identical data processing capabilities to the teacher network. Typically, the teacher network can be a complex model, while the student network can be a simple one; through knowledge distillation, the feature representations learned by the complex model can be transferred to the simpler model.

[0156] (11) Experience transfer

[0157] Experience transfer can refer to a node recording its data pattern information in the form of a model or data and passing it to another node for training the model, thereby realizing the mutual transfer of experience between nodes.

[0158] Current federated learning methods require that each client trains a model with the same structure so that the central node can aggregate models uploaded by multiple clients through parameter averaging. However, in some cases, the local data on different clients may be of different modalities, meaning the data on different clients is not independent and identically distributed. Therefore, to adapt to local data as much as possible, the models trained on different clients may have different structures. Consequently, existing federated learning methods cannot be applied to scenarios with heterogeneous models and struggle to achieve model aggregation.

[0159] In view of this, this application provides a federated learning method. When the aggregation node obtains models with different network structures, it divides the models with the same network structure into the same group and performs parameter aggregation on the models in the same group to obtain multiple aggregated models with different network structures. Furthermore, for each aggregated model, it performs knowledge distillation training on each aggregated model based on the original multiple models to realize the transfer of experience between models with different network structures, thereby integrating the knowledge and experience of models with various network structures and improving the prediction accuracy of the model.

[0160] In this way, by aggregating parameters of models with the same network structure and performing knowledge distillation on the aggregated model based on models with various network structures, the advantages of parameter aggregation and knowledge distillation in integrating model experience can be combined to achieve the aggregation of models with different network structures and ensure the prediction accuracy of the model obtained after aggregation.

[0161] For ease of understanding, the system architecture applied to the federated learning method provided in the embodiments of this application will be introduced below.

[0162] Please see Figure 5 , Figure 5 This is a schematic diagram of a system architecture 500 provided in an embodiment of this application. For example... Figure 5 As shown, in this system architecture 500, the aggregation node 501 is implemented by one or more servers. Optionally, the aggregation node 501 can work with other computing devices, such as data storage, routers, load balancers, etc. The aggregation node 501 can be deployed on a single physical site or distributed across multiple physical sites.

[0163] Users can interact with the aggregation node 501 by operating their own user devices (e.g., client node 511, client node 512... client node 51n). Each client node can represent any computing device, such as a personal computer, server, computer workstation, smartphone, tablet, smart camera, smart car or other types of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0164] Each client node can interact with the aggregation node 501 through any communication mechanism / standard communication network. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0165] In the specific implementation, aggregation node 501 acts as the central node, responsible for receiving models uploaded by various client nodes. The network structures of the models uploaded by different client nodes may differ. Thus, aggregation node 501 aggregates the models based on the federated learning method provided in this embodiment, and then returns the aggregated model to each client node.

[0166] Specifically Figure 5 The system architecture described is a centralized system architecture, in which a unified central node is responsible for aggregating the models on each client node.

[0167] Please see Figure 6 , Figure 6 This is a schematic diagram of a system architecture 600 provided in an embodiment of this application. Figure 6 As shown, this system architecture 600 includes distributed nodes 601-608. Each node in system architecture 600 can represent any computing device, such as a server, personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other types of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0168] In this system architecture 600, nodes 601 and 602 can be nodes with stronger computing power (e.g., servers), meaning their computing power can be greater than that of nodes 603-608. Therefore, in system architecture 600, nodes with relatively weaker computing power can send models to nearby nodes with stronger computing power, allowing these nodes to receive models sent by multiple nearby nodes. Furthermore, the network structures of the models sent by different nodes may differ. For example, nodes 603-606 send models to nearby node 601; nodes 605-608 send models to adjacent node 602.

[0169] In this way, after receiving multiple models, nodes with stronger computing power can aggregate the models based on the federated learning method provided in this embodiment, and then return the aggregated models to the nodes with weaker computing power.

[0170] Specifically Figure 6 The system architecture described is a decentralized distributed system architecture, in which some nodes can receive models sent by other nodes, thereby realizing model aggregation. The nodes responsible for model aggregation can be determined in various ways, such as selecting nodes with strong computing power, or selecting nodes that are close to multiple nodes. This embodiment does not limit the nodes responsible for model aggregation in the distributed system architecture.

[0171] The methods used in the embodiments of this application have been described above. The execution process of the methods provided in the embodiments of this application will be described in detail below. Please refer to... Figure 7 , Figure 7 This is a flowchart illustrating a federated learning method provided in an embodiment of this application. Figure 7 As shown, the federated learning method includes the following steps 701-708.

[0172] Step 701: The aggregation node obtains multiple models, which are obtained from multiple nodes.

[0173] In this embodiment, the aggregation node can obtain multiple models from multiple nodes to aggregate the models deployed on the multiple nodes. Each of the multiple nodes can send one or more models to the aggregation node, enabling the aggregation node to obtain the aforementioned multiple models.

[0174] The models obtained by the aggregation node can be models trained by other nodes based on local data, and the aggregation node further aggregates these models.

[0175] Optionally, the aggregation node can be a central node in a centralized system architecture, solely responsible for aggregating models obtained from other nodes. Alternatively, the aggregation node can be a node in a distributed system architecture, responsible for aggregating models obtained from other nodes, and also deploying models on this node to process local data. Therefore, when the aggregation node is a node in a distributed system architecture, the multiple models obtained by the aggregation node can include models deployed on the aggregation node itself, i.e., models local to the aggregation node.

[0176] For example, such as Figure 7As shown, when the aggregation node is the central node in a centralized system architecture, the aggregation node can obtain the above multiple models from the first node to the Nth node; when the aggregation node is the central node in a distributed system architecture, the aggregation node can obtain the above multiple models from the aggregation node itself and the first node to the Nth node.

[0177] It should be noted that the multiple models obtained by the aggregation node are used to perform the same processing task, such as image processing or natural language processing tasks. Image processing tasks may include image classification, image recognition, and image enhancement. Natural language processing tasks may include text recognition, text generation, speech recognition, and text translation.

[0178] Having the same network structure for two models means that both models include the same neural network layers and have the same connections between those layers. However, the weight parameters of the neural network layers in models with the same network structure may differ. For example, consider multiple models used to perform image classification tasks. These models can all be convolutional neural network structures, with some models having the same structure and others having different structures.

[0179] Step 702: Based on the network structure of multiple models, the aggregation node divides the multiple models into multiple groups, where each group includes at least one model, and the models in the same group have the same network structure.

[0180] Since the aggregation node obtains multiple models from multiple different nodes, and the network structures of the models sent to the aggregation node by different nodes may be the same or different, the multiple models received by the aggregation node may correspond to multiple network structures; that is, different models may have different network structures.

[0181] For example, please refer to Figure 8 , Figure 8 This is a schematic diagram illustrating an aggregation node receiving multiple models, as provided in an embodiment of this application. Figure 8 As shown, the aggregation node receives models from nodes 1 through n. Node 1 sends a model to the aggregation node, node 2 sends model 2, node 3 sends model 3, and so on, with node n sending model n. Model 1 sent by node 1 and model 3 sent by node 3 have the same network structure, as do model 2 sent by node 2 and model n sent by node n. However, model 1 and model 2 have different network structures.

[0182] In this embodiment, the aggregation node can divide multiple models into multiple groups based on the network structure of each model. During the grouping process, models with the same network structure are grouped into the same group, and models with different network structures are grouped into different groups. Therefore, in the resulting multiple groups, each group includes at least one model, and the models in the same group have the same network structure.

[0183] For example, please refer to Figure 9 , Figure 9 This is a schematic diagram illustrating grouping and parameter aggregation of multiple models, as provided in an embodiment of this application. Figure 9 As shown, the models 1-n obtained by the aggregation node correspond to M network structures. Therefore, the aggregation node can divide the network structures of models 1-n into M groups (i.e., group 1-group M). Among them, models 1 and 3 with the same network structure are assigned to group 1, and models 2 and n with the same network structure are assigned to group 2. Of course, group 1 may also include other models with the same network structure as models 1 and 3; group 2 may also include other models with the same network structure as models 2 and n. This embodiment does not specifically limit this.

[0184] Step 703: The aggregation node performs parameter aggregation on the models in each of the multiple groups to obtain multiple aggregated models. The models in the same group are aggregated to obtain one aggregated model, and the multiple aggregated models correspond to multiple groups.

[0185] In this embodiment, since models within the same group have the same network structure, they often include the same number of weight parameters. Therefore, an aggregation node can aggregate the parameters of multiple models within the same group to obtain an aggregated model. Thus, by aggregating the parameters of models in each of multiple groups, an aggregation node can obtain multiple aggregated models, each corresponding to one group; that is, multiple aggregated models correspond one-to-one with multiple groups. Furthermore, the network structure of each aggregated model is the same as the network structure of the models in its corresponding group.

[0186] Taking the first group out of multiple groups as an example, the process of parameter aggregation for the models in the first group can specifically include: weighted summation of the weight parameters of multiple models in the first group to obtain a first aggregated model; where the first group belongs to multiple groups, and the first aggregated model belongs to multiple aggregated models. Since the multiple models in the first group have the same network structure, the network structure positions with weight parameters in these multiple models are also the same, and the network structure positions with weight parameters in the first aggregated model obtained by aggregating multiple models are also the same. Therefore, the weight parameter value at any network structure position in the first aggregated model can be obtained by weighted summation of the weight parameters of multiple models at that network structure position.

[0187] For example, suppose the network structure positions with weight parameters in multiple models in the first group include positions 1 to M. Since the first aggregated model has the same network structure as the multiple models in the first group, the network structure positions with weight parameters in the first aggregated model also include positions 1 to M. Furthermore, the weight parameter at position 1 of the first aggregated model is obtained by weighted summation of the weight parameters at position 1 of multiple models; the weight parameter at position 2 of the first aggregated model is obtained by weighted summation of the weight parameters at position 2 of multiple models; and so on. The weight parameter at each position in the first aggregated model is obtained by weighted summation of the weight parameters at the same position of multiple models.

[0188] In general, the network structure of each aggregation model is the same as that of the models in its corresponding group, and the weight parameters in each aggregation model can be obtained by weighted summation of the weight parameters of one or more models in the same group.

[0189] For example, such as Figure 9 As shown, Model 1 and Model 3 in Group 1 are aggregated to obtain Aggregated Model 1. The network structure of Aggregated Model 1 is the same as that of Model 1 and Model 3, and the weight parameters of Aggregated Model 1 are obtained by weighted summation of the weight parameters of Model 1 and Model 3. Similarly, Model 2 and Model n in Group 2 are aggregated to obtain Aggregated Model 2. The network structure of Aggregated Model 2 is the same as that of Model 2 and Model n, and the weight parameters of Aggregated Model 2 are obtained by weighted summation of the weight parameters of Model 2 and Model n.

[0190] Step 704: Using multiple models as teacher models, the aggregation node performs knowledge distillation training on each of the multiple aggregation models to obtain multiple trained models.

[0191] Since each aggregation model is obtained by aggregating models with the same network structure in the same group, each aggregation model actually only integrates the knowledge learned by models with the same network structure, and does not integrate the knowledge of models with different network structures.

[0192] Therefore, in this embodiment, the multiple models originally received by the aggregation node are used as teacher models. Knowledge distillation training is performed on each aggregation model to achieve knowledge transfer between models with different network structures, ensuring that each aggregation model can learn knowledge from models with other different network structures during knowledge distillation training. In this way, the trained model obtained by performing knowledge distillation training on the aggregation model actually integrates the knowledge and experience between models with different network structures, which can effectively improve the prediction accuracy of the trained model.

[0193] To facilitate understanding, the following will take the knowledge distillation training performed on the first aggregation model among multiple aggregation models by the aggregation node as an example to explain in detail the process of knowledge distillation training performed on the aggregation model.

[0194] Specifically, during the knowledge distillation training of the first aggregation model, training samples are first input into the first aggregation model and multiple models respectively to obtain the target prediction result and multiple prediction results. The first aggregation model can be any one of the multiple aggregation models, the target prediction result is the prediction result output by the first aggregation model, and the multiple prediction results are the prediction results output by the aforementioned multiple models respectively. The training samples can be local data and / or public data obtainable on the aggregation node.

[0195] Then, based on the target prediction result output by the first aggregation model and the multiple prediction results output by multiple models, a loss function can be constructed. This loss function can be derived based on the difference between the target prediction result and the multiple prediction results. Thus, the first aggregation model is trained based on the loss function, resulting in the trained first aggregation model.

[0196] In this scheme, a distillation loss function is constructed based on the difference between the prediction results of the original multiple models and the prediction results of the aggregated model. This enables the knowledge distillation method to support the transfer of experience between models with different network structures, allowing the aggregated model to further aggregate knowledge and experience under various network structures and improve the prediction accuracy of the aggregated model.

[0197] Optionally, the loss function includes a first sub-loss function and a second sub-loss function. The first sub-loss function is obtained based on the difference between the target prediction result and multiple prediction results, while the second sub-loss function is obtained based on the difference between the true label of the training sample and the target prediction result. For example, assuming the first aggregation model is used to perform an image classification task, the training samples are images, the true label of the training samples is the actual category of the training samples, and the target prediction result is the category prediction result output by the first aggregation model.

[0198] Furthermore, the loss function can be obtained by weighted summation of the first sub-loss function and the second sub-loss function. The weights of the first and second sub-loss functions can be determined or adjusted based on the actual training of the model; this embodiment does not impose specific limitations on this.

[0199] In this scheme, a distillation loss function is constructed based on the difference between the prediction results of the original multiple models and the prediction results of the aggregated model, and a label loss function is constructed based on the real labels of the training samples. Under the constraint of the label loss function, knowledge distillation is used to support the transfer of experience between models with different network structures, so that the aggregated model can effectively aggregate knowledge and experience under various network structures and improve the prediction accuracy of the aggregated model.

[0200] The first loss function mentioned above can be constructed in various ways.

[0201] In one possible implementation, the first loss function is derived based on the difference between the target prediction and each of the multiple predictions. That is, the first loss function can actually be obtained by a weighted sum of multiple difference values, including the difference between the target prediction and each of the multiple predictions.

[0202] In another possible implementation, the first loss function is derived from the difference between the target prediction and the weighted average of multiple predictions. That is, after obtaining multiple predictions, a weighted sum can be calculated, with each prediction having the same weight, to obtain a weighted average. Then, the difference between the target prediction and the weighted average of the multiple predictions is calculated to obtain the first loss function. For example, when the first aggregation model is used to perform an image classification task, the multiple predictions output by the multiple models are actually prediction probabilities for each category (i.e., each prediction includes prediction probabilities for each category). Therefore, the process of weighted summing of multiple predictions is actually weighted summing of the prediction probability values ​​for each category, thus obtaining a weighted summed probability value for each category.

[0203] For example, please refer to Figure 10 , Figure 10 This is a schematic diagram illustrating knowledge distillation of multiple aggregation models, provided as an embodiment of this application. Figure 10 As shown, the aggregation node obtains aggregated models 1-M with different network structures based on the originally received models 1-n. Furthermore, for each aggregated model in aggregated models 1-M, the aggregation node performs knowledge distillation on each aggregated model based on the originally received models 1-n, thereby obtaining the trained models 1-M.

[0204] Step 705: The aggregation node sends multiple trained models to multiple nodes.

[0205] After the aggregation node performs knowledge distillation training on each aggregation model, it can obtain multiple trained models. These multiple trained models are actually obtained by integrating models based on different network structures. Therefore, the aggregation node can send multiple trained models to multiple nodes, allowing multiple nodes to select one or more models to continue local training.

[0206] For example, such as Figure 10 As shown, after performing knowledge distillation on each aggregation model, the aggregation point obtains the trained model 1 to the trained model M, and sends the trained model 1 to the trained model M to each node in node 1 to node n.

[0207] Understandably, since the aggregation node still obtains multiple models with different network structures after aggregating the models uploaded by various nodes (i.e., multiple trained models as mentioned above), if the aggregation node sends multiple trained models to each node, there may be a large amount of data communication. Especially in the scenario of federated learning, the aggregation node and other nodes often need to interact with the model in multiple rounds, further increasing the amount of data communication.

[0208] Based on this, the aggregation node can pre-acquire some data from each node or data similar to the local data on each node, and then use this data to select a portion of the models for each node, avoiding sending too many models to each node.

[0209] For example, the aggregation node may determine a subset of trained models from multiple trained models based on the target data and send this subset to the first node. The target data is the data obtained by the aggregation node from the first node, which is one of multiple nodes. The subset of trained models is the model with the best performance when processing the target data. The number of subsets can be determined or adjusted based on the processing capacity of the first node, and is not specifically limited here.

[0210] Specifically, the first node may pre-select a portion of local data as target data and send this target data to the aggregation node. The target data selected by the first node can be typical data from the first node, effectively representing the characteristics of the local data on the first node.

[0211] Alternatively, if the first node does not wish to transmit its actual data to the aggregator node, the first node can train a data generator based on its local data. This data generator can output data similar to the local data on the first node even without any input data. In this way, after the first node sends the data generator to the aggregator node, the aggregator node can generate the target data described above based on the data generator.

[0212] For example, please refer to Figure 11 , Figure 11 This is a schematic diagram illustrating how an aggregation node sends a trained model to various nodes, as provided in an embodiment of this application. Figure 11 As shown, the aggregation node can pre-acquire data from various nodes and, based on this data, select a portion of the trained models for each node. It then sends the selected portion of the trained models to each node, instead of sending all the trained models to each node. For example, the aggregation node only sends trained model 1 and trained model 2 to node 1, and only sends trained model 2 and trained model n to node 2.

[0213] Step 706: The first node receives multiple trained models from the aggregation node.

[0214] The first node is one of the multiple nodes mentioned above. These multiple trained models have different network structures, and the multiple models are obtained by aggregating the models on different nodes.

[0215] Specifically, these multiple trained models are obtained by the first node after receiving models from different nodes, aggregating the parameters of models with the same network structure, and performing knowledge distillation on the aggregated models based on the models from different nodes.

[0216] Step 707: The first node determines the target model from multiple trained models based on the training data.

[0217] Since the first node receives multiple trained models from the aggregation node, and the performance of these multiple trained models on the training data in the first node is different, the first node can determine the target model suitable for processing the local data of the first node from the multiple trained models based on its local training data (i.e., select the target model that is suitable for the local data of the first node).

[0218] The target model determined by the first node can include one or more trained models. If the processing power of the first node is poor, the first node can determine one trained model as the target model; if the processing power of the first node is strong, the first node can determine multiple trained models as the target models.

[0219] Specifically, there are several ways for the first node to determine the target model among multiple trained models.

[0220] In one possible implementation, the first node can input training data into a gate network, obtaining multiple weight values ​​output by the gate network. These weight values ​​correspond one-to-one with multiple trained models, and each weight value represents a weight in the weighted summation process of the outputs of the multiple trained models. In other words, the multiple trained models work in conjunction with the gate network to process the training data. The gate network outputs weight values ​​corresponding to each of the multiple trained models based on the training data. Thus, after each trained model outputs its corresponding training data output, the multiple outputs can be weighted and summed according to their respective weight values ​​to obtain the final output.

[0221] The gate network can be a network pre-trained based on the local data of the first node. During the training of the gate network, the first node can simultaneously input local data into the gate network and multiple trained models, and the gate network outputs the weight values ​​corresponding to each trained model. Thus, after each trained model outputs its training data, the multiple outputs are weighted and summed according to the weight values ​​of each trained model to obtain the final output. Therefore, the weight values ​​output by the gate network reflect the performance of multiple trained models on the local data of the first node. For any trained model, a higher weight value output by the gate network indicates better performance of the trained model on the local data of the first node, meaning the trained model is more well-suited to the local data of the first node; conversely, a lower weight value output by the gate network indicates worse performance of the trained model on the local data of the first node, meaning the trained model is less well-suited to the local data of the first node.

[0222] In this way, the first node can determine the target model based on the multiple weight values ​​output by the gate network. The target model includes one or more models with the highest corresponding weight values ​​among the multiple trained models. In other words, the first node can measure the degree of fit of each trained model to the local data of the first node by the weight values ​​corresponding to each trained model, and then select one or more models with the highest degree of fit (i.e., the target model) from the multiple trained models.

[0223] For example, please refer to Figure 12A and Figure 12B , Figure 12A A schematic diagram of a target screening model provided in an embodiment of this application; Figure 12B This is a schematic diagram of a target screening model provided in an embodiment of this application. Figure 12A and Figure 12B As shown, after obtaining the trained model 1 to trained model M, the first node can construct an overall model based on the gate network and the trained model 1 to trained model M. The input data is simultaneously input into the gate network and the trained model 1 to trained model M, and the gate network outputs the weight values ​​corresponding to the trained model 1 to trained model M. Then, the output results of each trained model are weighted and summed based on the weight values ​​to obtain the final output result.

[0224] exist Figure 12A In this process, after training the gate network based on the local data of the first node, the target model can be selected based on the weight values ​​output by the gate network, resulting in Model 1 to Model K. That is, in... Figure 12AIn this context, the target model can include multiple models.

[0225] exist Figure 12B In this process, after training the gate network using the local data from the first node, the target model can be selected based on the weight values ​​output by the gate network to obtain model K. That is, in Figure 12B In this context, the target model can consist of only one model.

[0226] In this scheme, the weight values ​​corresponding to each model are output by training a gate network, thereby measuring the degree of adaptation of each model to local data and ensuring that the node can select the target model with the best performance when processing the node's local data from multiple models.

[0227] In another possible implementation, the first node can input the training data into each of the multiple trained models, and determine the prediction accuracy of each trained model based on the prediction results output by each trained model. For example, the first node can prepare multiple training samples and input these multiple training samples into multiple trained models, thereby obtaining the prediction accuracy of each trained model based on the prediction accuracy of the multiple trained models for each training sample.

[0228] Then, the first node determines the target model based on the prediction accuracy of each trained model. The target model is one or more models with the highest prediction accuracy among multiple trained models.

[0229] For example, please refer to Figure 13 , Figure 13 This is a schematic diagram of another target screening model provided in an embodiment of this application. For example... Figure 13 As shown, after obtaining the trained model 1 to the trained model M, the first node can input the training data into each trained model to obtain the prediction accuracy corresponding to each trained model. Then, based on the prediction accuracy of the model, the target model is selected to obtain the model K with the highest prediction accuracy.

[0230] In this scheme, the prediction accuracy of each model is determined based on the local data of the node, thereby measuring the degree of adaptation of each model to the local data and ensuring that the node can select the target model with the best performance when processing the local data of the node from multiple models.

[0231] Step 708: The first node uses multiple trained models as teacher models to perform knowledge distillation training on the target model to obtain the trained target model.

[0232] The process of the first node performing knowledge distillation training on the target model is similar to the process of the aggregation node performing knowledge distillation training on the aggregation model in step 704 above. The difference is that the first node uses local data on its own data to perform knowledge distillation training on the target model, while the aggregation node uses local data on its own data and / or public data obtained by the aggregation node to perform knowledge distillation training on the aggregation model. For details, please refer to step 704 above, which will not be repeated here.

[0233] Furthermore, when the target model comprises multiple models, it also needs to work with a gate network to perform data processing. Therefore, during the knowledge distillation training of the target model, the corresponding gate network is simultaneously trained to ensure that the gate network can correctly output the weight values ​​corresponding to each model in the target model.

[0234] In this embodiment, after the first node obtains the trained target model, it can deploy the trained target model on the first node to perform inference tasks based on the local data on the first node.

[0235] For example, such as Figure 12A and Figure 12B As shown, after determining the target model, knowledge distillation training (i.e., fine-tuning the target model based on local data) can be performed on the target model based on the local data of the first node to achieve further training of the target model.

[0236] In this scheme, after the first node selects the target model, it uses the original received model as the teacher model to perform knowledge distillation training on the target model. This ensures that the target model does not forget its original experience, based on the consideration of performing personalized fine-tuning on the local data of the first node. This effectively avoids the drastic fluctuation of the target model's prediction accuracy due to the poor quality of the local data of the first node (i.e., avoids the phenomenon of overfitting of the target model).

[0237] It is understandable that steps 701-708 above actually describe one round of iterative process in federated learning. In practical applications, federated learning can involve multiple rounds of iterative processes. That is, after each node trains the target model, it sends the trained target model to the aggregation node, which then aggregates the models uploaded by each node, thus executing a new round of iterative process.

[0238] The above describes how the aggregation node aggregates the models uploaded by various nodes, distributes the aggregated models to each node, and then each node further filters and trains the aggregated models.

[0239] In a distributed system architecture, since the aggregation node itself also deploys models and needs models to process local data, in some embodiments, after aggregating multiple models, the aggregation node can further select models suitable for the aggregation node and then train the selected models.

[0240] For example, please refer to Figure 14 , Figure 14 This is a flowchart illustrating another federated learning method provided in an embodiment of this application. Figure 14 As shown, the federated learning method includes the following steps 1401-1407.

[0241] Step 1401: The aggregation node obtains multiple models, which are obtained from multiple nodes.

[0242] In this embodiment, the aggregation node is a node in a distributed system architecture, and models are deployed on the aggregation node. Therefore, the multiple models obtained by the aggregation node also include the models deployed on the aggregation node.

[0243] Step 1402: Based on the network structure of multiple models, the aggregation node divides the multiple models into multiple groups, where each group includes at least one model, and the models in the same group have the same network structure.

[0244] Step 1403: The aggregation node performs parameter aggregation on the models in each of the multiple groups to obtain multiple aggregated models. The models in the same group are aggregated to obtain one aggregated model, and the multiple aggregated models correspond to multiple groups.

[0245] Step 1404: Using multiple models as teacher models, the aggregation node performs knowledge distillation training on each of the multiple aggregation models to obtain multiple trained models.

[0246] Step 1405: The aggregation node sends multiple trained models to multiple nodes.

[0247] Steps 1401-1405 are similar to steps 701-705 above. Please refer to steps 701-705 above for details, which will not be repeated here.

[0248] Step 1406: The aggregation node determines the target model from multiple trained models based on the training data.

[0249] Since the aggregation node itself also needs to process the local data on the aggregation node based on the model, the aggregation node can also determine the target model from multiple trained models based on the training data on the aggregation node.

[0250] Step 1407: The aggregation node uses multiple trained models as teacher models to perform knowledge distillation training on the target model to obtain the trained target model.

[0251] Similarly, the process of the aggregation node determining the target model and performing knowledge distillation training on the target model is similar to steps 707-708 above. For details, please refer to the process of the first node determining the target model and performing knowledge distillation training on the target model in steps 707-708 above, which will not be repeated here.

[0252] The methods provided in the embodiments of this application have been described in detail above. Next, the device for performing the above methods provided in the embodiments of this application will be described.

[0253] Please see Figure 15 , Figure 15 This is a schematic diagram of a federated learning device provided as an embodiment of this application. Figure 15 As shown, this federated learning device belongs to the aggregation node and includes:

[0254] The acquisition module 1501 is used to acquire multiple models, which are obtained from multiple nodes;

[0255] The processing module 1502 is also used to divide the multiple models into multiple groups according to the network structure of the multiple models, wherein each group includes at least one model, and the models in the same group have the same network structure.

[0256] The processing module 1502 is also used to perform parameter aggregation on the models in each of the multiple groups to obtain multiple aggregated models, wherein the models in the same group are aggregated to obtain one aggregated model, and the multiple aggregated models correspond to multiple groups;

[0257] The processing module 1502 is also used to perform knowledge distillation training on each of the multiple aggregate models, using multiple models as teacher models, to obtain multiple trained models.

[0258] The sending module 1503 is used to send multiple trained models to multiple nodes.

[0259] In one possible implementation, the processing module 1502 is further configured to:

[0260] The training samples are input into the first aggregation model and multiple models respectively to obtain the target prediction result and multiple prediction results. The first aggregation model is any one of the multiple aggregation models. The target prediction result corresponds to the first aggregation model, and the multiple prediction results correspond to the multiple models.

[0261] The first aggregate model is trained based on the loss function, and the trained first aggregate model is obtained. The loss function is obtained based on the difference between the target prediction result and multiple prediction results.

[0262] In one possible implementation, the loss function is derived based on the difference between the target prediction and each of the multiple predictions.

[0263] Alternatively, the loss function can be derived from the difference between the target prediction and the weighted average of multiple predictions.

[0264] In one possible implementation, the loss function includes a first sub-loss function and a second sub-loss function. The first sub-loss function is obtained based on the difference between the target prediction result and multiple prediction results, and the second sub-loss function is obtained based on the difference between the true label of the training sample and the target prediction result.

[0265] In one possible implementation, the processing module 1502 is further configured to:

[0266] Based on the training data, the target model is determined from multiple trained models;

[0267] Using multiple trained models as teacher models, knowledge distillation training is performed on the target model to obtain an updated target model. The updated target model is then used to perform reasoning tasks based on local data.

[0268] In one possible implementation, the processing module 1502 is further configured to:

[0269] The training data is input into each of the multiple models, and the prediction accuracy of each model is determined based on the prediction results output by each model.

[0270] The target model is determined based on the prediction accuracy of each model. The target model includes one or more models with the highest prediction accuracy among multiple trained models.

[0271] In one possible implementation, the processing module 1502 is further configured to:

[0272] The training data is input into the gate network to obtain multiple weight values ​​output by the gate network. Each weight value corresponds to one of the multiple trained models. The multiple weight values ​​are used to represent the weights of the output results of the multiple trained models in the weighted summation process. The gate network is a pre-trained network.

[0273] The target model is determined based on multiple weight values. The target model includes one or more models with the highest corresponding weight values ​​among multiple trained models.

[0274] In one possible implementation,

[0275] The processing module 1502 is also used to determine a partial trained model from multiple trained models based on the target data. The target data is the data obtained from the first node, and the partial trained model is the model with the best performance in processing the target data among multiple trained models.

[0276] The sending module 1503 is also used to send part of the trained model to the first node.

[0277] In one possible implementation, the processing module 1502 is further configured to perform a weighted summation of the weight parameters of multiple models in the first group to obtain a first aggregated model;

[0278] Wherein, the first group belongs to the plurality of groups, and the first aggregation model belongs to the plurality of aggregation models.

[0279] Please see Figure 16 , Figure 16 This is a schematic diagram of a federated learning device provided as an embodiment of this application. Figure 16 As shown, the federated learning device includes:

[0280] The receiving module 1601 is used to receive multiple models from the aggregation node. The multiple models have different network structures, and the multiple models are obtained by the aggregation node aggregating the models on different nodes.

[0281] Processing module 1602 is used to determine the target model among multiple models based on training data;

[0282] The processing module 1602 is also used to perform knowledge distillation training on the target model using multiple models as teacher models to obtain the trained target model.

[0283] In one possible implementation, the processing module 1602 is further configured to:

[0284] The training data is input into each of the multiple models, and the prediction accuracy of each model is determined based on the prediction results output by each model.

[0285] The target model is determined based on the prediction accuracy of each model, and the target model is one or more models with the highest prediction accuracy among the plurality of models.

[0286] In one possible implementation, the processing module 1602 is further configured to:

[0287] The training data is input into the gate network to obtain multiple weight values ​​output by the gate network. Each weight value corresponds to one of the multiple trained models. Each weight value is used to represent the weight of the output of the multiple trained models in the weighted summation process. The gate network is a pre-trained network.

[0288] The target model is determined based on the plurality of weight values, wherein the target model includes one or more models with the highest corresponding weight values ​​among the plurality of models.

[0289] In one possible implementation, the multiple models are obtained by the aggregation node after receiving models from different nodes, performing parameter aggregation on models with the same network structure, and then training the aggregated model by knowledge distillation based on the models from the different nodes.

[0290] Please see Figure 17 , Figure 17 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1700 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited thereto. Specifically, the execution device 1700 includes: a receiver 1701, a transmitter 1702, a processor 1703, and a memory 1704 (wherein the execution device 1700 may have one or more processors 1703). Figure 17 (Taking a processor as an example), processor 1703 may include application processor 17031 and communication processor 17032. In some embodiments of this application, receiver 1701, transmitter 1702, processor 1703 and memory 1704 may be connected via a bus or other means.

[0291] Memory 1704 may include read-only memory and random access memory, and provides instructions and data to processor 1703. A portion of memory 1704 may also include non-volatile random access memory (NVRAM). Memory 1704 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0292] Processor 1703 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0293] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 1703. Processor 1703 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 1703 or by instructions in software form. Processor 1703 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 1703 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1704. Processor 1703 reads the information in memory 1704 and, in conjunction with its hardware, completes the steps of the above method.

[0294] Receiver 1701 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1702 can be used to output digital or character information through the first interface; transmitter 1702 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1702 may also include a display device such as a display screen.

[0295] The electronic device provided in this application embodiment can specifically be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the execution device to execute the methods described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, such as random access memory (RAM).

[0296] For details, please refer to Figure 18 , Figure 18 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1800. The NPU 1800 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1803, which is controlled by the controller 1804 to extract matrix data from the memory and perform multiplication operations.

[0297] In some implementations, the arithmetic circuit 1803 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1803 is a two-dimensional pulsating array. The arithmetic circuit 1803 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1803 is a general-purpose matrix processor.

[0298] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1802 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1801 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1808.

[0299] Unified memory 1806 is used to store input and output data. Weight data is directly transferred to weight memory 1802 via Direct Memory Access Controller (DMAC) 1805. Input data is also transferred to unified memory 1806 via DMAC.

[0300] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1809.

[0301] The Bus Interface Unit (BIU) 1810 is used by the instruction fetch memory 1809 to fetch instructions from external memory, and also by the memory access controller 1805 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0302] The DMAC is mainly used to move input data from external memory DDR to unified memory 1806, or to weight data to weight memory 1802, or to input data to input memory 1801.

[0303] The vector computation unit 1807 includes multiple processing units that further process the output of the computation circuit 1803 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0304] In some implementations, the vector computation unit 1807 can store the processed output vector in the unified memory 1806. For example, the vector computation unit 1807 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1803, such as performing linear interpolation on feature planes extracted by a convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1807 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1803, for example, for use in subsequent layers of the neural network.

[0305] The instruction fetch buffer 1809 connected to the controller 1804 is used to store the instructions used by the controller 1804;

[0306] Unified memory 1806, input memory 1801, weighted memory 1802, and instruction fetch memory 1809 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0307] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0308] Please refer to Figure 19 , Figure 19 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this application. This application also provides a computer-readable storage medium in some embodiments, wherein the above-described... Figure 6 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0309] Figure 19 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0310] In one embodiment, the computer-readable storage medium 1900 is provided using a signal bearer medium 1901. The signal bearer medium 1901 may include one or more program instructions 1902, which, when executed by one or more processors, can provide the above-mentioned... Figure 6 The described function or part of the function.

[0311] In some examples, the signal carrying medium 1901 may include a computer-readable medium 1903, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video optical disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0312] In some embodiments, the signal-bearing medium 1901 may include a computer-recordable medium 1904, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal-bearing medium 1901 may include a communication medium 1905, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.). Therefore, for example, the signal-bearing medium 1901 may be transmitted by a wireless communication medium 1905 (e.g., a wireless communication medium conforming to the IEEE 802.X standard or other transmission protocols).

[0313] One or more program instructions 1902 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1902 conveyed to the computing device via a computer-readable medium 1903, a computer-recordable medium 1904, and / or a communication medium 1905.

[0314] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0315] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0316] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0317] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A federated learning method, characterized in that, include: Multiple models are obtained from multiple nodes; Based on the network structure of the multiple models, the multiple models are divided into multiple groups, wherein each group includes at least one model, and the models in the same group have the same network structure; The models in each of the multiple groups are aggregated to obtain multiple aggregated models, wherein models in the same group are aggregated to obtain one aggregated model, and the multiple aggregated models correspond to the multiple groups; Using the multiple models as teacher models, knowledge distillation training is performed on each of the multiple aggregate models to obtain multiple trained models; Send the multiple trained models to the multiple nodes; Sending the multiple trained models to the multiple nodes includes: Based on the target data, a subset of trained models are determined from the plurality of trained models. The target data is the data obtained from the first node, and the subset of trained models is the model with the best performance in processing the target data among the plurality of trained models. Send the partially trained model to the first node.

2. The method according to claim 1, characterized in that, The process involves using the multiple models as teacher models, and performing knowledge distillation training on each of the multiple aggregate models to obtain multiple trained models, including: The training samples are input into the first aggregation model and the plurality of models respectively to obtain the target prediction result and the plurality of prediction results, wherein the first aggregation model is any one of the plurality of aggregation models, the target prediction result corresponds to the first aggregation model, and the plurality of prediction results correspond to the plurality of models; The first aggregation model is trained based on the loss function to obtain the trained first aggregation model. The loss function is obtained based on the difference between the target prediction result and the multiple prediction results.

3. The method according to claim 2, characterized in that, The loss function is obtained based on the difference between the target prediction result and each of the plurality of prediction results; Alternatively, the loss function is derived based on the difference between the target prediction result and the weighted average of the multiple prediction results.

4. The method according to claim 2 or 3, characterized in that, The loss function includes a first sub-loss function and a second sub-loss function. The first sub-loss function is obtained based on the difference between the target prediction result and the plurality of prediction results, and the second sub-loss function is obtained based on the difference between the true label of the training sample and the target prediction result.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: Based on the training data, the target model is determined from the plurality of trained models; Using the multiple trained models as teacher models, knowledge distillation training is performed on the target model to obtain an updated target model, which is then used to perform reasoning tasks based on local data.

6. The method according to claim 5, characterized in that, The step of determining the target model from the plurality of trained models based on training data includes: The training data is input into each of the multiple models, and the prediction accuracy of each model is determined based on the prediction results output by each model. The target model is determined based on the prediction accuracy of each model, and the target model includes one or more models with the highest prediction accuracy among the plurality of trained models.

7. The method according to claim 5, characterized in that, The step of determining the target model from the plurality of trained models based on training data includes: The training data is input into the gate network to obtain multiple weight values ​​output by the gate network. Each weight value corresponds to one of the multiple trained models. Each weight value is used to represent the weight of the output of the multiple trained models in the weighted summation process. The gate network is a pre-trained network. The target model is determined based on the plurality of weight values, wherein the target model includes one or more models with the highest corresponding weight values ​​among the plurality of trained models.

8. The method according to any one of claims 1-3, characterized in that, The parameter aggregation of the models in each of the multiple groups yields multiple aggregated models, including: The weight parameters of multiple models in the first group are weighted and summed to obtain the first aggregate model; Wherein, the first group belongs to the plurality of groups, and the first aggregation model belongs to the plurality of aggregation models.

9. A federated learning method, characterized in that, include: Send the target data to the aggregation node; The aggregation node receives a partially trained model, which is determined from multiple trained models based on the target data. The partially trained model is the model with the best performance in processing the target data among the multiple trained models. The multiple trained models are obtained by the aggregation node after receiving models from different nodes, aggregating the parameters of models with the same network structure, and training the aggregated model with knowledge distillation based on the models from the different nodes.

10. A federated learning device, characterized in that, include: The acquisition module is used to acquire multiple models, which are obtained from multiple nodes; The processing module is further configured to divide the multiple models into multiple groups according to the network structure of the multiple models, wherein each group includes at least one model, and the models in the same group have the same network structure; The processing module is further configured to perform parameter aggregation on the models in each of the multiple groups to obtain multiple aggregated models, wherein models in the same group are aggregated to obtain one aggregated model, and the multiple aggregated models correspond to the multiple groups; The processing module is further configured to use the plurality of models as teacher models to perform knowledge distillation training on each of the plurality of aggregate models to obtain a plurality of trained models; The processing module is further configured to determine a subset of trained models from among the plurality of trained models based on the target data, wherein the target data is data obtained from the first node, and the subset of trained models is the model with the best performance in processing the target data among the plurality of trained models. The sending module is used to send the partially trained model to the first node.

11. The apparatus according to claim 10, characterized in that, The processing module is further configured to: The training samples are input into the first aggregation model and the plurality of models respectively to obtain the target prediction result and the plurality of prediction results, wherein the first aggregation model is any one of the plurality of aggregation models, the target prediction result corresponds to the first aggregation model, and the plurality of prediction results correspond to the plurality of models; The first aggregation model is trained based on the loss function to obtain the trained first aggregation model. The loss function is obtained based on the difference between the target prediction result and the multiple prediction results.

12. The apparatus according to claim 11, characterized in that, The loss function is obtained based on the difference between the target prediction result and each of the plurality of prediction results; Alternatively, the loss function is derived based on the difference between the target prediction result and the weighted average of the multiple prediction results.

13. The apparatus according to claim 11 or 12, characterized in that, The loss function includes a first sub-loss function and a second sub-loss function. The first sub-loss function is obtained based on the difference between the target prediction result and the plurality of prediction results, and the second sub-loss function is obtained based on the difference between the true label of the training sample and the target prediction result.

14. The apparatus according to any one of claims 10-12, characterized in that, The processing module is further configured to: Based on the training data, the target model is determined from the plurality of trained models; Using the multiple trained models as teacher models, knowledge distillation training is performed on the target model to obtain an updated target model, which is then used to perform reasoning tasks based on local data.

15. The apparatus according to claim 14, characterized in that, The processing module is further configured to: The training data is input into each of the multiple models, and the prediction accuracy of each model is determined based on the prediction results output by each model. The target model is determined based on the prediction accuracy of each model, and the target model includes one or more models with the highest prediction accuracy among the plurality of trained models.

16. The apparatus according to claim 14, characterized in that, The processing module is further configured to: The training data is input into the gate network to obtain multiple weight values ​​output by the gate network. Each weight value corresponds to one of the multiple trained models. Each weight value is used to represent the weight of the output of the multiple trained models in the weighted summation process. The gate network is a pre-trained network. The target model is determined based on the plurality of weight values, wherein the target model includes one or more models with the highest corresponding weight values ​​among the plurality of trained models.

17. A federated learning device, characterized in that, include: The sending module is used to send target data to the aggregation node; The receiving module is used to receive a partially trained model from the aggregation node. The partially trained model is determined from multiple trained models based on the target data. The partially trained model is the model with the best performance in processing the target data among the multiple trained models. The multiple trained models are obtained by the aggregation node after receiving models from different nodes, aggregating the parameters of models with the same network structure, and training the aggregated model with knowledge distillation based on the models from the different nodes.

18. A federated learning device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the device performs the method as described in any one of claims 1 to 9.

19. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 9.

20. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 9.

Images