Reversed knowledge distillation method and system based on federated large model

Abstract

The present invention provides a method and system for reverse knowledge distillation based on a federated large model, relating to the field of federated learning. The method includes: a client uses a private dataset to update the weight parameters of a small-parameter model to obtain a client model; a public dataset is used to output the client model soft logic and send it to a server; the server's large-parameter model uses the public dataset to output the server model soft logic; the weight of each client is calculated based on the correlation between the server model soft logic and the soft logic of multiple client models and the accuracy of each client model on the public dataset; a weighted client soft logic is calculated based on the client weight and the client model soft logic; and knowledge distillation is performed using the server model soft logic and the weighted client soft logic. The present invention uses multiple small-parameter models to perform knowledge distillation on a single large-parameter model, providing a new method for updating and iterating large-parameter models within a server.

Description

Technical Field

[0001] The present invention relates to the field of federated learning, and in particular to a method and system for reverse knowledge distillation based on a federated large model. Background Art

[0002] With the emergence and gradual maturity of large models, their reliance on ever-larger training data and data privacy have given rise to a new challenge: how to jointly train a large model using isolated, private data without violating data privacy laws. A large-scale model training architecture and method based on federated learning has emerged to address the distributed training of large models. Federated learning trains models in a distributed manner, eliminating the need to centralize data, effectively protecting user privacy. Knowledge distillation, on the other hand, transfers the knowledge of complex models to smaller models, achieving lightweight models and efficient inference. In edge computing scenarios, combining federated learning and knowledge distillation can improve model performance on edge devices while ensuring data privacy, making intelligent applications more efficient and convenient. Federated knowledge distillation is an advanced technology that combines federated learning and knowledge distillation. It improves overall performance and accuracy by collaboratively training models in a distributed environment. In this framework, different devices and nodes can share their local model knowledge without directly sharing raw data, thereby protecting data privacy. Furthermore, knowledge distillation transfers the knowledge of complex models to lighter models, optimizing inference efficiency and resource consumption. This method is particularly suitable for scenarios with high requirements for computing efficiency, such as the healthcare and financial industries, to achieve efficient and accurate intelligent applications.

[0003] In the process of implementing the present invention, the applicant discovered that the traditional knowledge distillation strategy is to transfer knowledge from a trained large model to a small or untrained model, often ignoring the contribution of the small model with personalized data to the large model. These traditional strategies cannot allow the large model to fully learn on specific datasets in certain vertical fields, resulting in suboptimal performance and personalized user experience. Previous research still has a significant limitation. Most studies are limited to federated knowledge distillation for models with the same architecture, and do not consider collaborative training between heterogeneous client devices with limited computing resources.

[0004] Therefore, how to improve the training efficiency and performance of large-parameter models on the server side becomes a technical problem that needs to be solved. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art or related technology, and discloses a reverse knowledge distillation method and system based on a federated large model. By transferring knowledge to a large parameter model through multiple small parameter models, the present invention can reduce network resource consumption, improve accuracy, reduce training rounds, and enhance user experience.

[0006] Terminology: Federated learning is a distributed machine learning technology that allows multiple data owners to build machine learning models without sharing data. The goal of federated learning is to enable collaborative modeling and improve the effectiveness of AI models while ensuring data privacy, security, and legal compliance. Knowledge distillation is a model compression technique that trains a small neural network (called the "student model") to mimic a large, pre-trained neural network (called the "teacher model"). This method can be viewed as a transfer of knowledge from the teacher model to the student model, hence the name "knowledge distillation." The first aspect of the present invention discloses a reverse knowledge distillation method based on a federated large model, including: a server sends a small parameter model to a local client of the client, so that the client uses a private data set to update the weight parameters of the small parameter model to obtain a client model; the client model uses a public data set to output the client model soft logic, and sends the client model soft logic to the server; the server's large parameter model uses the public data set to output the server model soft logic; the weight of each client is calculated based on the correlation between the server model soft logic and the soft logic of multiple client models and the accuracy of each client model for the public data set; the weighted client soft logic is calculated based on the client weight and the client model soft logic; and the server model soft logic and the weighted client soft logic are used to perform knowledge distillation to update the weight parameters of the server's large parameter model.

[0007] In this technical solution, a federated knowledge distillation method is disclosed in which multiple client small-parameter models transfer knowledge to a single server large-parameter model. The present invention defines this technical framework as reverse knowledge distillation. Due to the limitation of client computing resources, the server sends a compressed version of the large-parameter model, i.e., the small-parameter model, to the client. The client accepts the small-parameter model from the server and trains it using a private data set. The client model soft logic output is to output a soft logic for the public image data set using the trained client model. The client weight calculation is to calculate the weight of each client in the process of federated knowledge distillation. The sum of the client weights is 1. The server model knowledge distillation refers to the server obtaining the client weighted soft logic and the server soft logic, and performing the knowledge distillation process on the server model to update the server model.

[0008] According to the reverse knowledge distillation method based on the federated large model disclosed in the present invention, preferably, the knowledge distillation specifically includes: calculating the distillation loss between the server model soft logic and the weighted client soft logic; performing weighted summation of the distillation loss and the task loss of the server model to obtain the total loss of the server model; backpropagating the server model according to the total loss of the server model, thereby updating the server model in the same manner as the client model is updated. The task loss of the server model refers to the gap between the prediction output by the server model for the data set and the actual prediction of the data set, which is usually calculated using the cross-entropy loss function. According to the reverse knowledge distillation method based on the federated large model disclosed in the present invention, preferably, the small parameter model is compressed from the large parameter model of the server.

[0009] According to the federated large model-based inverse knowledge distillation method disclosed in the present invention, preferably, the public dataset consists of data provided by various clients.

[0010] According to the inverse knowledge distillation method based on the federated large model disclosed in the present invention, preferably, the client model soft logic specifically includes: using a softmax normalization function as the soft logic output function of the client model.

[0011] According to the inverse knowledge distillation method based on the federated large model disclosed by the present invention, preferably, the correlation refers to the cosine distance.

[0012] According to the inverse knowledge distillation method based on the federated large model disclosed in the present invention, preferably, the large parameter model of the server is a ResNet38 model, and the client model is a ResNet14 model.

[0013] In this technical solution, ResNet (Residual Network) is a deep learning model proposed by Microsoft Research in 2015. Its core concept is to introduce a residual learning framework and use skip connections to solve the problems of vanishing and exploding gradients in deep neural network training. The core of the ResNet model is its unique residual blocks. Common ResNet models include ResNet-14, ResNet-38, and ResNet-50. The numbers 14, 38, and 50 represent the number of residual blocks in the model. For example, ResNet-38 contains 38 residual blocks, and ResNet-14 contains 14 residual blocks. In the ResNet model, residual blocks are classified into two categories: basicblock and bottleblock. If the ResNet model uses basicblock, the number x of residual blocks in the ResNet model should satisfy the equation (x-2)% 6=0; if the ResNet model uses bottleblock, the number x of residual blocks in the ResNet model should satisfy the equation (x-2)% 9=0.

[0014] According to the inverse knowledge distillation method based on the federated large model disclosed in the present invention, preferably, the client is a personal computer or a handheld mobile device, and the server is a base station or an edge server.

[0015] The second aspect of the present invention discloses an inverse knowledge distillation system based on a federated large model, comprising: a memory for storing program instructions; a processor for calling the program instructions stored in the memory to implement an inverse knowledge distillation method based on a federated large model as any of the above technical solutions.

[0016] Practical application scenarios of the technical solution provided by the present invention: With the development of artificial intelligence technology, most cameras now have image recognition capabilities. However, in some cases, cameras are unable to effectively recognize certain objects that have never been seen. Through the method provided by the present invention, for certain unseen objects, only some cameras need to perform local updates and then perform a distillation process on the model in a server. In this way, when some cameras cannot recognize certain objects or a new camera is connected, a request can be sent to the server to obtain a new model without the need for multiple local updates. This reduces energy consumption, reduces camera latency, and improves performance.

[0017] The beneficial effects of the present invention include at least: the present invention takes into account the user privacy leakage problem that may be caused by the model aggregation process in traditional federated learning, deletes the model aggregation process, and improves the utilization rate of network resources. The technical solution provided by the present invention extends the concept of combining federated learning and knowledge distillation to the traditional network environment, and uses a unique method of multiple small parameter models to perform knowledge distillation on a single large parameter model, providing a new method for updating and iterating large parameter models in the server, and using federated learning and knowledge distillation to handle the requirements of large parameter models in the server for multiple data types. It can enable a well-performing model to run locally on the client while improving the performance of the server model on certain unknown data sets, and at the same time save service resource consumption by reducing the number of training rounds of the server model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flow chart of a reverse knowledge distillation method based on a federated large model according to an embodiment of the present invention is shown.

[0019] Figure 2 A schematic diagram of a client weight calculation process according to an embodiment of the present invention is shown.

[0020] Figure 3 A schematic diagram of a knowledge distillation workflow of a server model according to an embodiment of the present invention is shown.

[0021] Figure 4 A schematic block diagram of an inverse knowledge distillation system based on a federated large model according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0022] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0023] like Figure 1As shown, according to one embodiment of the present invention, a reverse knowledge distillation method based on a federated large model is disclosed, including: the server sends a small parameter model to the client; the client performs local training to form a trained client local model; calculates the client model accuracy and client model soft logic; determines the client weight; determines the client model soft logic based on the weights of each client to calculate the weighted client soft logic, and then uses the server model soft logic and the weighted client soft logic to perform a knowledge distillation process to update the weight parameters of the server large parameter model, thereby completing the training of the server model. Specifically, it includes the following steps:

[0024] Assume that there are a total of round, and in each communication round There are a total of 200 customer model training rounds Rounds, each training round is recorded as .

[0025] First For example, the following communication:

[0026] S1, after receiving the small parameter model sent from the server, the client uses the private dataset to update the weight parameters of the small parameter model;

[0027] S2: The trained client model uses the public image dataset to output the client model soft logic and sends the client model soft logic to the server;

[0028] S3: The server large parameter model outputs the server model soft logic for the public image dataset. The weight of each client is calculated based on the correlation between the server model soft logic and the soft logic of each client model and the accuracy of each client model for the public image dataset.

[0029] S4, server model knowledge distillation: Calculate weighted customer soft logic based on each customer weight and customer model soft logic, and then use the server model soft logic and weighted customer soft logic to perform knowledge distillation to update the weight parameters of the server large parameter model.

[0030] The above step S1 specifically includes: S1.1, receiving the small parameter model sent from the server ,in Is the ID of each customer, a total of Clients; S1.2, client-side small parameter model training uses client-side private dataset For small parameter models For training, the training process is as follows:

[0031] in It is the learning rate of the customer model update, which represents the degree of each customer model update. Represents the Locally trained customer small parameter model The weight parameter, Represents the Locally trained customer small parameter model The weight parameter, is the cross entropy loss function, Based on local datasets The loss function Model parameters The inverse of , that is, the gradient operation.

[0032] S1.3, after each customer model is trained, the customer model is recorded as , indicating that the customer model participating in the distillation process in each communication round is the customer model trained locally.

[0033] The above step S2 specifically includes:

[0034] S2.1, after each customer model is trained, it is used on the public image dataset Output the soft logic corresponding to each model , the output formula is as follows:

[0035]

[0036] in is the distillation temperature parameter, It is an activation function. The combined use of the two can soften the probability distribution of the customer model for the public image dataset, making the output of the customer model smoother, so that the customer output contains more information and reducing the probability of overfitting of the model. is the predicted output of the model, It is a public image dataset A sample of Represents the customer model The weight parameter of .

[0037] S2.2, after each customer model is trained, it is used on the public image dataset Output the accuracy of each model , the output formula is as follows:

[0038]

[0039] in is the cross entropy loss function, It is a public image dataset Data sample The true label.

[0040] S2.3, each customer gets his own customer model Public image datasets Soft Logic and accuracy Afterwards, and Send to the server.

[0041] like Figure 2 As shown, the above step S3 specifically includes:

[0042] S3.1, First Server Model Output to public image dataset Soft Logic , the output formula is as follows:

[0043]

[0044] in Is a server model The weight parameter, is the predicted output of the model, It is a public image dataset Sample.

[0045] S3.2, the server obtains the public image dataset from each client model Soft Logic With server model Soft Logic Then, the soft logic of each customer model is calculated based on the cosine distance function. Server Model Soft Logic The correlation between , the calculation formula is as follows: in is the cosine distance function.

[0046] S3.3, calculate the soft logic of each customer model With server model Soft Logic The correlation between After that, the customer weight calculation module will calculate the customer weight according to the received customer model. Public image datasets Accuracy and relevance The weight corresponding to each customer is calculated jointly. The calculation formula is as follows: in It is The weight corresponding to each customer, The accuracy of each customer and relevance The sum of .

[0047] like Figure 3 As shown, the above step S4 specifically includes:

[0048] S4.1, calculate the weight corresponding to each customer After that, the server will calculate the weight of each customer. With soft logic To calculate the weighted customer soft logic, the calculation formula is as follows: in The weight of each customer With soft logic The weighted sum of .

[0049] S4.2, after obtaining weighted customer soft logic After that, the server soft logic can be calculated With weighted customer soft logic The distillation loss during the distillation is calculated as follows:

[0050] in It is server soft logic With weighted customer soft logic Distillation loss, It is the KL divergence function, which is often used in conjunction with the cross entropy loss function in knowledge distillation to obtain a comprehensive loss function.

[0051] It is a public image dataset The number of categories in , and Weighted Customer Soft Logic Server soft logic No. categories.

[0052] S4.3, after obtaining the distillation loss, it can be concluded that in the framework of the present invention, the server model has The total loss in is calculated as follows:

[0053]

[0054] in and are the weights of the server model task loss and distillation loss, respectively.

[0055] S4.4, after obtaining the comprehensive loss, the server model can be updated in the same way as the client model is updated:

[0056] in Based on public image datasets The sum loss function Model parameters The inverse of , that is, the gradient operation. is the server's distillation learning rate.

[0057] According to another embodiment of the present invention, an implementation of the above-mentioned inverse knowledge distillation method based on the federated large model is also disclosed, including:

[0058] The server compresses the server model and sends it to each client according to the different devices of each client. At the same time, it receives data from each client to form a public image dataset and sends the public image dataset to each client.

[0059] After receiving the model from the server, the client trains the model using each client’s dataset;

[0060] After the customer completes training on the received model, they use the trained model to output the predicted probability for the public image dataset, which is the customer's soft logic.

[0061] The server accepts the soft logic of the public image dataset from all clients participating in the distillation process, and outputs its own prediction probability for the public image dataset, namely the server soft logic;

[0062] The server uses the server soft logic and each customer soft logic to calculate the proportion of each customer in the distillation process. Then, the server performs a weighted summation of each customer proportion and each customer soft logic to obtain the weighted customer soft logic.

[0063] The weighted client model soft logic and the server model soft logic are calculated to obtain the server model distillation loss in each communication round. The weighted summation with the task loss of the server model is used to obtain the total loss of the server model. The server model is back-propagated to update the server model.

[0064] According to another embodiment of the present invention, a specific implementation of the above-mentioned federated large model-based inversion knowledge distillation method in an actual application scenario is also disclosed, including the following steps:

[0065] Step 1: The server initializes the model and sends the specific model to the client:

[0066] The server initializes the server model: The model receives data input and outputs the corresponding distribution probability density. In this example, the server model is the ResNet38 model. A residual network model for multi-task learning (MTL) is defined, which includes several auxiliary classes and functions. The following is a detailed explanation of the classes and functions defined in the code:

[0067] MTL functions:

[0068] Function: Initialize and return a multi-task learning residual network model.

[0069] parameter:

[0070] class_num: The number of categories for the classification task.

[0071] pretrained: Boolean value indicating whether to use the pre-trained model. The default value is False.

[0072] path: The path of the pre-trained model, only used when pretrained=True.

[0073] **kwargs: Additional keyword arguments for passing to the model class.

[0074] Reshape class:

[0075] Function: Implement a reshape layer to reshape the input tensor into a specified shape.

[0076] parameter:

[0077] *args: An argument list containing the target shape.

[0078] method:

[0079] forward(self, x): Forward propagation method, reshapes the input x into the specified shape.

[0080] MTLResNet class:

[0081] Function: Implement a multi-task learning residual network model.

[0082] parameter:

[0083] block: The type of residual block, usually Bottleneck.

[0084] layers: A list specifying the number of layers for each residual block.

[0085] num_classes: The number of categories for the classification task, the default is 10.

[0086] groups: The number of groups for grouped convolution, the default is 1.

[0087] width_per_group: The width of each group, the default is 64.

[0088] method:

[0089] _make_layer(self, block, planes, blocks, stride=1, dilate=False) : Creates a residual block layer.

[0090] forward(self, x): Forward propagation method, calculates classification and discrimination outputs.

[0091] Bottleneck class:

[0092] Function: Implement a bottleneck residual block.

[0093] parameter:

[0094] inplanes: The number of channels of the input feature map.

[0095] planes: The number of channels of the convolutional layer.

[0096] stride: The step size of the convolution, the default is 1.

[0097] downsample: Optional parameter, used to adjust the dimension of the submodule.

[0098] groups: The number of groups for grouped convolution.

[0099] base_width: base width.

[0100] dilation: dilation rate of convolution.

[0101] norm_layer: Batch normalization layer, default is nn.BatchNorm2d.

[0102] conv3x3 function:

[0103] Function: Create a 3x3 convolutional layer with specified padding and dilation rates.

[0104] parameter:

[0105] in_planes: number of input channels.

[0106] out_planes: number of output channels.

[0107] stride: The step size of the convolution, the default is 1.

[0108] groups: The number of groups for grouped convolution.

[0109] dilation: dilation rate of convolution.

[0110] conv1x1 function:

[0111] Function: Create a 1x1 convolutional layer.

[0112] parameter:

[0113] in_planes: number of input channels.

[0114] out_planes: number of output channels.

[0115] stride: The step size of the convolution, the default is 1.

[0116] Model structure:

[0117] Shared Layer: A sequence of convolutional layers, batch normalization layers, and ReLU activation layers is used to extract features. Classifier: This layer uses adaptive average pooling and fully connected layers to map features to a number of categories. The server accepts model requests from clients, performs specific operations on the model, and sends the model parameters to the client.

[0118] Step 2: Client model initialization

[0119] Client devices each contain a data set 、 、 、 .

[0120] The client accepts the model parameters sent by the server.

[0121] Client model initialization: The model receives data input and outputs the corresponding distribution probability density. In this embodiment, the client model is a ResNet14 model, which includes:

[0122] conv3x3 function:

[0123] Function: Create a 3x3 convolutional layer with a padding of 1 and no bias.

[0124] parameter:

[0125] in_planes: The number of channels of the input feature map.

[0126] out_planes: The number of channels of the output feature map.

[0127] stride: The step size of the convolution, the default is 1.

[0128] BasicBlock class:

[0129] Function: Implement the basic residual block structure.

[0130] parameter:

[0131] inplanes: The number of channels of the input feature map.

[0132] planes: The number of channels of the convolutional layer.

[0133] stride: The step size of the convolution, the default is 1.

[0134] downsample: Optional parameter, used to adjust the dimension of the submodule, used when the number of input and output channels is different.

[0135] is_last: Boolean value indicating whether it is the last block in the sequence, affecting the output format.

[0136] structure:

[0137] Contains two convolutional layers (conv1 and conv2), each followed by a batch normalization layer (bn1 and bn2).

[0138] Use the ReLU activation function.

[0139] Residual connection: If downsample is not empty, the input x is downsampled.

[0140] Bottleneck class:

[0141] Function: Implement the bottleneck residual block structure.

[0142] parameter:

[0143] inplanes: The number of channels of the input feature map.

[0144] planes: The number of channels in the middle convolutional layer.

[0145] stride: The step size of the convolution, the default is 1.

[0146] downsample: Optional parameter, used to adjust the dimension of the submodule, used when the number of input and output channels is different.

[0147] is_last: Boolean value indicating whether it is the last block in the sequence, affecting the output format.

[0148] structure:

[0149] Contains three convolutional layers (conv1, conv2, and conv3), each followed by a batch normalization layer (bn1, bn2, and bn3).

[0150] Conv1 and Conv3 use 1x1 convolution kernels, and Conv2 uses 3x3 convolution kernels.

[0151] Use the ReLU activation function.

[0152] Residual connection: If downsample is not empty, the input x is downsampled.

[0153] ResNet class:

[0154] Function: Implement residual network.

[0155] parameter:

[0156] depth: The depth of the network, that is, the number of layers.

[0157] num_filters: List of the number of output channels for each layer.

[0158] block_name: Select the block type to use, which can be BasicBlock or Bottleneck. The default is BasicBlock.

[0159] num_classes: The number of categories for the classification task, the default is 10.

[0160] structure:

[0161] Contains an initial convolutional layer (conv1), a batch normalization layer (bn1), and a ReLU activation layer.

[0162] Construct multiple residual layers (layer1, layer2, and layer3) through the _make_layer method, each layer using a different number of channels and stride.

[0163] An adaptive average pooling layer (avgpool) is used to resize the feature map to 1x1.

[0164] Finally, a fully connected layer (fc) is used to map the features to the number of categories.

[0165] _make_layer method:

[0166] Function: Construct a residual layer based on the given block type and parameters.

[0167] parameter:

[0168] block: Block type, which can be BasicBlock or Bottleneck.

[0169] planes: The number of channels of the convolutional layer.

[0170] blocks: The number of blocks in this layer.

[0171] stride: The step size of the convolution, the default is 1.

[0172] structure:

[0173] If dimensionality adjustment is required, create a downsampling module.

[0174] Create multiple blocks and connect them in sequence.

[0175] get_feat_modules method:

[0176] Function: Get the feature extraction module in the network.

[0177] Returns: A list containing feature extraction modules.

[0178] get_bn_before_relu method:

[0179] Function: Get the batch normalization layer before the ReLU activation function in each residual layer.

[0180] Returns: A list containing batch normalization layers.

[0181] forward method:

[0182] Function: Define the forward propagation process of the network.

[0183] parameter:

[0184] x: input data.

[0185] is_feat: Boolean value indicating whether to return feature maps.

[0186] preact: Boolean value indicating whether to return the pre-activated feature map before ReLU activation.

[0187] return:

[0188] If is_feat is True, returns a list of feature maps and the final output.

[0189] Otherwise, just the final output is returned.

[0190] resnet14 model function:

[0191] Function: Create a residual network model with a depth of 14 layers.

[0192] parameter:

[0193] depth: The depth of the network, that is, the number of layers, is fixed at 14.

[0194] num_filters: List of the number of output channels for each layer, fixed to [16, 16, 32, 64].

[0195] The first 16 represents the number of output channels of the first convolutional layer.

[0196] The second 16 represents the number of output channels of the first residual block.

[0197] 32 represents the number of output channels of the second residual block, and from here the size of the feature map is halved.

[0198] 64 represents the number of output channels of the third residual block.

[0199] block_name: block type, fixed to 'basicblock', indicating the use of the basic block structure.

[0200] **kwargs: Additional keyword parameters that can be used to override the default parameters in the ResNet class initialization, for example:

[0201] num_classes: The number of categories for the classification task, defaults to 10, but can be modified via kwargs.

[0202] Other possible parameters, such as weight initialization method, etc.

[0203] Step 3: Complete the initialization of each customer model Use the corresponding customer user data set Model parameters for each customer model Updates.

[0204] 1. Train the customer model: Use binary cross entropy as the loss function to measure the model's classification performance. Use the SGD optimizer and set the learning rate to 0.001.

[0205] Optimizer SGD Function: Creates a stochastic gradient descent (SGD) optimizer for the customer model.

[0206] parameter:

[0207] Resnet11.parameters(): Gets all the parameters of the customer resnet11 model, which will be used by the optimizer for gradient updates.

[0208] lr=opt.clr: Learning rate, which controls the step size of gradient updates. Here, an external variable opt.clr=0.001 is used, which represents the value of the learning rate.

[0209] Step 4: Updated customer models Output prediction soft logic for each customer based on public data sets and accuracy .

[0210] 1. Customer model output soft logic: Use the softmax normalization function as the soft logic output function of the customer model.

[0211] Softmax function function: By generating soft labels and feature representations, it helps the student network learn richer and more generalized knowledge, thereby improving its performance.

[0212] parameter:

[0213] : The raw unnormalized scores (logits) tensor.

[0214] : Temperature parameter. In the softmax function, the temperature parameter is used to control the smoothness of the probability distribution. Lower temperature values will make the probability distribution more sharp, while higher temperature values will make the distribution smoother.

[0215] dim=1: Specifies the dimension along which the log-softmax is calculated. In this example, dim=1 indicates that the calculation is performed along the first dimension of the tensor (usually the dimension of features or categories).

[0216] 2. Accuracy of customer model output for public datasets: Use a custom accuracy calculation function.

[0217] parameter:

[0218] output: The raw scores (logits) output by the model, usually a two-dimensional tensor with a shape of [batch_size, num_classes], where batch_size is the number of samples and num_classes is the number of categories.

[0219] target: The actual label, usually a one-dimensional tensor with a shape of [batch_size], where each element is an integer representing the true category of the corresponding sample.

[0220] topk: A tuple or list specifying the k value of the top k highest prediction values for which the accuracy is to be calculated. The default value is (1,), which means that only the top-1 accuracy is calculated.

[0221] Step 5: Each customer gets the customer model Distillation dataset Soft Logic and accuracy Afterwards, and Send to the server.

[0222] Step 6: The server large parameter model outputs the server model soft logic for the distilled dataset. The client weight calculation will calculate the weight of each client based on the correlation between the server model soft logic and the soft logic of each client model and the accuracy of each client model for the distilled dataset.

[0223] Server Model Output distillation dataset Soft Logic .

[0224] The server gets the distilled dataset from each client model. Soft Logic With server model Soft Logic Then, the soft logic of each customer model is calculated based on the cosine distance function. Server Model Soft Logic The correlation between .

[0225] Cosine distance function: It is a method to measure the similarity or difference between two vectors in direction.

[0226] parameter:

[0227] soft_targets_n: customers The soft target is usually a batch of model output probability distributions with a shape of [batch_size, num_classes].

[0228] soft_targets_s: soft targets of the server, also of shape [batch_size, num_classes].

[0229] Calculate the soft logic of each customer model Server Model Soft Logic The correlation between After that, the customer weight calculation module will calculate the customer weight according to the received customer model. Distillation dataset Accuracy and relevance The weight corresponding to each customer is calculated jointly.

[0230] Step 7: Perform a weighted summation of each customer's proportion and each customer's soft logic to obtain the weighted customer soft logic.

[0231] Step 8: Calculate the weighted client model soft logic and the server model soft logic to obtain the server model distillation loss in each communication round, and perform a weighted summation with the task loss of the server model to obtain the total loss of the server model. Perform a backpropagation process on the server model to update the server model.

[0232] In this step, the server model is updated using binary cross entropy as the loss function to measure the model's classification performance. The SGD optimizer is used, with a learning rate of 0.01. The SGD optimizer function creates a stochastic gradient descent (SGD) optimizer for the client model.

[0233] parameter:

[0234] lr=opt.dlr: Learning rate, which controls the step size of gradient updates. An external variable opt.dlr=0.01 is used, which represents the value of the learning rate.

[0235] like Figure 4 As shown, according to another embodiment of the present invention, a reverse knowledge distillation system 400 based on a federated large model is also disclosed, including: a memory 401 for storing program instructions; a processor 402 for calling the program instructions stored in the memory to implement the reverse knowledge distillation method based on a federated large model as in the above embodiment.

[0236] All or part of the steps in the various methods of the above embodiments can be completed by controlling related hardware through a program. The program can be stored in a readable storage medium, and the storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electronically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0237] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A reverse knowledge distillation method based on a federated large model, characterized by: include: The server sends the small parameter model to the client local to the client, so that the client updates the weight parameters of the small parameter model using the private data set to obtain the client model; The client model outputs client model soft logic using a public image dataset, and sends the client model soft logic to the server; The large parameter model of the server uses the public image dataset to output server model soft logic; Calculating the weight of each client based on the correlation between the server model soft logic and the soft logic of multiple client models and the accuracy of each client model on the public image dataset; Calculate weighted client soft logic based on multiple client weights and client model soft logic; Performing knowledge distillation using the server model soft logic and the weighted client soft logic to update weight parameters of the server large parameter model; The knowledge distillation specifically includes: Calculating a distillation loss between the server model soft logic and the weighted client soft logic; The distillation loss and the task loss of the server model are weighted summed to obtain the total loss of the server model; The server model is back-propagated according to the total loss of the server model, so as to update the server model in the same manner as the client model is updated.

2. The reverse knowledge distillation method based on the federated large model according to claim 1 is characterized in that: The small parameter model is obtained by compressing the large parameter model of the server.

3. The reverse knowledge distillation method based on the federated large model according to claim 1 is characterized in that: The public image dataset is composed of data provided by various clients.

4. The reverse knowledge distillation method based on the federated large model according to claim 1 is characterized in that: The client model soft logic specifically includes: using a softmax normalization function as the soft logic output function of the client model.

5. The reverse knowledge distillation method based on the federated large model according to claim 1 is characterized in that: The correlation refers to the cosine distance.

6. The reverse knowledge distillation method based on a federated large model according to any one of claims 1 to 5, characterized in that: The large parameter model of the server is the ResNet38 model, and the client model is the ResNet14 model.

7. The reverse knowledge distillation method based on a federated large model according to any one of claims 1 to 5, characterized in that: The client is a personal computer or a handheld mobile device, and the server is a base station or an edge server.

8. A reverse knowledge distillation system based on a federated large model, characterized by: include: a memory for storing program instructions; A processor, configured to call the program instructions stored in the memory to implement the reverse knowledge distillation method based on a federated large model as described in any one of claims 1 to 7.

Images