Reverse knowledge distillation method and system based on federal large model

Abstract

The invention provides an inverse knowledge distillation method and system based on a federal large model, and relates to the field of federal learning, and the method comprises the steps: a client uses a private data set to update the weight parameters of a small parameter model, and obtains a client model; outputting client model soft logic by using the public data set and sending the client model soft logic to a server; the large-parameter model of the server uses the public data set to output server model soft logic; calculating the weight of each client according to the correlation between the soft logic of the server model and the soft logic of the plurality of client models and the accuracy of each client model to the public data set; calculating weighted client soft logic according to the client weight and the client model soft logic; knowledge distillation is performed using server model soft logic and weighted client soft logic. According to the method, knowledge distillation is carried out on a single large-parameter model by using a plurality of small-parameter models, and a brand-new method is provided for updating and iteration of the large-parameter models in a server.

Description

Technical Field

[0001] The present invention relates to the field of federated learning, and in particular to a method and system for reverse knowledge distillation based on a federated large model. Background Art

[0002] With the emergence and gradual maturity of large models, the dependence of large models on larger-scale training data and data privacy have given rise to a new problem: how to jointly train a large-scale model using isolated data in the private domain without infringing on data privacy legal provisions. A training architecture and method for large models based on federated learning has emerged to solve the problem of distributed large model training. Federated learning trains models in a distributed manner without centralizing data to a central location, thereby effectively protecting user privacy. Knowledge distillation achieves lightweight and efficient reasoning of models by transferring the knowledge of complex models to smaller models. In edge computing scenarios, the combination of federated learning and knowledge distillation technology can improve the performance of models on edge devices while ensuring data privacy, making smart applications more efficient and convenient. Federated knowledge distillation is an advanced technology that combines federated learning and knowledge distillation, which improves overall performance and accuracy by collaboratively training models in a distributed environment. In this framework, different devices and nodes can share their local model knowledge without directly sharing raw data, thereby protecting data privacy. At the same time, through knowledge distillation technology, the knowledge of complex models is transferred to lighter models, optimizing the reasoning efficiency and resource consumption of the models. This method is particularly suitable for scenarios with high requirements for computing efficiency, such as the healthcare and financial industries, to achieve efficient and accurate intelligent applications.

[0003] In the process of implementing the present invention, the applicant found that the traditional knowledge distillation strategy is to transfer knowledge from a trained large model to a small or untrained model, which often ignores the contribution of the small model with personalized data to the large model. These traditional strategies cannot allow the large model to fully learn on specific data sets in certain vertical fields, resulting in unsatisfactory performance and personalized user experience. Previous research still has an obvious limitation. Most studies are limited to federated knowledge distillation for models of the same architecture, and do not consider collaborative training between heterogeneous client devices with limited computing resources.

[0004] Therefore, how to improve the training efficiency and performance of large-parameter models on the server side becomes a technical problem that needs to be solved. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art or related technology, and discloses an inverse knowledge distillation method and system based on a federated large model. Knowledge is transferred to a large parameter model through multiple small parameter models, which can reduce network resource consumption, improve accuracy, reduce training rounds and enhance user experience.

[0006] Terminology explanation: Federated Learning is a distributed machine learning technology that allows multiple data owners to build machine learning models without sharing data. The goal of federated learning is to achieve joint modeling and improve the effectiveness of AI models while ensuring data privacy and legal compliance. Knowledge Distillation is a model compression technology that trains a small neural network (called the "student model") to imitate a large, pre-trained neural network (called the "teacher model"). This method can be seen as a transfer of knowledge from the teacher model to the student model, hence the name "knowledge distillation." The first aspect of the present invention discloses a reverse knowledge distillation method based on a federated large model, including: a server sends a small parameter model to a local client of the client, so that the client uses a private data set to update the weight parameters of the small parameter model to obtain a client model; the client model uses a public data set to output the client model soft logic, and sends the client model soft logic to the server; the server's large parameter model uses the public data set to output the server model soft logic; the weight of each client is calculated based on the correlation between the server model soft logic and the soft logic of multiple client models and the accuracy of each client model for the public data set; the weighted client soft logic is calculated based on the client weight and the client model soft logic; the server model soft logic and the weighted client soft logic are used to perform knowledge distillation to update the weight parameters of the server's large parameter model.

[0007] In this technical solution, a method for federated knowledge distillation is disclosed in which multiple client small parameter models transfer knowledge to a single server large parameter model. The present invention defines this technical framework as reverse knowledge distillation. Due to the limitation of client computing resources, the server sends a compressed version of the large parameter model, i.e., the small parameter model, to the client. The client accepts the small parameter model from the server and trains it using a private data set. The client model soft logic output is to output a soft logic for the public image data set using the trained client model. The client weight calculation is to calculate the weight of each client in the process of federated knowledge distillation. The sum of the client weights is 1. The server model knowledge distillation refers to the server obtaining the client weighted soft logic and the server soft logic, and performing the knowledge distillation process on the server model to update the server model.

[0008] According to the reverse knowledge distillation method based on a federated large model disclosed in the present invention, preferably, knowledge distillation specifically includes: calculating the distillation loss between the server model soft logic and the weighted client soft logic; weighted summing the distillation loss and the task loss of the server model to obtain the total loss of the server model; backpropagating the server model according to the total loss of the server model, thereby updating the server model in the same manner as the client model is updated. The task loss of the server model refers to the gap between the prediction output by the server model for the data set and the actual prediction of the data set, which is usually calculated using a cross entropy loss function. According to the reverse knowledge distillation method based on a federated large model disclosed in the present invention, preferably, the small parameter model is compressed from the large parameter model of the server.

[0009] According to the inverse knowledge distillation method based on the federated large model disclosed in the present invention, preferably, the public data set consists of data provided by various clients.

[0010] According to the inverse knowledge distillation method based on the federated large model disclosed in the present invention, preferably, the client model soft logic specifically includes: using a softmax normalization function as the soft logic output function of the client model.

[0011] According to the inverse knowledge distillation method based on the federated large model disclosed by the present invention, preferably, the correlation refers to the cosine distance.

[0012] According to the inverse knowledge distillation method based on the federated large model disclosed in the present invention, preferably, the large parameter model of the server is a ResNet38 model, and the client model is a ResNet14 model.

[0013] In this technical solution, ResNet (residual network) is a deep learning model proposed by Microsoft Research in 2015. Its core idea is to introduce a residual learning framework to solve the gradient vanishing and gradient exploding problems in deep neural network training through skip connections. The core of the ResNet model is the residual blocks unique to the ResNet model. Common ResNet models include ResNet-14, ResNet-38, ResNet-50, etc., where 14, 38, 50, etc. represent the number of residual blocks in the model. For example, ResNet-38 contains 38 residual blocks, and ResNet-14 contains 14 residual blocks. In the ResNet model, residual blocks are classified into two categories: basicblock and bottleblock. If the ResNet model uses basicblock, the number x of residual blocks basicblock in the ResNet model should satisfy the equation (x-2)% 6=0; if the ResNet model uses bottleblock, the number x of residual blocks bottleblock in the ResNet model should satisfy the equation (x-2)% 9=0.

[0014] According to the reverse knowledge distillation method based on the federated large model disclosed in the present invention, preferably, the client is a personal computer or a handheld mobile device, and the server is a base station or an edge server.

[0015] The second aspect of the present invention discloses an inverse knowledge distillation system based on a federated large model, comprising: a memory for storing program instructions; a processor for calling the program instructions stored in the memory to implement an inverse knowledge distillation method based on a federated large model such as any of the above-mentioned technical solutions.

[0016] Practical application scenarios of the technical solution provided by the present invention: With the development of artificial intelligence technology, most cameras have image recognition functions. However, in some cases, the camera cannot effectively identify some objects that have never been seen. Through the method provided by the present invention, for some objects that have never been seen, only some cameras need to be updated locally and then the model in a server is distilled. In this way, when some cameras cannot recognize certain objects or new cameras are connected, a request can be sent to the server to obtain a new model without multiple local updates, which can reduce energy consumption while also reducing camera latency and improving performance.

[0017] The beneficial effects of the present invention include at least: the present invention takes into account the user privacy leakage problem that may be caused by the model aggregation process in traditional federated learning, and deletes the model aggregation process, thereby improving the utilization rate of network resources. The technical solution provided by the present invention extends the concept of the combination of federated learning and knowledge distillation to the traditional network environment, and uses a unique method of multiple small parameter models to perform knowledge distillation on a single large parameter model, providing a new method for updating and iterating large parameter models in the server, and using federated learning and knowledge distillation to handle the requirements of large parameter models in the server for multiple data types. It can allow a well-performing model to improve the performance of the server model on certain unknown data sets while running locally on the client, and can also save service resource consumption by reducing the training rounds of the server model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flow chart of a reverse knowledge distillation method based on a federated large model according to an embodiment of the present invention is shown.

[0019] Figure 2 A schematic diagram of a client weight calculation process according to an embodiment of the present invention is shown.

[0020] Figure 3 A schematic diagram of a knowledge distillation workflow of a server model according to an embodiment of the present invention is shown.

[0021] Figure 4 A schematic block diagram of an inverse knowledge distillation system based on a federated large model according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0022] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and therefore, the present invention is not limited to the limitations of the specific embodiments disclosed below.

[0023] like Figure 1As shown, according to an embodiment of the present invention, a reverse knowledge distillation method based on a federated large model is disclosed, including: a server sends a small parameter model to a client; the client performs local training to form a trained client local model; calculates the client model accuracy and the client model soft logic; determines the client weight; determines the client model soft logic according to each client weight to calculate the weighted client soft logic, and then uses the server model soft logic and the weighted client soft logic to perform a knowledge distillation process to update the weight parameters of the server large parameter model, thereby completing the training of the server model. Specifically includes the following steps: Assume that there are a total of round, and in each communication round There are a total of 100 customer model training rounds Rounds, each training round is recorded as .

[0024] First For example, the following communication: S1, after receiving the small parameter model sent from the server, the client uses the private data set to update the weight parameters of the small parameter model; S2, the trained client model uses the public image dataset to output the client model soft logic and sends the client model soft logic to the server; S3, the server large parameter model outputs the server model soft logic for the public image dataset, and calculates the weight of each client based on the correlation between the server model soft logic and the soft logic of each client model and the accuracy of each client model for the public image dataset; S4, server model knowledge distillation: Calculate weighted customer soft logic based on each customer weight and customer model soft logic, and then use server model soft logic and weighted customer soft logic to perform knowledge distillation process to update the weight parameters of the server large parameter model.

[0025] The above step S1 specifically includes: S1.1, receiving the small parameter model sent from the server ,in Is the logo of each customer, there are clients; S1.2, client small parameter model training uses client private dataset For small parameter models For training, the training process is as follows:

[0026] in It is the learning rate of customer model update, which represents the degree of each customer model update. Represents the Locally trained customer small parameter model The weight parameter, Represents the Locally trained customer small parameter model The weight parameter, is the cross entropy loss function, Based on local dataset The loss function For model parameters The inverse of , that is, the gradient operation.

[0027] S1.3, after each customer model is trained, the customer model is recorded as , indicating that the customer model participating in the distillation process in each communication round is the customer model trained locally.

[0028] The above step S2 specifically includes: S2.1, after each customer model is trained, it is used on the public image dataset Output the soft logic corresponding to each model , the output formula is as follows:

[0029] in is the distillation temperature parameter, It is an activation function. The combined use of the two softens the probability distribution of the customer model for the public image dataset, making the output of the customer model smoother, allowing the customer output to contain more information, and reducing the probability of overfitting of the model. is the predicted output of the model, It is a public image dataset A sample of Represents the customer model The weight parameter of .

[0030] S2.2, after each customer model is trained, it is used on the public image dataset Output the accuracy of each model , the output formula is as follows:

[0031] in is the cross entropy loss function, It is a public image dataset Medium data sample The real label.

[0032] S2.3, each customer gets its own customer model Public image datasets Soft Logic and accuracy After that, and Send to the server.

[0033] like Figure 2 As shown, the above step S3 specifically includes: S3.1, first server model Output to public image datasets Soft Logic , the output formula is as follows:

[0034] in Is a server model The weight parameter, is the predicted output of the model, It is a public image dataset of the sample.

[0035] S3.2, the server obtains the public image dataset from each client model Soft Logic With server model Soft Logic Then, the soft logic of each customer model is calculated according to the cosine distance function. Server Model Soft Logic The correlation between , the calculation formula is as follows: in is the cosine distance function.

[0036] S3.3, after calculating the soft logic of each customer model With server model Soft Logic The correlation between After that, the customer weight calculation module will calculate the customer weight according to the received customer model. Public image datasets The accuracy Relevance The weight corresponding to each customer is calculated jointly, and the calculation formula is as follows: in It is The weight corresponding to each customer, The accuracy rate of each customer Relevance The summed value of .

[0037] like Figure 3 As shown, the above step S4 specifically includes: S4.1, calculate the weight corresponding to each customer After that, the server will With soft logic To calculate the weighted customer soft logic, the calculation formula is as follows: in The weight of each customer With soft logic The weighted sum of .

[0038] S4.2, after obtaining the weighted customer soft logic After that, the server soft logic can be calculated With weighted customer soft logic The distillation loss during the experiment is calculated as follows:

[0039] in It is server soft logic With weighted customer soft logic Distillation loss, It is the KL divergence function, which is often used in conjunction with the cross entropy loss function in knowledge distillation to obtain a comprehensive loss function.

[0040] It is a public image dataset The number of categories in , and Weighted Customer Soft Logic Server Soft Logic No. categories.

[0041] S4.3, after obtaining the distillation loss, it can be concluded that in the framework of the present invention, the server model has The total loss in is calculated as follows:

[0042] in and are the weights of the server model task loss and distillation loss, respectively.

[0043] S4.4, after obtaining the comprehensive loss, the server model can be updated in the same way as the client model is updated:

[0044] in Based on a public image dataset The sum loss function For model parameters The inverse of , that is, the gradient operation. is the distillation learning rate of the server.

[0045] According to another embodiment of the present invention, an implementation of the above-mentioned inversion knowledge distillation method based on the federated large model is also disclosed, including: The server compresses the server model and sends it to each client according to the different devices of each client. At the same time, the server receives data from each client to form a public image dataset and sends the public image dataset to each client. After receiving the model from the server, the client uses each client’s data set to train the model; After the customer completes the training of the received model, the trained model is used to output the predicted probability of the public image dataset, which is the customer soft logic; The server accepts the soft logic of the public image dataset from all clients participating in the distillation process, and outputs its own prediction probability for the public image dataset, namely the server soft logic; The server uses the server soft logic and each customer soft logic to calculate the proportion of each customer in the distillation process, and then performs a weighted summation of each customer proportion and each customer soft logic to obtain the weighted customer soft logic; The weighted client model soft logic is calculated with the server model soft logic to obtain the distillation loss of the server model in each communication round, and the weighted sum is taken with the task loss of the server model to obtain the total loss of the server model. The server model is back-propagated to update the server model.

[0046] According to another embodiment of the present invention, a specific implementation of the above-mentioned inversion knowledge distillation method based on the federated large model in an actual application scenario is also disclosed, including the following steps: Step 1: The server initializes the model and sends a specific model to the client: The server initializes the server model: the model receives data input and outputs the corresponding distribution probability density. In this embodiment, the server model is the ResNet38 model. A multi-task learning (MTL) residual network model is defined, which contains some auxiliary classes and functions. The following is a detailed explanation of the classes and functions defined in the code: MTL functions: Function: Initialize and return a multi-task learning residual network model.

[0047] parameter: class_num: The number of categories for the classification task.

[0048] pretrained: Boolean value indicating whether to use the pretrained model. The default value is False.

[0049] path: The path of the pre-trained model, only used when pretrained=True.

[0050] **kwargs: Additional keyword arguments for passing to the model class.

[0051] Reshape class: Function: Implement a reshape layer that reshapes the input tensor into a specified shape.

[0052] parameter: *args: An argument list containing the target shape.

[0053] method: forward(self, x): Forward propagation method, reshapes the input x into the specified shape.

[0054] MTLResNet class: Function: Implement a multi-task learning residual network model.

[0055] parameter: block: The type of residual block, usually Bottleneck.

[0056] layers: A list specifying the number of layers for each residual block.

[0057] num_classes: The number of categories for the classification task, the default is 10.

[0058] groups: The number of groups for grouped convolution, the default is 1.

[0059] width_per_group: The width of each group, the default is 64.

[0060] method: _make_layer(self, block, planes, blocks, stride=1, dilate=False) : Creates a residual block layer.

[0061] forward(self, x): forward propagation method, computing classification and discrimination outputs.

[0062] Bottleneck Class: Function: Implement a bottleneck residual block.

[0063] parameter: inplanes: The number of channels of the input feature map.

[0064] planes: The number of channels of the convolutional layer.

[0065] stride: The step size of the convolution, the default is 1.

[0066] downsample: Optional parameter, used to adjust the dimension of the submodule.

[0067] groups: The number of groups for grouped convolution.

[0068] base_width: base width.

[0069] dilation: dilation rate of convolution.

[0070] norm_layer: Batch normalization layer, default is nn.BatchNorm2d.

[0071] conv3x3 function: Function: Create a 3x3 convolutional layer with specified padding and dilation rates.

[0072] parameter: in_planes: number of input channels.

[0073] out_planes: number of output channels.

[0074] stride: The step size of the convolution, the default is 1.

[0075] groups: The number of groups for grouped convolution.

[0076] dilation: dilation rate of convolution.

[0077] conv1x1 function: Function: Create a 1x1 convolutional layer.

[0078] parameter: in_planes: number of input channels.

[0079] out_planes: number of output channels.

[0080] stride: The stride of the convolution, the default is 1.

[0081] Model structure: Shared layer: A sequence of convolutional layers, batch normalization layers, and ReLU activation layers to extract features. Classifier: Uses adaptive average pooling layers and fully connected layers to map features to the number of categories. The server accepts model requests from clients. The server performs specific operations on the model and sends the model parameters to the client.

[0082] Step 2: Client model initialization Client devices each contain a data set , , , .

[0083] The client accepts the model parameters sent by the server.

[0084] Client model initialization: The model receives data input and outputs the corresponding distribution probability density. In this embodiment, the client model is a ResNet14 model, which includes: conv3x3 function: Function: Create a 3x3 convolutional layer with a padding of 1 and no bias.

[0085] parameter: in_planes: The number of channels of the input feature map.

[0086] out_planes: The number of channels of the output feature map.

[0087] stride: The step size of the convolution, the default is 1.

[0088] BasicBlock class: Function: Implement the basic residual block structure.

[0089] parameter: inplanes: The number of channels of the input feature map.

[0090] planes: The number of channels of the convolutional layer.

[0091] stride: The step size of the convolution, the default is 1.

[0092] downsample: Optional parameter, used to adjust the dimension of the submodule, used when the number of input and output channels is different.

[0093] is_last: Boolean value indicating whether it is the last block in the sequence, affecting the output format.

[0094] structure: Contains two convolutional layers (conv1 and conv2), each followed by a batch normalization layer (bn1 and bn2).

[0095] Use the ReLU activation function.

[0096] Residual connection: If downsample is not NULL, downsample the input x.

[0097] Bottleneck Class: Function: Implement the bottleneck residual block structure.

[0098] parameter: inplanes: The number of channels of the input feature map.

[0099] planes: The number of channels of the intermediate convolutional layer.

[0100] stride: The step size of the convolution, the default is 1.

[0101] downsample: Optional parameter, used to adjust the dimension of the submodule, used when the number of input and output channels is different.

[0102] is_last: Boolean value indicating whether it is the last block in the sequence, affecting the output format.

[0103] structure: Contains three convolutional layers (conv1, conv2, and conv3), each followed by a batch normalization layer (bn1, bn2, and bn3).

[0104] conv1 and conv3 use 1x1 convolution kernels, and conv2 uses 3x3 convolution kernels.

[0105] Use the ReLU activation function.

[0106] Residual connection: If downsample is not NULL, downsample the input x.

[0107] ResNet class: Function: Implement residual network.

[0108] parameter: Depth: The depth of the network, that is, the number of layers.

[0109] num_filters: List of number of output channels for each layer.

[0110] block_name: Select the block type to use, which can be BasicBlock or Bottleneck. The default is BasicBlock.

[0111] num_classes: The number of categories for the classification task, the default is 10.

[0112] structure: Contains an initial convolutional layer (conv1), a batch normalization layer (bn1), and a ReLU activation layer.

[0113] Multiple residual layers (layer1, layer2, and layer3) are constructed through the _make_layer method, each layer using a different number of channels and stride.

[0114] The feature map is resized to 1x1 using an adaptive average pooling layer (avgpool).

[0115] Finally, a fully connected layer (fc) is used to map the features to the number of categories.

[0116] _make_layer method: Function: Construct a residual layer based on the given block type and parameters.

[0117] parameter: block: Block type, can be BasicBlock or Bottleneck.

[0118] planes: The number of channels of the convolutional layer.

[0119] blocks: The number of blocks in this layer.

[0120] stride: The stride of the convolution, the default is 1.

[0121] structure: If resizing is required, create a downsampling module.

[0122] Create multiple blocks and connect them in sequence.

[0123] get_feat_modules method: Function: Get the feature extraction module in the network.

[0124] Returns: A list containing feature extraction modules.

[0125] get_bn_before_relu method: Function: Get the batch normalization layer before the ReLU activation function in each residual layer.

[0126] Returns: A list containing batch normalization layers.

[0127] forward method: Function: Define the forward propagation process of the network.

[0128] parameter: x: input data.

[0129] is_feat: Boolean value indicating whether to return feature maps.

[0130] preact: Boolean value indicating whether to return the pre-activation feature map before ReLU activation.

[0131] return: If is_feat is True, a list of feature maps and the final output are returned.

[0132] Otherwise, just the final output is returned.

[0133] resnet14 model function: Function: Create a residual network model with a depth of 14 layers.

[0134] parameter: depth: The depth of the network, that is, the number of layers, is fixed at 14.

[0135] num_filters: List of the number of output channels for each layer, fixed to [16, 16, 32, 64].

[0136] The first 16 indicates the number of output channels of the first convolutional layer.

[0137] The second 16 represents the number of output channels of the first residual block.

[0138] 32 indicates the number of output channels of the second residual block, and the size of the feature map is halved from here.

[0139] 64 represents the number of output channels of the third residual block.

[0140] block_name: block type, fixed as 'basicblock', indicating the use of the basic block structure.

[0141] **kwargs: Additional keyword parameters that can be used to override the default parameters in the ResNet class initialization, for example: num_classes: The number of categories for the classification task, defaults to 10, but can be modified via kwargs.

[0142] Other possible parameters, such as weight initialization method, etc.

[0143] Step 3: Complete the initialization of each customer model Use the corresponding customer user data set Perform model parameters for each customer model Updates.

[0144] 1. Train the customer model: Use binary cross entropy as the loss function to measure the classification performance of the model. Use the optimizer SGD and set the learning rate to 0.001.

[0145] Optimizer SGD Function: Creates a stochastic gradient descent (SGD) optimizer for a customer model.

[0146] parameter: Resnet11.parameters(): Get all the parameters of the customer resnet11 model, which will be used by the optimizer for gradient updates.

[0147] lr=opt.clr: learning rate, which controls the step size of gradient update. Here, an external variable opt.clr=0.001 is used, which represents the value of the learning rate.

[0148] Step 4: Updated customer models Output prediction soft logic for each customer for public data sets And accuracy .

[0149] 1. Customer model output soft logic: Use the softmax normalization function as the soft logic output function of the customer model.

[0150] Softmax function function: By generating soft labels and feature representations, it helps the student network learn richer and more generalized knowledge, thereby improving its performance.

[0151] parameter: : The raw unnormalized scores (logits) tensor.

[0152] : Temperature parameter. In the softmax function, the temperature parameter is used to control the smoothness of the probability distribution. Lower temperature values ​​will make the probability distribution more sharp, while higher temperature values ​​will make the distribution smoother.

[0153] dim=1: specifies the dimension along which the log-softmax is calculated. In this example, dim=1 means that the calculation is performed along the first dimension of the tensor (usually the dimension of features or categories).

[0154] 2. Accuracy of customer model output for public datasets: Use a custom accuracy calculation function.

[0155] parameter: output: The raw scores (logits) output by the model, usually a two-dimensional tensor with a shape of [batch_size, num_classes], where batch_size is the number of samples and num_classes is the number of categories.

[0156] target: The actual label, usually a one-dimensional tensor with a shape of [batch_size], where each element is an integer representing the true category of the corresponding sample.

[0157] topk: A tuple or list specifying the k value of the top k highest prediction values ​​for which the accuracy is to be calculated. The default value is (1,), which means that only the top-1 accuracy is calculated.

[0158] Step 5: Each customer gets the customer model For the distillation dataset Soft Logic and accuracy After that, and Send to the server.

[0159] Step 6: The server large parameter model outputs the server model soft logic for the distilled data set. The customer weight calculation will calculate the weight of each customer based on the correlation between the server model soft logic and the soft logic of each client model and the accuracy of each customer model for the distilled data set.

[0160] Server Model Output distillation dataset Soft Logic .

[0161] The server gets the distilled dataset from each client model. Soft Logic With server model Soft Logic Then, the soft logic of each customer model is calculated according to the cosine distance function. Server Model Soft Logic The correlation between .

[0162] Cosine distance function: is a method to measure the similarity or difference between two vectors in direction.

[0163] parameter: soft_targets_n: customer The soft target is usually a batch of model output probability distributions with a shape of [batch_size, num_classes].

[0164] soft_targets_s: soft targets of the server, also in the shape of [batch_size, num_classes].

[0165] After calculating the soft logic of each customer model Server Model Soft Logic The correlation between After that, the customer weight calculation module will calculate the customer weight according to the received customer model. For the distillation dataset The accuracy Relevance The weight corresponding to each customer is calculated jointly.

[0166] Step 7: Perform a weighted summation of each customer proportion and each customer soft logic to obtain the weighted customer soft logic.

[0167] Step 8: Calculate the weighted client model soft logic and the server model soft logic to obtain the distillation loss of the server model in each communication round, and perform weighted summation with the task loss of the server model to obtain the total loss of the server model. Perform a back propagation process on the server model to update the server model.

[0168] In this step, the server model update steps include: using binary cross entropy as the loss function to measure the classification performance of the model. Using the optimizer SGD and setting the learning rate to 0.01. Optimizer SGD function: Create a stochastic gradient descent (SGD) optimizer for the client model.

[0169] parameter: lr=opt.dlr: learning rate, which controls the step size of gradient update. An external variable opt.dlr=0.01 is used, which represents the value of the learning rate.

[0170] like Figure 4 As shown, according to another embodiment of the present invention, a reverse knowledge distillation system 400 based on a federated large model is also disclosed, including: a memory 401, used to store program instructions; a processor 402, used to call the program instructions stored in the memory to implement the reverse knowledge distillation method based on a federated large model as in the above embodiment.

[0171] All or part of the steps in the various methods of the above embodiments can be completed by controlling the relevant hardware through a program, and the program can be stored in a readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0172] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A reverse knowledge distillation method based on a federated large model, It is characterized in that include: The server sends the small parameter model to the client local to the client, so that the client updates the weight parameters of the small parameter model using the private data set to obtain the client model; The client model uses a public data set to output client model soft logic, and sends the client model soft logic to the server; The large parameter model of the server uses the public data set to output server model soft logic; Calculating the weight of each client according to the correlation between the server model soft logic and the multiple client model soft logics and the accuracy of each client model for a public data set; Calculate weighted client soft logic according to multiple client weights and client model soft logic; The server model soft logic and the weighted client soft logic are used to perform knowledge distillation to update the weight parameters of the server large parameter model.

2. According to claim 1, the inverse knowledge distillation method based on the federated large model, It is characterized in that The knowledge distillation specifically includes: Calculating a distillation loss between the server model soft logic and the weighted client soft logic; The distillation loss and the task loss of the server model are weighted summed to obtain the total loss of the server model; The server model is back-propagated according to the total loss of the server model, so as to update the server model in the same manner as the client model is updated.

3. According to claim 1, the inverse knowledge distillation method based on the federated large model, It is characterized in that The small parameter model is obtained by compressing the large parameter model of the server.

4. The inverse knowledge distillation method based on the federated large model according to claim 1, It is characterized in that The public data set is composed of data provided by various clients.

5. The inverse knowledge distillation method based on the federated large model according to claim 1, It is characterized in that The client model soft logic specifically includes: using a softmax normalization function as a soft logic output function of the client model.

6. The inverse knowledge distillation method based on the federated large model according to claim 1, It is characterized in that The correlation is the cosine distance.

7. The inverse knowledge distillation method based on a federated large model according to any one of claims 1 to 6, It is characterized in that The large parameter model of the server is the ResNet38 model, and the client model is the ResNet14 model.

8. The inverse knowledge distillation method based on a federated large model according to any one of claims 1 to 6, It is characterized in that The client is a personal computer or a handheld mobile device, and the server is a base station or an edge server.

9. A reverse knowledge distillation system based on a federated large model, It is characterized in that include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory to implement the reverse knowledge distillation method based on a federated large model as described in any one of claims 1 to 8.

Images