A federated reverse knowledge distillation method and system based on personalized model compression

Abstract

The application provides a federated reverse knowledge distillation method and system based on personalized model compression, aiming to solve the deficiencies of traditional methods in communication efficiency, model personalization and heterogeneous device adaptation. In this method, the server provides a reference template for the compression strategy when issuing the global model. Upon receiving the model, the client does not directly use the original structure for local training, but first adjusts the compression strategy based on its own hardware conditions, network bandwidth, local data distribution and input feature complexity, thereby generating a more adaptable personalized model structure. During the compression process, unlike traditional fixed ratio compression methods, the application can adaptively optimize the compression ratio, parameter retention position and precision adjustment, enabling the final personalized model to run efficiently on local devices while maintaining close accuracy to the original model in task performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of federated learning, and in particular to a federated reverse knowledge distillation method and system based on personalized model compression. BACKGROUND

[0002] With the emergence and gradual maturity of large models, edge computing and terminal intelligent devices have developed rapidly. How to achieve efficient and low communication load intelligent model training while protecting data privacy has become a research hotspot. Federated learning, as a distributed modeling method, allows terminal devices to retain data locally and participate in model training, effectively avoiding the privacy leakage risk brought by data centralization. At the same time, knowledge distillation technology achieves the dual goals of model compression and generalization ability improvement by transferring the prediction results between multiple models. In this context, federated knowledge distillation combines the advantages of both, improves communication efficiency and model security by uploading model output instead of model parameters.

[0003] In the prior art, although federated learning can avoid direct transmission of raw data in multi-terminal collaborative training, it still has great limitations in actual end-edge-cloud collaborative deployment. Especially in an environment with significant device heterogeneity, different clients have great differences in processor performance, memory capacity, storage space, and network bandwidth. If the same size and structure of the global model are uniformly issued to all clients, it will often cause devices with weak computing power to be unable to complete training or reasoning within an acceptable time, or even cause model loading failure, while devices with abundant computing power may waste resources and fail to fully utilize hardware potential. At the same time, the local data collected by different clients also has significant differences in distribution characteristics, sample size, and modal complexity. This difference makes it difficult for a single global model to maintain optimal performance in various task scenarios, especially in applications that require personalized responses. SUMMARY

[0004] The purpose of the present application is to provide a federated reverse knowledge distillation method and system based on personalized model compression, which introduces a personalized compression mechanism to dynamically adjust the model structure of each client according to local resources and data characteristics, and combines reverse distillation technology to improve the accuracy and adaptability of the server model, suitable for intelligent application scenarios such as medical, financial, and mobile devices that require high privacy, security, and efficiency.

[0005] Term explanation:

[0006] Personalized Compression Client (PCC): An innovative mechanism proposed by the invention, the client can adaptively compress the server model using low-rank decomposition and structured pruning based on hardware performance, data distribution characteristics and task complexity, so as to generate a small model that fits the local conditions to participate in training, and achieve a balance between communication, computation and precision.

[0007] Federated Learning (FL): A distributed machine learning technique that allows multiple data holders to collaboratively train models without sharing raw data. Data is always kept locally, and only model updates or intermediate representations are uploaded.

[0008] Knowledge Distillation (KD): A model compression and knowledge transfer method that trains a smaller "student model" to mimic the output of a larger and more powerful "teacher model", thereby reducing computational and storage overhead while retaining high accuracy.

[0009] Reverse Knowledge Distillation: A strategy opposite to the traditional distillation direction, in which multiple small models (clients) fuse and transfer the knowledge they have learned to a large model (server), to improve the generalization ability of the large model in multiple tasks and multiple fields.

[0010] To solve the above problems, the invention discloses a federated reverse knowledge distillation method based on personalized model compression, comprising the following steps:

[0011] S1, the server sends the current global model to each client participating in federated learning, and the client formulates a personalized layer-by-layer model compression strategy according to the local private data distribution characteristics, the complexity of the global model parameters and the local computing resources, and determines the structure ratio of each layer network that should be retained;

[0012] S2, the client performs layer-by-layer low-rank compression operation on the received global model according to the formulated multiple layer-by-layer candidate compression schemes, generates multiple compressed models, and evaluates the accuracy of the multiple compressed models using the local validation dataset, and selects the best one as the training model for this round;

[0013] S3, the client performs multiple rounds of local training on the training model using local private data, and obtains an updated model that fits the local data distribution;

[0014] S4, the client uses the updated model to perform forward inference on a shared unlabeled public dataset to generate prediction probabilities for samples, and uploads the prediction probabilities to the server;

[0015] S5, after the server receives the prediction probabilities of the plurality of clients, the server calculates the weight of each client according to the accuracy of the prediction probabilities of each client model and the consistency between the prediction probabilities of each client model and the server model output;

[0016] S6, the server weights and fuses the prediction probabilities of the plurality of clients according to the weights to form a weighted prediction probability distribution, the server takes the current model as a student model, completes a distillation training by minimizing the difference between the output of the student model and the weighted fusion result, and updates the parameters of the server global model.

[0017] Preferably, the step S1 specifically comprises:

[0018] I. The server directly distributes the parameters of the current global model to all clients participating in federated learning at the beginning, and each client receives the global model with the same structure and parameters;

[0019] II. Each client formulates its own personalized layer-by-layer compression strategy according to the local private data distribution characteristics, the global model parameter complexity, and the local computing resources, and determines the structure proportion to be reserved for each layer of network.

[0020] Preferably, the step S1 of formulating the personalized layer-by-layer compression strategy by the client comprises using the singular value decomposition method to generate different compression rates layer by layer.

[0021] Preferably, the step S2 specifically comprises:

[0022] (1) The client first performs multiple low-rank compressions on the global model parameters according to the generated different compression rates, and performs singular value decomposition operation on each layer to be compressed using the corresponding compression rate to generate a plurality of compressed models with different compression degrees;

[0023] (2) For each compressed model generated under each compression rate, the client quickly tests the accuracy of the compressed model using the local validation dataset;

[0024] (3) Compare the accuracies of the plurality of compressed models, and the client selects the best compressed model on the local validation dataset.

[0025] Preferably, the server model is a ResNet-50 model, and the compressed model is a ResNet-34 model.

[0026] Preferably, the client is a mobile terminal, a fixed terminal or an Internet of Things terminal, and the server is a base station or an edge server.

[0027] Preferably, the consistency in the step S5 is calculated by using an Euclidean distance function.

[0028] The application further discloses a federated reverse knowledge distillation system based on personalized model compression, comprising a memory for storing program instructions and a processor for calling the program instructions stored in the memory to implement the federated reverse knowledge distillation method based on personalized model compression.

[0029] Advantages:

[0030] The application provides a federated reverse knowledge distillation method based on personalized model compression, aiming to solve the deficiencies of traditional methods in communication efficiency, model personalization and heterogeneous device adaptation. In the method, the server provides a reference template of compression strategy when issuing the global model, and the client does not directly use the original structure for local training after receiving the model, but first adjusts the compression strategy according to the local hardware conditions, network bandwidth, local data distribution and input feature complexity, thereby generating a more adaptable personalized model structure. In the compression process, unlike the traditional fixed ratio compression method, the application can adaptively optimize the compression ratio, parameter retention position and precision adjustment, so that the finally generated personalized model can not only run efficiently on the local device, but also maintain close accuracy in task performance to the original model.

[0031] In the local training stage, since the personalized compression model is matched in size with the computing power and storage capacity of the local device, the training throughput is significantly improved, and the device can complete more rounds of local updates in a limited time, thereby accelerating the convergence speed of the model. After training is completed, the client does not upload the complete model parameter file, but performs inference on the pre-agreed unlabeled public data set or generated data set to obtain the prediction probability output and upload it to the server as knowledge expression. After receiving the prediction probability results from different clients, the server calculates the fusion weight of the knowledge of each client according to the performance in accuracy, consistency and task relevance, and performs weighted fusion and update of the global model accordingly.

[0032] Thus, the whole federal training process can be flexibly adapted to different hardware conditions while maintaining high efficiency. Since the client uploads the prediction results instead of the complete parameters, the amount of communication data is significantly reduced, effectively reducing the bandwidth pressure. In the global model updating process, the knowledge of the personalized model can also benefit the server model, thereby improving the generalization performance and personalized adaptation ability of the global model under heterogeneous data tasks. Through the implementation of the present application, low-cost and efficient training among multiple clients can be achieved in an end-side-cloud collaborative scenario, not only reducing communication overhead and training time, but also enhancing the personalized service ability and practical deployment feasibility of the system, which has wide application prospects in intelligent connected vehicles, mobile terminals, industrial Internet of Things and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and are used to interpret the illustrative embodiments of the present application and their descriptions, and do not constitute improper limitations on the present application. In the drawings:

[0034] Figure 1 A flowchart of a federal reverse knowledge distillation method based on personalized model compression disclosed by the present application is shown;

[0035] Figure 2 A flowchart of a client making a compression strategy disclosed by the present application is shown;

[0036] Figure 3 A flowchart of a client performing model compression and selecting an optimal compression model disclosed by the present application is shown;

[0037] Figure 4 A flowchart of a client local training disclosed by the present application is shown;

[0038] Figure 5 A flowchart of a client model prediction probability disclosed by the present application is shown;

[0039] Figure 6 A flowchart of a client weight calculation disclosed by the present application is shown;

[0040] Figure 7 A flowchart of a server distillation training and model updating disclosed by the present application is shown;

[0041] Figure 8 An example flowchart of an embodiment of the present application is shown;

[0042] Figure 9 A whole process flowchart of another embodiment of the present application is shown;

[0043] Figure 10A schematic block diagram of a federated reverse knowledge distillation system based on personalized model compression is shown. DETAILED DESCRIPTION

[0044] In order to enable a clearer understanding of the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the present application, however, the present application can also be implemented in other ways different from those described herein, and all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0045] The purpose of the present application is to propose a federated reverse knowledge distillation method based on personalized model compression, aiming to improve the communication efficiency and model performance in non-independent and identically distributed and device heterogeneous environments. This method allows each client to determine the compression ratio independently according to the local private data distribution characteristics, the complexity of the input sample and the local computing resource situation, and to perform structural low-rank compression on the received global model, thereby generating a personalized small model for local training. At the same time, a reverse knowledge distillation mechanism is adopted, and only the prediction results are uploaded instead of parameter updates, and the global model is aggregated and updated by the server, thereby reducing the communication overhead, improving the accuracy and enhancing the system adaptability.

[0046] The present application discloses a federated reverse knowledge distillation method based on personalized model compression, which includes the following four core processes: client personalized compression, small model training, soft logic output and weight calculation, and server-side reverse distillation. Due to the limited computing resources and bandwidth of the client devices, the server does not directly issue a large model, but sends it to the client, which then adaptively compresses it into a small model that meets its own capabilities. In each round of training, the client updates the small model using private data, then generates soft logic on the public data set and uploads it to the server. The server evaluates the weight of the client according to the accuracy and output consistency, aggregates the soft logic and performs reverse knowledge distillation, thereby optimizing the parameters of the server large model.

[0047] As shown in Figure 1 , a flowchart of the federated reverse knowledge distillation method based on personalized model compression is disclosed:

[0048] Suppose the total number of federated learning communications is , and each round of communication is numbered . In each round, the client performs rounds of local training, and the training step number is . Take the first For example, the process includes the following steps:

[0049] S1. Client formulates compression strategy: The server sends the global model of the current round to each participating client. The client formulates a personalized model compression strategy according to the characteristics of the local private data distribution, the complexity of the global model parameters, and the local computing resources, and specifies the structural proportion of each layer network that should be retained.

[0050] S2. Client performs model compression and selects the optimal configuration: The client performs layer-by-layer low-rank compression operations on the received global model according to the formulated multiple layer-by-layer candidate compression schemes, generating multiple compressed models. The client then evaluates the performance of these compressed models in terms of accuracy using the local validation dataset, and selects the compressed model with the best performance as the training model for this round.

[0051] S3. Client local training: The client performs multiple rounds of local training on the training model for this round using local private data, thereby obtaining an updated model that is more suitable for the local data distribution. This model has been optimized in structure for the client's hardware environment, and the training process is lightweight and efficient.

[0052] S4. Client model prediction probability: After training, the client uses the updated model to perform forward inference on a shared unlabeled public dataset, generating prediction probability outputs for each sample. Instead of uploading model parameters or gradients, the client uploads these prediction probability results to the server, thereby significantly reducing the amount of communication data.

[0053] S5. Client weight calculation: After receiving the prediction probability results from multiple clients, the server calculates the weight of each client according to the accuracy of the prediction probability of each client model and the consistency between the prediction probability of each client model and the server model output. This weight reflects the reliability and representativeness of the client's knowledge, which is used in the subsequent knowledge fusion process.

[0054] S6. Server distillation training updates model: The server weights and fuses the prediction probability results of all clients according to their weights to form a weighted prediction probability distribution. Then, the server takes its current model as a student model, and completes a distillation training by minimizing the difference between the output of the student model and the weighted fusion result, and finally updates the parameters of the server global model.

[0055] Specifically, in step S1, as shown in Figure 2 , a client compression strategy workflow diagram is disclosed:

[0056] I. Server issues global model

[0057] The server directly issues the parameters of the current round global model to all client devices participating in federated learning at the beginning of each round of iteration. Each client receives the same initial global model structure and parameters.

[0058] II. Client formulates personalized model compression strategy

[0059] Each client formulates its own personalized layer-by-layer compression strategy according to the local private data distribution characteristics, global model parameter complexity, and local computing resource hardware capabilities (e.g., memory, computing power), i.e., determines the structure proportion that should be retained for each layer of the network.

[0060] The client first sets a preliminary compression rate interval based on its own hardware resources.

[0061] For each layer, a low-rank decomposition method such as singular value decomposition is used to generate multiple compressed models with different compression rates (i.e., different ranks / parameter quantities).

[0062] Singular value decomposition achieves effective simplification of parameters in the form of optimal low-rank approximation.

[0063] (1) Generalized weight matrix approximation

[0064] For any weight matrix that needs to be compressed (the dimension of ), it can be approximately represented as the product of two smaller matrices and through low-rank decomposition. Among them, the dimension of matrix is , the dimension of matrix is , and the rank is much smaller than and . In other words, the original large matrix is split into the form of multiplication of two low-dimensional matrices, which significantly reduces the storage and computing overhead while maintaining the main feature information.

[0065] (2) Optimal approximation target

[0066] To maximize the recovery of the original global model's expression ability while significantly compressing the parameter quantity, the invention formalizes the "low-rank matrix approximation" as an optimization problem that minimizes the reconstruction error. Specifically, given the weight matrix to be compressed, we want to find all possible low-rank factors and In this process, we seek the pair of combinations that minimizes the reconstruction error. This error is measured using the Frobenius norm, which is the square root of the sum of the squares of the differences between all elements of the matrix, denoted as . The objective is essentially to search for a subspace in a high-dimensional matrix space such that the overall difference between the projection of the original matrix onto this subspace and the original matrix itself is minimized. Solving this optimization problem is equivalent to... Perform truncated singular value decomposition (SVD) to obtain the theoretically optimal rank. approximate.

[0067] To further improve the semantic consistency of the compressed model features, we adopted a module-level feature distillation strategy. Assume the output of the i-th layer of the original model is... The corresponding output after compression is In the The layer introduces a distillation loss. , used to measure the difference between the two.

[0068] This distillation loss occurs in a batch size of... Calculated on the sample, for each sample The loss consists of two parts: the first part is the original output. With compression output The average absolute error between them, that is, their Distance divided by feature dimension The second part is the directional consistency between the two, calculated by passing through their Euclidean distance and... The loss function is represented by taking the logarithm after function mapping. This loss function comprehensively considers the proximity of feature values ​​and the consistency of their directions, which helps to more effectively preserve the structural information of the compressed features.

[0069] (3) Correspondence between compression ratio and rank retention

[0070] The final reserved rank determined for the client Equal to its personalized optimal compression ratio With system constraint values The product of the two values ​​is then rounded down. This rank value is determined together with the base compression ratio, hardware correction, and fine-tuning verification results to ensure that each client can match the optimal compression scheme under heterogeneous conditions.

[0071] By combining the characteristics of local data, such as model complexity, number of categories, and entropy, we select the compression scheme that optimizes the accuracy of the validation set through adaptive fine-tuning and validation, i.e., determining the optimal layer retention ratio. Specifically, we adjust the compression ratio in the hardware. Three candidate compression ratio values ​​are set nearby. The three values are 1.2 times, but are limited in the interval . By fine-tuning and verifying each candidate compression rate in the set , the performance of the local data is dynamically adjusted, and the compression rate that achieves the highest accuracy on the validation set is finally selected, thereby realizing personalized and efficient compression strategy.

[0072] III. Each layer structure is clear and forms a personalized compression model

[0073] After the above adaptive fine-tuning and local evaluation process, the client finally determines the actual compression ratio for each layer of the received global model, thereby forming a unique personalized low-rank compression model, ready for subsequent local training and federated distillation.

[0074] In step S2, as shown in Figure 3 , the client performs model compression and selects the optimal compression model configuration process diagram, including:

[0075] I. Layer-by-layer compression candidate scheme generation

[0076] The client first generates multiple low-rank compressed models for the global model parameters sent by the server according to the multiple candidate compression rates set previously, which are usually determined based on hardware conditions. The specific method is as follows: for each layer to be compressed, each candidate compression rate is used to perform operations such as singular value decomposition, generating multiple compressed models with different compression degrees.

[0077] II. Local verification of compression model performance

[0078] For each compressed model generated at each compression rate, the client quickly tests the performance of the compressed model, such as accuracy, using the local validation dataset, and records the performance of all compressed model versions. This step can find the adaptability and effect difference of the compressed model for its own data under different compression intensities.

[0079] III. Selecting the best model configuration

[0080] After comparing the validation performance of all candidate compressed models, the client finally selects the best compressed model on the local validation set - that is, selects the "optimal compression rate" and the corresponding compressed model structure that best suits the current device and data. The selected model will be used for subsequent real local training and federal communication.

[0081] The client searches for the compression rate that maximizes the local validation accuracy from a set of multiple candidate compression rates ​​ And denote the optimal value as .in This represents the final determined personalized optimal compression ratio. This indicates the performance of the model on the local training set under the candidate compression ratio.

[0082] In step S3, as Figure 4 As shown, a schematic diagram of the local training process on the client side is disclosed, including:

[0083] I. Start local training on the client

[0084] After the client completes the personalized compression of the model, it starts local training of the compressed model. The structure of the compressed model has been customized and simplified in combination with local computing power and data characteristics.

[0085] II. Iterative training using local private data in multiple rounds

[0086] Each client uses its own local real-world dataset for multiple rounds of model training. This allows the compressed model to fully absorb the characteristics of the client's data distribution, improving its adaptability to local data.

[0087] III. Obtaining model updates that fit local distributions and have lightweight structures

[0088] After multiple rounds of local training, the client obtains new model parameters that take into account the characteristics of computational sensitivity, lightweight efficiency and local data distribution. These parameters will be used for subsequent knowledge distillation, efficient communication or further aggregation to improve the customization and performance of the overall federated learning system.

[0089] In step S4, as Figure 5 As shown, a schematic diagram of the client-side update model prediction probability process is disclosed, including:

[0090] I. Each customer model, after being trained, is used on an unlabeled public dataset. Output the predicted probability for each sample It doesn't upload the model parameters unchanged; instead, it first uses its current small model to process the public unlabeled image set. Each sample Perform a "soft scoring" process. Specifically, first, extract the raw output of the last layer of the model. Send into a heated of This yields a smooth probability table. The temperature here Like a knob for adjusting the heat—the higher the temperature, the better. The more even the output probability distribution, the more detailed information is preserved; the lower the temperature, the sharper the distribution tends to be. In this way, the client no longer provides hard, black-and-white labels, but rather decimal probabilities containing rich nuances, allowing the server to learn more and reducing the risk of overfitting in the local model. The entire process affects all... Treat all clients equally, every client Generate their respective ones in sequence Then, only these lightweight probability tables are sent back for the server to perform weighted aggregation.

[0091] II. Each client model will use the public dataset after completing local training. Perform inference tests and calculate the prediction accuracy of the model on this dataset, denoted as . ,in Indicates the first A client-side model.

[0092] Accuracy is calculated based on the cross-entropy loss function, which is the prediction result of the client-side compressed model on a public dataset. Compared with the true label of each sample in the dataset The comparison is performed, using cross-entropy loss to measure the consistency between the predicted results and the true labels. Public datasets. The true label record The corresponding input sample is denoted as The accuracy obtained in this way reflects the generalization performance of the client model on public data.

[0093] III. Each client obtains its own client compression model. For public datasets Predicted probability ,Will Send to the server.

[0094] In step S5, as Figure 6 As shown, a schematic diagram of the customer weight calculation process is disclosed, including:

[0095] First, the server model will process public image datasets. Perform inference to generate the corresponding predicted probability distribution, denoted as Specifically, the server model will process each sample in the public dataset. Forward propagation function input to the model In this process, the raw predicted probability output is obtained, which is the unnormalized class score, also known as... Subsequently, these predicted probability outputs will pass through a temperature coefficient of... of The function is normalized to obtain the predicted probability distribution. This probability distribution represents the confidence level of the server model across each category and is used in the subsequent knowledge distillation process.

[0096] II. When the server receives compressed models from each client for the public dataset The generated prediction probability distribution And the predicted probability distribution generated by the server's own model for the same dataset. Next, the correlation score between the two will be calculated based on the Euclidean distance function, denoted as . .

[0097] Specifically, the Euclidean distance function It compares two probability distribution vectors and measures the consistency of the prediction results in direction by calculating the cosine of the angle between them in the vector space. The value of is usually between -1 and 1. The closer the value is to 1, the more similar the prediction results of the client compression model and the server model are on the public dataset, and the higher the directional consistency. The closer the value is to 0 or is negative, the greater the difference between the two in prediction patterns.

[0098] III. Calculate the predicted probability for each client-side compression model. With server model Predicted probability Correlation between Afterwards, the client-side weight calculation module will comprehensively consider the client-side model's performance on the public dataset. Prediction accuracy Correlation This is used to determine the weight of the client in the knowledge distillation process.

[0099] Specifically, the first step is to score the client's relevance. With accuracy The scores are summed to obtain a score that measures the overall contribution of that client. This score is then divided by the sum of the scores for all clients in the same category; that is, the sum of the relevance scores and accuracy scores for all clients, thus yielding the weight coefficient for that client. This weighting coefficient reflects the proportion of each client in the joint knowledge distillation: if a client's prediction is highly similar to the server model and has high accuracy on the public dataset, its weight will be relatively higher, thus playing a greater role in updating the server model.

[0100] In step S6, as Figure 7As shown, a schematic diagram of the server distillation training and updating model process is disclosed, including:

[0101] I. Obtain the weight corresponding to each client in the client-side weight calculation module. The server will then use these weights along with the predicted probabilities for each client. A weighted calculation is performed to obtain a comprehensive weighted client prediction probability.

[0102] The specific method involves multiplying the prediction probability of each client by its corresponding weight, and then summing the weighted results of all clients to form an overall weighted prediction distribution. This weighted prediction probability is denoted as... This represents the combined opinion of all clients on the prediction results of the public dataset, based on their respective contribution proportions.

[0103] II. Obtaining the weighted customer prediction probability The server will then compare it with its own predicted probability. The distillation losses are then calculated by comparing the results.

[0104] Specifically, distillation loss is used to measure and The difference in the category prediction distribution is usually achieved through... divergence ( KL divergence is used to measure the overall deviation. It calculates the probability difference between categories one by one and sums the results to obtain an overall degree of deviation. In knowledge distillation, this loss is often combined with the cross-entropy loss function to form a comprehensive optimization objective. Here, the total number of categories... Corresponding public dataset All categories in, and and Representing the first The values ​​of the weighted client prediction probability and server prediction probability for each category.

[0105] In the specific calculation, the KL divergence will traverse all categories in the public dataset. The number of categories in the public dataset is used as... The calculation steps for each category are as follows: First, take the probability value of that category in the weighted client-predicted probability; then calculate the ratio of the probability value of that category between the weighted client-predicted probability and the server-predicted probability, and take the natural value of this ratio. The logarithm is then multiplied by the class probability of the weighted client prediction probability. This process is repeated for all classes, and the results are summed to obtain the overall distillation loss.

[0106] III.After calculating the distillation loss, we can obtain the total loss of the server model in each round of communication. This total loss consists of two parts: one part is the prediction loss of the server model on the real labels of the public dataset, usually measured by the cross-entropy loss function; the other part is the distillation loss between the server prediction probability and the weighted client prediction probability.

[0107] To balance these two parts of loss, we respectively give them weight parameters, where the weight of the task loss is , and the weight of the distillation loss is . In addition, we introduce a distillation temperature parameter to smooth the output, making the distillation process more stable and effective. In the calculation of the total loss , the distillation loss term is multiplied by the square of the temperature parameter to adjust its influence. Through this weighted fusion, the server model not only can accurately predict the real label, but also can effectively absorb the knowledge of the client compression model, achieving performance improvement.

[0108] IIII. After obtaining the comprehensive loss , we can update the parameters of the server model according to the way the client compression model updates. Specifically, the parameters of the server model in the th round are obtained by subtracting the gradient of the comprehensive loss on the model parameters multiplied by the distillation learning rate from the parameters of the th round. Here, the gradient represents the derivative of the comprehensive loss function with respect to the current server model parameters , and the calculation is based on the public dataset . The distillation learning rate is used to control the step size of parameter update, affecting the convergence speed and effect of the model. Through this gradient-based iterative update, the server model can be continuously optimized, improving the overall performance.

[0109] Actual application scenarios:

[0110] In the edge computing environment, terminal devices such as cameras, drones, intelligent sensors, etc. often encounter unknown target categories, resulting in a decrease in recognition accuracy. Traditional methods require multiple rounds of local training and uploading of model parameters by all devices, consuming a large amount of energy and bandwidth and increasing latency. The present invention uses a personalized compression client mechanism to enable devices to generate lightweight models based on their own computing power and data characteristics, and only a portion of the clients generate prediction results on a public unlabeled dataset and upload them. The server quickly integrates multi-source knowledge to update the large model using reverse knowledge distillation, and other devices can directly obtain the new model without the need for repeated training, thereby reducing communication and energy consumption and improving response speed and accuracy.

[0111] In a specific implementation, the client of the present application can be a mobile terminal, a fixed terminal or an Internet of Things terminal, and the server can be a base station or an edge server.

[0112] Embodiment 1

[0113] As shown in Figure 8 , the flow of the present embodiment is as follows:

[0114] Step 1: Server initialization and model distribution

[0115] Server initialization of server model: In the present embodiment, the server model is a ResNet-50 model, and the initial weights use ImageNet pre-training parameters. At the beginning of each round, the server broadcasts the current global model parameters to the participating clients. The clients are five GPU servers, the GPU server is an NVIDIA T4 with 64GB of video memory, and the local data consists of three categories, with 60% of the first category, 20% of the second category, and 20% of the third category. The public dataset consists of 10,000 unlabeled images with a resolution of 512x512.

[0116] Step 2: Client adaptive compression model

[0117] Each GPU performs the following individualized compression process according to its own conditions such as local data characteristics and device capabilities: calculate the data entropy to measure the diversity of the data label distribution, and calculate the data complexity to measure the image detail level. Calculate the initial compression rate and correct it according to the hardware capability, and verify and select the optimal compression rate to execute SVD compression model weights.

[0118] Step 3: Client local training and soft prediction generation

[0119] The GPU uses the compressed model to perform 5 rounds of iteration on the local private dataset to update the model parameters.

[0120] Then, using the compressed ResNet-34 model (compression rate 40%), the forward inference is performed on the 10,000 public data sets to generate the class probability distribution of each sample.

[0121] Step 4: Client uploads predicted probabilities

[0122] Each client uploads its predicted probabilities instead of the complete model parameters or gradients, significantly reducing communication overhead. In terms of data size, uploading 14 × 10000 = 140,000 floating-point numbers, approximately 0.56 MB, is a 99.3% reduction compared to uploading the full model parameters (85 MB). Gaussian noise (σ = 0.01) is added to the predicted probabilities as a privacy protection mechanism.

[0123] Step 5: The server evaluates and aggregates client weights.

[0124] A global model (uncompressed ResNet-50) generates predictions on the same public dataset; accuracy and similarity are evaluated for each client. Client aggregation weights are calculated, and the combined output from these five GPUs is obtained.

[0125] Step 6: Server backdistillation to update the model

[0126] Backpropagation is used with the Adam optimizer (lr=3e-4) to update the global model.

[0127] Example 2

[0128] like Figure 9 As shown, Figure 9 The diagram illustrates the entire process of this embodiment, which includes multiple users (User 1, User 2... User K) and a server.

[0129] Each user refers to a customer device, such as a mobile phone, tablet, or laptop, located in a different geographical location. The server refers to a base station or edge server whose coverage area includes the different geographical locations of the customers.

[0130] The process of Example 2 is as follows:

[0131] The server sends the server model to each user according to their different devices (User 1, User 2, ..., User K), and simultaneously receives data from each user to form a public image dataset, which is then sent to each user.

[0132] After receiving the model from the server, each user's client formulates a compression strategy, performs model compression, selects the optimal configuration, and then trains the model using each user's dataset.

[0133] After each user completes training on the received model, they use the trained model to output the predicted probability of the public image dataset, which is the client's soft logic.

[0134] The server accepts soft logics from all users participating in the distillation process on the public data set, while outputting its own prediction probability on the public image data set, i.e., server soft logic.

[0135] The server uses the server soft logic to operate with each client soft logic to obtain the proportion of each user in the distillation process. Then, the proportion of each user is weighted and summed with each user soft logic to obtain the weighted client soft logic.

[0136] The weighted client model soft logic is calculated with the server model soft logic to obtain the server model distillation loss in each communication round, and the server model task loss is weighted and summed to obtain the total loss of the server model. The server model is subjected to a back propagation process to update the server model.

[0137] As shown in Figure 10 The present application also discloses a federated reverse knowledge distillation system based on personalized model compression 400, comprising: a memory 401 for storing program instructions; a processor 402 for calling the program instructions stored in the memory to realize the federated reverse knowledge distillation method based on personalized model compression of the above-mentioned embodiments.

[0138] All or part of the steps in the various methods of the above-mentioned embodiments can be completed by a program controlling related hardware, and the program can be stored in a readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk memories, magnetic disk memories, magnetic tape memories, or any other readable medium capable of carrying or storing data.

[0139] The applicant states that the above description is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and it should be understood by those skilled in the art that any changes or replacements within the technical scope disclosed by the present application can be easily thought out by those skilled in the art, and all of them fall within the protection scope and disclosure scope of the present application.

Claims

1. A federated reverse knowledge distillation method based on personalized model compression, characterized in that, Includes the following steps: S1, the server sends the current global model to each client participating in federated learning. The client formulates a personalized layer-by-layer model compression strategy based on the local private data distribution characteristics, the complexity of the global model parameters, and local computing resources, and clarifies the structural proportion that should be retained in each layer of the network. S2, the client performs layer-by-layer low-rank compression on the received global model according to the multiple layer-by-layer candidate compression schemes, generating multiple compressed models. The client uses a local validation dataset to evaluate the accuracy of the multiple compressed models and selects the best-performing compressed model as the training model for this round. S3, The client uses local private data to perform multiple rounds of local training on the current training model to obtain an updated model that fits the local data distribution; S4, the client uses the updated model to perform forward inference on a shared unlabeled public dataset to generate predicted probabilities for the samples, and the client uploads the predicted probabilities to the server; S5, after receiving the predicted probabilities of multiple clients, the server calculates the weight of each client based on the accuracy of the predicted probability of each client model and the consistency between the predicted probability of each client model and the output of the server model. S6, the server weights and fuses the predicted probabilities of multiple clients to form a weighted predicted probability distribution. The server uses its current model as a student model, and completes a distillation training by minimizing the difference between the output of the student model and the weighted fusion result, and updates the parameters of the server's global model.

2. The federated reverse knowledge distillation method based on personalized model compression according to claim 1, characterized in that, Step S1 specifically includes: I. At the beginning, the server directly sends the parameters of the current global model to all the clients participating in federated learning, and each client will receive the global model with the same structure and parameters; II. Each client formulates its own personalized layer-by-layer compression strategy based on the local private data distribution characteristics, the complexity of the global model parameters, and local computing resources, and determines the proportion of structure that should be retained in each layer of the network.

3. The federated reverse knowledge distillation method based on personalized model compression according to claim 2, characterized in that, In step S1, the client formulates a personalized layer-by-layer compression strategy, including using singular value decomposition to generate different compression ratios for each layer.

4. The federated reverse knowledge distillation method based on personalized model compression according to claim 3, characterized in that, Step S2 specifically includes: (1) The client first performs low-rank compression on the global model parameters according to the different compression ratios generated. For each layer that needs to be compressed, singular value decomposition is performed using the corresponding compression ratio to generate multiple compressed models with different compression degrees. (2) For each compression model generated at the compression ratio, the client uses a local validation dataset to quickly test the accuracy of the compression model; (3) The accuracy of multiple compression models is compared, and the client selects the compression model that performs best on the local validation dataset.

5. The federated reverse knowledge distillation method based on personalized model compression according to claim 1, characterized in that, The server model is a ResNet-50 model, and the compressed model is a ResNet-34 model.

6. The federated reverse knowledge distillation method based on personalized model compression according to claim 1, characterized in that, The client is a mobile terminal, a fixed terminal, or an IoT terminal, and the server is a base station or an edge server.

7. The federated reverse knowledge distillation method based on personalized model compression according to claim 1, characterized in that, The consistency in step S5 is calculated using the Euclidean distance function.

8. A federated reverse knowledge distillation system based on personalized model compression, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke the program instructions stored in the memory to implement the federated reverse knowledge distillation method based on personalized model compression as described in any one of claims 1 to 7.

Images