Federal knowledge distillation method and application for rubber tire production parameter sharing

Abstract

The federal knowledge distillation method and application for rubber tire production parameter sharing divide the federal learning process into preheating and distillation two parts, the preheating part adopts the FedAvg federal training strategy, the distillation part utilizes the knowledge generated by each client, carries out the binned formation grouping knowledge, carries out the aggregation after the distribution to each client for the knowledge distillation in the server side.The invention proposes a new federal knowledge distillation method, through the method in the invention, the distributed training of the production line prediction model can be realized, and the model prediction accuracy of the server side is also higher than that of ordinary training.The invention can be used for the distributed training of the rubber tire production line key parameter prediction model, to improve the training efficiency, and in the case of not sharing data, the training result generated by the data can be shared, and the privacy can be well protected.

Description

Technical Field

[0001] This invention is a federated knowledge distillation method and application for sharing parameters in rubber tire production. It combines federated learning and knowledge distillation to achieve distributed training of prediction models for key parameters of industrial production lines, improve training efficiency, and share training results from different data without sharing the data itself, thus protecting the data privacy of each production line. It belongs to the fields of deep learning and tire production technology. Background Technology

[0002] The rubber tire manufacturing industry is a type of process industry, which refers to an industrial type that uses raw materials as a basis and transforms them into finished or semi-finished products through continuous or batch chemical, physical, or biological processes. These industries typically involve complex production processes that are often continuous. Although powerful modeling tools exist for process industries that can effectively handle production tasks, some problems remain. Rubber industry production lines often consist of multiple lines, each with its own data. Because these lines differ, their data is not shared, creating an "island" phenomenon. Each production line can only train independently using its own data, leading to inefficiency. Furthermore, each line requires separate training, and the limited data diversity results in insufficient data diversity. If data from multiple production lines could be used for training, the model training would be more comprehensive, leading to better performance.

[0003] Federated learning addresses the data silo problem by employing distributed training. It allows clients to share parameters without sharing data. Each client trains locally using its own data, uploading parameters to a server, which then aggregates and distributes the parameters to the server. This eliminates the need to upload data itself, thus resolving the issue of insufficient data leading to poor model performance and effectively protecting data privacy. The elimination of data uploads avoids various data security risks. Therefore, federated learning is an excellent solution.

[0004] For example, in a rubber tire production line, multiple lines need to be trained for key parameter prediction. However, training each line individually is extremely inefficient, and each line only uses its own data for training, which greatly reduces data diversity. Federated knowledge distillation can achieve distributed training, improving training efficiency. Furthermore, it shares training results without sharing the data itself, thus protecting privacy while providing the server-side model with richer training results, effectively increasing data diversity. Summary of the Invention

[0005] This invention is a federated knowledge distillation method and application for sharing parameters in rubber tire production. It mainly aims to achieve parameter sharing in rubber tire production lines, solve the problem of low training efficiency in existing models, enhance knowledge exchange between production lines through distributed training, achieve the same effect as data sharing through parameter sharing, and improve the accuracy of key production parameter prediction. It can be used for key parameter prediction in rubber tire production lines.

[0006] A time-series large-scale model knowledge distillation method for predicting key parameters of rubber tire production lines includes:

[0007] Step 1. Collect key parameter data and influencing factor data on each tire production line, clean and preprocess the collected data, divide the data using Dirichlet distribution, and generate local training data and test data for each client prediction model.

[0008] When collecting data, it is advisable to have no less than 24,000 data entries. First, delete abnormal data, including outliers and the first 100 data entries, because the data is not stable enough when the machine is first started, so it cannot be used for training.

[0009] Eighty percent of the data was designated as the training set, and twenty percent as the test set.

[0010] It is necessary to predict the key parameters of the production line based on the parameters of various influencing factors in the production line. For example, the width and thickness of the rubber tire sheet are key parameters that need to be predicted.

[0011] After splitting the training and test data, each client trains using its local training dataset and measures its individual capabilities using its local test set. The sizes of the training and test datasets for each client are randomized. The server-side model has no training data and uses the global test set to measure the generalization ability of the aggregate model.

[0012] The client-side model and server-side model mentioned above are prediction models for key parameters of the rubber tire production line, including...

[0013] LSTNet, Linear, SimpleNet.

[0014] Step 2: Warm-up phase of federated training. The FedAvg method is used to perform 3 epochs of federated training on a subset of clients to form stable sample features. The specific FedAvg training process is as follows:

[0015] The server randomly initializes model parameters and distributes them to each client model. Using a random selection strategy, a portion of clients are chosen for local training. The local training of client models uses the stochastic gradient descent (SGD) optimization algorithm to update local model parameters. After four epochs of local training, the updated parameters are uploaded to the server. The server's global model performs a weighted sum of the client parameters based on the amount of client data, then updates the global model parameters and distributes them to each client model. This training process is repeated for three epochs, completing the warm-up phase of training.

[0016] Step 3: Set all clients to evaluation mode, input local data for prediction, and input the data into the model in batches of b data points each. Collect sample features a from all clients. i Train a Gaussian mixture model classifier (a type of Gaussian mixture model) to obtain the sample feature distribution:

[0017] After the warm-up phase is complete, all clients are set to evaluation mode, local data is input for prediction, and sample features from all clients are collected:

[0018] a i ={f(x i )∣(x i ,y i )∈D n}

[0019] This invention defines the prediction model as two parts: the last linear layer is defined as a classifier, and the preceding feature extraction part is called the feature extractor, denoted as f and a. i Let a be a three-dimensional vector. i ∈R b×c×w b is the quantity per batch, c is the number of channels, w is the number of features, and the values ​​of c and w are determined by the prediction model itself. n Let x represent the training dataset for the nth client. i y i Represent the data and labels of this dataset respectively; combine all feature vectors a i ∈R b×c×w Flattened to a ′ i ∈R b×cw Let F(n) = [a′1, a′2, ...]. Then, use the collected sample features to train a Gaussian mixture model classifier to obtain the Gaussian mixture model parameters. The probability density function formula of the Gaussian mixture model is as follows:

[0020]

[0021] Where M is the number of predefined Gaussian distributions, πm Let the mixing weights of the m-th Gaussian distribution satisfy... And 0≤π m ≤1, Let F(n) be the probability density function of the m-th Gaussian distribution, and let F(n) be the observed data, where μ represents the flattened sample characteristics. m Let Σ be the mean vector of the m-th Gaussian distribution. m Let be the covariance matrix of the m-th Gaussian distribution. This formula represents the M distributions of the feature, where each distribution is considered as a group, and each group is labeled as m = {1, 2, 3…M}.

[0022] Step 4: Distillation phase of federated training:

[0023] 4.1 Select a portion of clients to perform FedAvg training once, with the training process being the same as step 2; 4.2 After training, each client generates knowledge using the local prediction model and local data, including feature knowledge and label knowledge. The Gaussian mixture model classifier maps all features and labels to the corresponding distribution to group the knowledge. The features and labels within each group are aggregated, and Gaussian noise is added. The knowledge and client model parameters are then encapsulated and uploaded to the server; 4.3 After the server performs global knowledge aggregation and parameter aggregation, it distributes the knowledge to each client for 4 epochs of knowledge distillation; (4.4) The global model repeats steps 4.2-4.3 for a total of 47 epochs (i.e., global federated training plus the warm-up phase totals 50 epochs); After training, the parameter sharing of the key parameter prediction model for the rubber tire production line is realized using federated knowledge distillation.

[0024] The steps will now be explained in detail.

[0025] 4.1 Select a subset of clients for FedAvg federated training; the training process is the same as in step 2.

[0026] 4.2 After training is complete, perform local knowledge generation: Set the local prediction model to evaluation mode, input local data into the local prediction model, and obtain the features of each sample. With predictive labels To construct knowledge, sample features are input into a Gaussian mixture model to determine which distribution the knowledge belongs to.

[0027]

[0028] n represents the client number, m represents the group number to which the knowledge belongs, and F represents the... Features after flattening This represents the probability density function, which groups sample knowledge with the same feature distribution into the same group, and aggregates knowledge within the same group to reduce upload load. An averaging method is used to aggregate knowledge within the same group.

[0029]

[0030] Where n represents the client number, P is the number of samples belonging to the T distribution in the client, and ε is the Gaussian noise added to the sample features. Dividing all samples into groups completes the knowledge generation and grouping of the local model. Finally, the aggregated features, aggregated predicted labels, and the number of samples in each group are packaged into knowledge and uploaded to the server. This significantly reduces the load on the upload server, and the addition of Gaussian noise also effectively protects data privacy.

[0031] 4.3 After the server performs global knowledge aggregation and parameter aggregation, it distributes the knowledge to each client for 4 epochs of knowledge distillation;

[0032] 4.3.1 The server performs global knowledge aggregation and parameter aggregation, both using weighted summation. Then, the server distributes the aggregated knowledge and model parameters to the client's prediction model.

[0033] 4.3.2 Each client updates its local parameters based on the global parameters;

[0034] 4.3.3 Select a portion of clients for local training, and after training, extract sample features for classification;

[0035] 4.3.4 Calculate the MSE loss for the sample features and the global features in the same group, and use it as the feature distillation loss;

[0036] 4.3.5 Input the global feature knowledge into the classifier of the client's local model, and then calculate the MSE loss as the label distillation loss by combining its output with the global label knowledge;

[0037] 4.3.6 The MSE loss is calculated by combining the client's local model prediction with the true label and used as the prediction loss. This loss is then weighted and summed with the two distillation losses mentioned above to form the total loss. Backpropagation is then performed, and the parameters are updated using the stochastic gradient descent (SGD) optimization algorithm.

[0038] 4.3.1 Specifically:

[0039] After receiving the knowledge from all trained clients, the server needs to perform global knowledge aggregation using a weighted average method. Let client n generate local knowledge K in round t (t = 1, 2, ..., 47):

[0040]

[0041] This includes the average sample features of each group. Average predicted label Sample size m represents the group number. The server calculates a weighted sum of the knowledge from each client based on the number of clients belonging to that group.

[0042]

[0043] in Let n be the number of client n samples that belong to a Gaussian distribution m (i.e., m groups). This calculates the total number of samples belonging to the Gaussian distribution m. The knowledge aggregated locally on the client is aggregated on the server to obtain global knowledge. The client model parameters are also weighted and summed. Then, the aggregated knowledge and model parameters are distributed to all clients. Each client updates its local model parameters based on the global parameters, and then uses this knowledge for subsequent knowledge distillation.

[0044] 4.3.3-4.3.6 are detailed as follows: Client-side knowledge distillation is divided into feature distillation and label distillation. Feature distillation involves extracting sample features for classification after training, flattening the sample features, grouping them using a Gaussian mixture model, and then calculating the MSE loss with the global knowledge features of the corresponding group, which serves as the feature distillation loss. Label distillation involves obtaining labels from the global feature knowledge through the client's classifier, then calculating the MSE loss with the global label knowledge of the corresponding group, which serves as the label distillation loss. In addition, the client's output and the true labels are used to calculate the MSE loss as the supervision loss. Finally, a weighted sum of the three losses is calculated, denoted as L. total The parameters are updated using gradient backpropagation and stochastic gradient descent optimization algorithms.

[0045] L total =L ce +γL feat +λL logit

[0046] Where γ and λ are the loss weights of the feature distillation loss and the label distillation loss, respectively, and L ce To monitor the loss, L feat For characteristic distillation loss, L logit This refers to the label distillation loss.

[0047] The global model repeats the knowledge distillation step for 47 epochs, meaning the total global federated training plus the warm-up phase is 50 epochs. After training, federated knowledge distillation is used to achieve parameter sharing among the key parameter prediction models for the rubber tire production line. Client models with strong personalization capabilities (i.e., outperforming the global model) can be used for their own production line key parameter prediction, while client models with weaker personalization capabilities can use the global model for key parameter prediction to achieve better results.

[0048] Advantages of the invention

[0049] This invention makes a series of improvements to the optimization of industrial prediction models. First, it solves the problem of low efficiency caused by training different models separately. Through distributed training, models from different production lines are trained simultaneously, which greatly improves training efficiency. In addition, it improves the prediction accuracy of the server-side model. This improvement is achieved through parameter sharing. Different production lines do not need to share data, thus protecting the data privacy of different production lines. However, by uploading parameters to share parameters, the diversity of data for model training is indirectly increased, thereby greatly improving the generalization ability of the server model and thus improving the prediction accuracy.

[0050] This invention innovatively introduces a binning strategy, transforming a continuous prediction problem into a discretized classification task. Specifically, by structurally binning the feature space, the original regression or ranking problem is transformed into a multi-class classification problem, thereby achieving discrete modeling of the feature distribution. This transformation not only reduces the complexity of distribution alignment but also enables explicit matching of local feature distributions between different clients using class probability distributions.

[0051] To refine the feature distribution matching process, this invention employs a Gaussian Mixture Model (GMM) to probabilistically cluster client sample features. GMM models the feature space through a linear combination of multiple Gaussian components and iteratively optimizes the weights, mean, and covariance parameters of each component using the Expectation-Maximization (EM) algorithm. Within the federated knowledge distillation framework, each client locally calculates the posterior probability of its sample features on each GMM component, thereby dividing the features into multiple subgroups with similar statistical properties. Attached Figure Description

[0052] Figure 1 This is a diagram illustrating the training process of federated knowledge distillation in this invention.

[0053] Figure 2 This is a diagram illustrating the knowledge transfer process in this invention.

[0054] Figure 3 This is a diagram illustrating the knowledge generation process in this invention.

[0055] Figure 4This is a schematic diagram of the characteristic distillation process in this invention.

[0056] Figure 5 This is a schematic diagram of label distillation in this invention.

[0057] Figure 6 This is a comparison chart of the server prediction loss of the method in this invention and the benchmark method. Detailed Implementation

[0058] This invention proposes a novel federated knowledge distillation method that enables distributed training of a production line key parameter prediction model, improving training efficiency and achieving higher prediction accuracy for the server-side model compared to ordinary training and other federated learning methods. It can be used for parameter sharing in the prediction of key parameters for rubber tire production lines.

[0059] like Figure 1 First, data should be collected from the rubber tire production line, ideally at least 24,000 records. Abnormal data, including outliers, should be removed initially, with the first 100 records being unsuitable for training as the data is unstable upon machine startup. Then, 80% of the data should be allocated to the training set and 20% to the test set. Key parameters of the production line need to be predicted based on various influencing factors; the width of the rubber tire sheet is one such key parameter.

[0060] It is necessary to predict the key parameters of the production line based on the parameters of various influencing factors in the production line. The width and thickness of the rubber tire sheet are key parameters that need to be predicted.

[0061] Factors affecting tire rubber sheet width include: calender main motor current (A), calender speed (mpm), calender outlet temperature (°C), calender pull-out roller speed (mpm), conveyor belt speed (mpm), calender trimmer blade spacing (mm), calender upper roller temperature control (°C), calender lower roller temperature control (°C), extruder main motor current (A), extruder speed (rpm), extruder die pressure (MPa), extruder die temperature (°C), extruder screw end pressure (MPa), extruder die temperature control (°C), extruder plasticizing 1 temperature control (°C), extruder plasticizing 2 temperature control (°C), extruder barrel temperature control (°C), and extruder screw temperature control (°C). Among these, width is a key parameter that needs to be predicted.

[0062] Factors affecting tire rubber thickness include: extruder head temperature (°C), extruder head pressure (MPa), extruder screw end pressure (MPa), extruder current (A), extruder speed (mpm), calender roll gap (mm), calender outlet rubber temperature (°C), calender current (A), calender speed (mpm), extruder screw section temperature (°C), extruder plasticizing temperature (°C), extruder extrusion section temperature (°C), calender upper roll temperature (°C), and calender lower roll temperature (°C). Among these, thickness is a key parameter that needs to be predicted.

[0063] A Dirichlet distribution is used to generate local training data partitions, dividing the training and test data. Each client uses one of the training datasets for training and uses its local test set to measure individual capabilities. The sizes of the training and test datasets for each client are randomized. The server-side model has no training data; instead, a global test set is used to measure the generalization ability of the aggregated model.

[0064] After the dataset is divided, a warm-up phase for training is carried out. The FedAvg method is used to perform federated training on multiple LSTNet networks for 3 epochs to form stable sample features.

[0065] The server randomly initializes the LSTNet network parameters and distributes them to each client network. All clients are selected for local training. The local training of the client models uses the stochastic gradient descent (SGD) optimization algorithm to update the parameters. After four epochs of local training, the updated parameters are uploaded to the server. The server's global model performs a weighted sum of the client parameters and then updates the global model parameters, which are then distributed to each client network.

[0066] After the warm-up phase ends, the feature classifier training begins. The Gaussian mixture model is trained by collecting features from all clients to obtain the distribution of sample features. The flattened features F(n) are then input into the Gaussian mixture model classifier to obtain the Gaussian mixture model parameters, and all sample features are divided into 5 distributions.

[0067]

[0068] Next, we perform the distillation part of the training, enabling all clients to undergo FedAvg federated training. After training is complete, local knowledge generation is performed. The local model is set to evaluation mode, and local samples are input into the local model to extract the features of each sample. With predictive labels To construct knowledge, the sample features are input into a Gaussian mixture model to determine which distribution the sample knowledge belongs to.

[0069]

[0070] Sample knowledge with the same feature distribution is grouped together, and knowledge aggregation is performed within the same group to reduce upload load. An averaging method is used to aggregate knowledge within the same group.

[0071]

[0072] Where P is the number of samples belonging to the T-distribution in the client, and ε is the Gaussian noise added to the sample features. Dividing all samples into groups completes the knowledge generation and binning of the local model. Finally, the aggregated features, aggregated predicted labels, and the number of samples in each group are packaged into knowledge and uploaded to the server. The knowledge generation process is as follows: Figure 3 As shown.

[0073] After receiving the knowledge from all trained clients, the server performs global knowledge aggregation, such as... Figure 2 As shown, using a weighted average method, let client n generate local knowledge K in round t:

[0074]

[0075] This includes the average sample feature f, the average predicted label z, the number of samples o, and m representing the group number for each group. The server performs a weighted summation of the knowledge from each client:

[0076]

[0077] By aggregating on the server side, global knowledge is obtained. Parameter aggregation is the same as the FedAvg method in step 3. Then, the aggregated knowledge and parameters are distributed to all clients. The clients will use this knowledge to perform knowledge distillation and update their local model parameters.

[0078] Each client updates its local parameters based on the global parameters, and then all clients begin local knowledge distillation training.

[0079] Three loss functions are calculated progressively. After training, sample features are extracted and classified. The sample features are grouped using a Gaussian mixture model, and then the MSE loss is calculated with the global knowledge features of the corresponding group as the feature distillation loss L. feat ,like Figure 4 As shown.

[0080] The global feature knowledge is processed by the client's classifier to obtain labels. Then, the MSE loss is calculated using these labels and the global label knowledge of the corresponding group, which serves as the label distillation loss L. logit ,like Figure 5 As shown.

[0081] The MSE loss is calculated using the client's output and the ground label, and used as the supervision loss L. ceThen, a weighted sum of the three losses is calculated, and the parameters are updated using gradient backpropagation and stochastic gradient descent optimization algorithms.

[0082] L total =L ce +γL feat +λL logit

[0083] Where γ and λ are the loss weights of feature distillation loss and label distillation loss, respectively, and are both 0.05. Each client repeats the above knowledge distillation for 4 epochs, and the global model update is performed for 47 epochs, that is, the global federated training plus the warm-up phase is a total of 50 epochs.

[0084] After testing, the final result is as follows: Figure 6 The method used in this study outperforms other methods on various models, demonstrating the superiority of this invention.

[0085] This invention proposes a novel federated knowledge distillation method that enables distributed training of a production line key parameter prediction model, improving training efficiency and achieving higher prediction accuracy for the server-side model compared to ordinary training and other federated learning methods. It can be used for parameter sharing in the prediction of key parameters for rubber tire production lines.

Claims

1. A federated knowledge distillation method for sharing rubber tire production parameters. Its features include the following steps: (1) Collect key parameter data and its influencing factor data on each tire production line, clean and preprocess the collected data, divide the data using Dirichlet distribution, and generate local training data and test data for each client prediction model. (2) Warm-up phase of federated training: FedAvg method is used to perform federated training on some clients for 3 epochs to form stable client model parameters; (3) Set all clients to evaluation mode, input local training data in batches for prediction, with each batch containing b data entries, and collect sample features from all clients. The Gaussian mixture model classifier is trained to obtain the distribution of sample features; (4) Distillation phase of federal training: (4.1) Select a portion of clients to perform FedAvg training once. The training process is the same as step (2). (4.2) After training, each client generates knowledge using the local prediction model and local data, including feature knowledge and label knowledge. The Gaussian mixture model classifier maps all features and labels to the corresponding distribution to group the knowledge. The features and labels in each group are aggregated and Gaussian noise is added. The knowledge and client model parameters are then encapsulated and uploaded to the server. (4.3) After the server performs global knowledge aggregation and parameter aggregation, it distributes the knowledge to each client for 4 epochs of knowledge distillation; (4.4) Repeat steps (4.2)-(4.3) for a total of 47 epochs for the global model. After training, the parameter sharing of the key parameter prediction model for the rubber tire production line is realized by using federated knowledge distillation. The specific steps (4.3) are as follows: (4.3.1) The server performs global knowledge aggregation and parameter aggregation. The aggregation method is weighted summation. Then the server distributes the aggregated knowledge and model parameters to the client prediction model. (4.3.2) Each client updates its local parameters based on the global parameters; (4.3.3) Select a portion of clients for local training, and after training, extract sample features for classification; (4.3.4) Calculate the MSE loss for the sample features and the global features in the same group as them, and use it as the feature distillation loss; (4.3.5) Input the global feature knowledge into the classifier of the client's local model, and then calculate the MSE loss with the global label knowledge as the label distillation loss; (4.3.6) The MSE loss is calculated by combining the client's local model prediction with the real label and used as the prediction loss. This loss is then weighted and summed with the two distillation losses mentioned above to form the total loss. Backward gradient propagation is then performed, and the parameters are updated using the stochastic gradient descent (SGD) optimization algorithm.

2. The method according to claim 1, characterized in that: The key parameters include tire rubber width, tire rubber thickness, and temperature of the conveyor belt during tire descent; among which: Factors affecting tire rubber width include: extruder temperature in each temperature control channel (°C), extruder head pressure (MPa), extruder head temperature (°C), extruder screw motor current (A), extruder screw speed (rpm), calender roller temperature (°C), calender pressure roller speed (mpm), calender pressure roller current (A), calender outlet rubber temperature (°C), calender trimming knife distance (mm), calender roller distance (mm), calender pull-out roller speed (mpm), and conveyor belt speed (mpm). Among these, width is a key parameter that needs to be predicted. Factors affecting tire rubber thickness include: extruder head temperature (°C), extruder head pressure (MPa), extruder screw end pressure (MPa), extruder current (A), extruder speed (mpm), calender roll gap (mm), calender outlet rubber temperature (°C), calender current (A), calender speed (mpm), extruder screw section temperature (°C), extruder plasticizing temperature (°C), extruder extrusion section temperature (°C), calender upper roll temperature (°C), and calender lower roll temperature (°C). Among these, thickness is a key parameter that needs to be predicted.

3. The method according to claim 1, characterized in that: Step (2) is as follows: 1) The server randomly initializes the model parameters and distributes them to each client; 2) Select a portion of clients for local training, and use the stochastic gradient descent (SGD) optimization algorithm to update the local model parameters. Local training is performed for 4 epochs. 3) Once training is complete, the client uploads the local parameters to the server. 4) The server performs a weighted sum of the parameters for each client, then updates the server-side model parameters, and then distributes the results to each client to update the client-side model parameters; 5) Repeat steps 2)-4) for 3 epochs.

4. The method according to claim 1, characterized in that: The specific steps (3) are as follows: The client-side prediction model is defined in two parts: the last linear layer is defined as the feature classifier, and the preceding part, defined as the feature extraction part, is called the feature extractor, denoted as […]. f , For the sample features collected from all clients, where a i Let a be a three-dimensional vector. i ∈R b×c×w b is the quantity per batch, c is the number of channels, w is the number of features, and D n Let x represent the training dataset for the nth client. i y i Represent the data and labels of this dataset respectively; combine all feature vectors a i ∈R b ×c×w Flattened ∈R b×cw , recorded as F ( n )=[ , …], then use F ( n The Gaussian mixture model (GMM) classifier is trained to obtain the GMM parameters. The probability density function of the GMM is as follows: Where M is the number of predefined Gaussian distributions, Let the mixing weights of the m-th Gaussian distribution satisfy the following condition: And 0≤π m ≤1, Let be the probability density function of the m-th Gaussian distribution. Let be the mean vector of the m-th Gaussian distribution. Let be the covariance matrix of the m-th Gaussian distribution; this formula represents the M distributions of the feature, each distribution is regarded as a group, and each group is labeled m, m∈{1,2,3…M}.

5. The method according to claim 1, characterized in that: The specific steps (4.2) are as follows: Set the local prediction model to evaluation mode, input local data into the local prediction model, and obtain the features of each sample. With predictive labels The knowledge is composed of sample features, which are then input into a Gaussian mixture model classifier to determine which distribution the knowledge belongs to. n represents the client number. This indicates the group number to which the knowledge belongs. Indicates will Features after flattening Represents the probability density function. express Belongs to the The distribution is divided into two groups; knowledge samples with the same feature distribution are grouped together and aggregated within the same group to reduce upload load: an averaging method is used to aggregate knowledge within the same group. Where n represents the client number, P is the number of samples belonging to the T distribution in the client, and ε is the Gaussian noise added to the sample features; after dividing all the samples into groups, the knowledge generation and grouping of the local model are completed. Finally, the aggregated features, aggregated prediction labels, and the number of samples in each group are packaged into knowledge and client model parameters and uploaded to the server.

6. The method according to claim 1, characterized in that: The specific steps (4.3.1) are as follows: After receiving the knowledge from all trained clients, the server needs to perform global knowledge aggregation using a weighted average method. Let client n generate local knowledge in round t. : Where t = 1, 2, ..., 47, Includes the average sample features of each group Average predicted label Sample size Where m is the group number, the server performs a weighted summation of the knowledge of each client based on the number of clients belonging to that group: in Let m be the number of samples in the m-th group, where n is a Gaussian distribution. Calculate the total number of samples belonging to the Gaussian distribution m; aggregate the knowledge aggregated locally on the client through the server to obtain global knowledge, and also perform weighted summation on the client model parameters. Then distribute the aggregated knowledge and model parameters to all clients, and each client updates its local model parameters according to the global parameters. Finally, use this knowledge for subsequent knowledge distillation.

7. The method according to claim 1, characterized in that: The specific steps (4.3.3)-(4.3.6) are as follows: The client-side knowledge distillation is divided into feature distillation and label distillation. Feature distillation involves extracting sample features for classification after training, flattening the features, grouping them using a Gaussian mixture model, and then calculating the MSE loss with the global knowledge features of the corresponding group, which serves as the feature distillation loss. Label distillation involves using the global feature knowledge to obtain labels from the client-side classifier, then calculating the MSE loss with the global label knowledge of the corresponding group, which serves as the label distillation loss. In addition, the client-side output and the true labels are used to calculate the MSE loss as the supervision loss. Finally, a weighted sum of the three losses is calculated, denoted as... The parameters are updated using gradient backpropagation and stochastic gradient descent optimization algorithms. Where γ, These are the loss weights for feature distillation loss and label distillation loss, respectively. To monitor losses, Characteristic distillation loss, This refers to the label distillation loss.

Images