A Contrastive Dual-Focus Knowledge Distillation Federated Learning Approach for Industrial Heterogeneous Equipment

Abstract

The invention belongs to the field of information technology, and discloses a comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment. A comparative dual-focus knowledge distillation federated learning model is constructed by one server and N clients. Each client is pre-trained on a local data set to obtain an initialized local model. The server receives knowledge from multiple clients, and promotes the server model to evolve into a server-side global model with higher accuracy and greater generalization capability through a dual-focus distillation strategy. The client calculates the feature contrast loss and relative entropy loss of the global model and the local model output. At the same time, the feature contrast loss of the local model and the global model and the previous round of local model output, as well as the cross entropy between the local model output and a hard label are calculated. Through the joint training of the knowledge distillation loss term and the local supervision loss term, the local model can obtain a higher global accuracy while maintaining the original data distribution advantage.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and in particular relates to a comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment. Background Art

[0002] With the rapid development of 6G networks, the industrial Internet is undergoing profound changes, among which the improvement of edge computing capabilities has become a key driving force. 6G networks not only provide high-speed, low-latency information transmission, but also enable a large number of distributed client devices to efficiently process data locally and perform preliminary model training through edge computing technology. This capability optimizes the data processing process. Client devices can analyze and refine key information in real time, and then upload the refined data and preliminary models to the server for further integration and optimization, thereby generating a high-performance comprehensive model. This model effectively reduces the amount of data transmission, reduces the computing burden of the server, and realizes efficient utilization and optimal configuration of resources, especially in process industries. The process industry has highly automated and complex production processes, and has strict requirements for real-time monitoring and high security.

[0003] However, with the surge in distributed intelligent devices, how to achieve efficient collaboration while ensuring data privacy has become a new challenge. Federated learning, as a distributed model training technology that does not require centralized data storage, has attracted much attention. By distributing model training tasks to each client device, federated learning not only protects data privacy, but also makes full use of the computing power of client devices to achieve distributed collaborative learning. Classic federated learning algorithms represented by Federated AveragingFedAvg update the model locally through each client device, and transmit the model update rather than the original data to the server for aggregation, thereby improving the global model performance. However, the data distribution in industrial scenarios usually presents the characteristics of non-independent identically distributed non-IID, and the client models are heterogeneous. This situation easily deviates from the global optimal solution during the model optimization process, thereby affecting the accuracy of the model.

[0004] For example, the Chinese invention patent with publication number CN 118709753 A discloses a non-independent and identically distributed data processing framework for industrial equipment clusters based on federated learning, and proposes a non-independent and identically distributed data processing framework for industrial equipment clusters based on federated learning, aiming to cope with the challenges of non-independent and identically distributed data that are prevalent in industrial equipment clusters. The framework combines an optimized reference sample selection strategy with federated learning technology to mitigate the adverse effects caused by data imbalance and missing data feature categories. However, the patent relies on oversampling technology to adjust the local data set in order to screen out clients with balanced data volume and relatively complete structure. The effectiveness of this strategy is based on a key assumption that at least some clients in the system must have relatively complete data structures. In addition, the framework does not consider the possible model heterogeneity between clients in its design, that is, different clients may use models with different structures or parameters for training. This limitation may limit the wide applicability of the framework in complex industrial environments.

[0005] For example, the Chinese invention patent with publication number CN 118279639 A discloses a medical image classification method of knowledge distillation federated learning under neural architecture search, and proposes a medical image classification method of knowledge distillation personalized federated learning under neural architecture search, which aggregates the heterogeneous models of each client by only sharing the output layer of the model. However, although the patent successfully supports the heterogeneity of client and server models in the federated learning method by introducing the knowledge distillation mechanism, the traditional knowledge distillation method it adopts may suffer a significant performance decline when dealing with industrial application environments with extremely significant differences in distribution scenarios.

[0006] Therefore, a contrastive dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment is needed, which can quickly respond to and accurately classify intelligent classifiers of industrial images. Summary of the invention

[0007] In view of the shortcomings of the above-mentioned background technology, the present invention proposes a contrastive bifocal distillation for federated learning method (CBDFL) for industrial heterogeneous equipment, which aims to improve the learning efficiency in non-IID data scenarios through knowledge transfer rather than model structure sharing.

[0008] To solve this problem, the CBDFL method is proposed. This method follows the basic paradigm of federated learning and is mainly composed of two core components: server and client. The main task of the server is to receive knowledge from multiple clients and promote the evolution of the server model into a server-side global model with higher accuracy and better generalization ability through the dual-focus distillation strategy. The main task of the client is to calculate the feature contrast loss and relative entropy (Kullback-Leibler Divergence KL divergence) loss between the global model and the client's local model output, and use them as the loss term of knowledge distillation. At the same time, the client also needs to calculate the feature contrast loss between the local model and the global model and the previous round of local model output, as well as the cross entropy between the local model output and the hard label, and use them as the local supervision loss term. Through the joint training of the knowledge distillation loss term and the local supervision loss term, the local model can achieve higher global accuracy while maintaining the original data distribution advantage.

[0009] The object of the present invention is achieved by the following technical solution: a comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment, the method comprising the following steps:

[0010] Step S1: Construct a comparative dual-focus knowledge distillation federated learning model through 1 server and N clients to complete K-class classification tasks. Each client has a local dataset. ; Each client performs pre-training on the local data set to obtain an initialized local model;

[0011] Step S2: The client and the server conduct federated learning training. In the qth round of communication between the client and the server, the knowledge output by the local model of each client is aggregated on the server to obtain knowledge Z C q =[ z C q 1 , z C q 2 , ⋯, z C q t , ⋯, z C q K ] , used to update the global model. In the i-th iteration, the global model The output knowledge is Z S i q =[ z S i q 1 , z S i q 2 , ⋯, z S i q t , ⋯, z S i q K ] ;

[0012] Step S3: Using the softmax function, based on the aggregated knowledge and global model Output of knowledge , respectively calculate the implicit probability and explicit probability matrices of the client and the implicit probability and explicit probability matrices of the server at the i-th iteration of the global model, and obtain the implicit loss term through KL divergence and explicit loss terms , using two hyperparameters , Implicit loss and explicit loss terms Combined as a dual-focus knowledge distillation loss term ;

[0013] Step S4: In the i-th iteration, use the dual-focus knowledge distillation loss term For the global model Update and generate a new round of global model , until i=E S , and obtain the global model , global model The output knowledge is Z S q =[ z S q 1 , z S q 2 , ⋯, z S q k ] , where E S The number of iterations for the set server;

[0014] Step S5: The server sends the global model and the knowledge it outputs Sent to each client, in the fth iteration, the local model The output knowledge is Z C n f q+1 =[ z C n f q+1 1 , z C n f q+1 2 , ⋯, z C n f q+1 K ] , ;

[0015] Step S6: Calculate the global model using KL divergence Output knowledge With local model Output knowledge The KL divergence loss term ;

[0016] Step S7: Based on the global model and local models The output eigenvalues ​​calculate the similarity matrix M and obtain the contrast loss term ,in, represents the average value;

[0017] Step S8: Contrastive loss term and KL divergence loss term use , The two hyperparameters are linked together as the total loss term for client distillation ;

[0018] Step S9: In the fth iteration, use the client to distill the total loss term For local models Update and generate local model ;

[0019] Step S10: Input local data into the local model In the above example, the cross entropy between the true label and the predicted result is calculated as a hard indicator supervision loss term. ;

[0020] Step S11: The local model of the f-1th iteration The output feature values ​​are regarded as negative samples, and the global model The output value is regarded as a positive sample and the comparative supervision loss term is calculated ;

[0021] Step S12: Use the hard indicator supervision loss item and the comparison supervision loss ,use , The two hyperparameters are linked together as the client model supervision total loss term ;

[0022] Step S13: In the fth iteration, the client model is used to supervise the total loss term , for the local model Update and generate a new round of local models ,

[0023] Step S14, repeat steps S6-S13 until f=E C , get the local model , where E C The number of iterations for the set client;

[0024] Step S15: Repeat steps S2-S4 to aggregate the knowledge output by the local models of each client on the server to obtain knowledge , used to update the server-side global model ;

[0025] Step S16: Repeat steps S2-S15 until the number of communication rounds between the client and the server reaches the set value E, and the training is completed.

[0026] Furthermore, in step S3, the softmax function is used to calculate the aggregated knowledge and global model Output of knowledge , respectively calculate the implicit probability and explicit probability matrices of the client and the implicit probability and explicit probability matrices of the server at the i-th iteration of the global model, and obtain the implicit loss term through KL divergence and explicit loss terms , using two hyperparameters , Implicit loss and explicit loss terms Combined as a dual-focus knowledge distillation loss term , specifically:

[0027] Based on aggregated knowledge Calculate the non-target probability of the client separately and target probability :

[0028] ;

[0029] in, is the dominant temperature coefficient,

[0030] According to the global model Output of knowledge Calculate the non-target probability of the server at the i-th iteration of the global model respectively and target probability :

[0031] ;

[0032] Then we get the client's explicit probability matrix and the server’s explicit probability matrix at the i-th iteration of the global model ;

[0033] Obvious loss term is obtained through KL divergence for:

[0034] ;

[0035] Define the implicit probability of the client , where each element is represented by:

[0036] ;

[0037] Define the implicit probability of the server at the i-th iteration of the global model , where each element It is expressed as:

[0038] ;

[0039] in, is the implicit temperature coefficient,

[0040] Obtain the implicit loss term through KL divergence for:

[0041] ;

[0042] Then the dual-focus knowledge distillation loss term for:

[0043] .

[0044] Furthermore, in step S4, in the i-th iteration, the loss term is distilled using the dual-focus knowledge For the global model Update and generate a new round of global model , specifically: ,in, is the model learning rate.

[0045] Further, in step S6, the global model is calculated using KL divergence Output knowledge With local model Output knowledge The KL divergence loss term , specifically:

[0046] Define the output probability of the server :

[0047] ;

[0048] Among them, each element It is expressed as:

[0049] ;

[0050] Define the client output probability at the fth iteration of the local model :

[0051] ;

[0052] Among them, each element It is expressed as:

[0053] ;

[0054] Where T is the temperature coefficient of client distillation;

[0055] Then the KL divergence loss term for:

[0056] .

[0057] Further, in step S8, the client distills the total loss item The expression is:

[0058] .

[0059] Furthermore, in step S10, the hard indicator supervision loss item The expression is:

[0060] ;

[0061] in, , are the local models in the fth iteration respectively. The output results are compared with the true labels of the client's local dataset.

[0062] Furthermore, in step S11, the comparison supervision loss term The expression is:

[0063] ;

[0064] in, is the local model of the fth iteration The output result is, is the local model of the f-1th iteration The output result is, For the global model The output result is, To compare the temperature coefficient of the supervision loss term, Represents cosine similarity.

[0065] Furthermore, in step S12, the client model supervises the total loss term The expression is:

[0066] .

[0067] Further, in step S13, in the fth iteration, the client model is used to supervise the total loss term , for the local model Update and generate a new round of local models , specifically: = ,in, is the model learning rate.

[0068] The present invention has the following technical effects: (1) The present invention proposes CBDFL, which divides the traditional knowledge distillation loss term into explicit loss term and implicit loss term, and introduces different temperature coefficient mechanisms. By adjusting the smoothness of the distillation process, it not only retains the stable contribution of the target knowledge, but also maximizes the potential of non-target knowledge to improve model performance. This method emphasizes the collaborative supervision of the global model and the local model, while improving the performance of the global model, retaining and strengthening the local characteristics, thus providing a new solution for the application of federated learning in non-IID scenarios.

[0069] (2) In order to address the heterogeneity challenge caused by the different models that client devices in the process industry may adopt due to differences in their environments, this paper innovatively combines federated learning and knowledge distillation technology. Through the alternative strategy of knowledge transfer, direct sharing of model parameters or gradients is avoided, thereby successfully achieving effective support for model heterogeneity. This method not only breaks through traditional limitations, but also provides a new solution for the collaboration of heterogeneous devices in the process industry.

[0070] (3) In response to the performance degradation problem faced by current federated learning methods that support model heterogeneity when processing client data sets with significant distribution differences, the present invention further proposes two strategies: dual-focus knowledge distillation and supervised comparative knowledge distillation. Among them, the dual-focus knowledge distillation strategy enables the client to focus more on extracting implicit support information when performing knowledge distillation to the server, thereby significantly improving the global performance of the server. The supervised comparative knowledge distillation strategy ensures that when the server performs distillation to the client, the client model can maintain good adaptability to local data while approaching the global optimal solution. These two strategies complement each other and jointly achieve a more efficient two-way distillation effect, significantly improving the overall performance of the federated learning system. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Flowchart for comparing the dual-focus knowledge distillation federated learning method CBDFL;

[0072] Figure 2 Schematic diagram of the CBDFL method;

[0073] Figure 3Schematic diagram of the server-side dual-focus knowledge distillation strategy;

[0074] Figure 4 Schematic diagram of the client-side supervised comparative knowledge distillation strategy. DETAILED DESCRIPTION

[0075] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0076] A comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment, the specific steps are as follows:

[0077] The dual-focus knowledge distillation federated learning model is constructed by 1 server and N clients to complete K-class classification tasks. The total number of communication rounds between the server and the client is E, and the total number of iterations of the server is E. S , the total number of iterations of the client is E C , each client has a local dataset ;

[0078] Step S1: Each client performs pre-training on a local data set and initializes a local model.

[0079] Step S2: In the qth round of communication between the client and the server, the knowledge output by the local model of each client is aggregated on the server to obtain knowledge Z C q =[ z C q 1 , z C q 2 , ⋯, z C q t , ⋯, z C q K ] , used to update the global model.

[0080] Step S3: Aggregate knowledge Divide into non-target knowledge and target knowledge, and convert non-target knowledge and target knowledge into non-target probabilities respectively and target probability :

[0081]

[0082] in, is the dominant temperature coefficient;

[0083] At the same time, in the i-th iteration of the global model, the global model Output of knowledge Z S i q =[ z S i q 1 , z S i q 2 , ⋯, z S i q t , ⋯, z S i q K ] Divide it into non-target knowledge and target knowledge, and convert it into the non-target probability of the server and target probability :

[0084]

[0085] The target probability and non-target probability of the client and server are combined into two explicit probability matrices, namely and ,in, b C =[ p t C , p non-t C ] , b S i =[ p t, i S , p non-t, i S ] ;

[0086] Calculate the client's explicit probability matrix and the explicit probability matrix output by the server at the i-th iteration The KL divergence of gets the explicit loss term of the i-th iteration :

[0087]

[0088] Define the implicit probability of the client p ̂ C =[ p ̂ C q 1 ,⋯, p ̂ C q t-1 , p ̂ C q t+1 , ⋯, p ̂ C q K ] , where each element It is expressed as:

[0089]

[0090] Define the implicit probability of the server output at the i-th iteration of the global model p ̂ S i =[ p ̂ S i q 1 , ⋯ , p ̂ S i q t-1 , p ̂ S i q t+1 , ⋯, p ̂ S i q K ] , where each element It is expressed as:

[0091]

[0092] in, is the implicit temperature coefficient;

[0093] Calculating client hidden probability and the implicit probability of the server output at the i-th iteration The KL divergence of gets the implicit loss term of the i-th iteration .

[0094]

[0095] The explicit loss term of the i-th iteration and hidden loss items pass , The two hyperparameters are linked as the server's global model in the i-th iteration of the dual-focus knowledge distillation loss term :

[0096]

[0097] and The two independent temperature coefficients introduced are used to adjust the different influence capabilities of the explicit probability matrix and the implicit probability.

[0098] Step S4: In the i-th iteration of the global model, the loss term is distilled through the dual-focus knowledge For the global model Update and generate a new round of global model ,

[0099]

[0100] in, is the model learning rate;

[0101] Until the number of iterations reaches the set E S , then the global model updated in the qth round of communication between the client and the server is obtained .

[0102] Step S5: The server sends the global model and its output knowledge Z S q =[ z S q 1 , z S q 2 , ⋯, z S q K ] Sent to each client, the client's local model in the fth iteration, the local model The output knowledge is Z C n f q+1 =[ z C n f q+1 1 , z C n f q+1 2 , ⋯, z C n f q+1 K ] , .

[0103] Step S6, calculate knowledge through KL divergence and local models The output knowledge is The KL divergence loss term :

[0104]

[0105] in, p S =[ p S q 1 , p S q 2 , ⋯, p S q K ] According to the global model Output of knowledge Get the output probability of the server, each element It is expressed as:

[0106]

[0107] p f C n =[ p f C n q+1 1 , p f C n q+1 2 , ⋯, p f C n q+1 K ] Based on the local model Output of knowledge Get the client's output probability, each element of which It is expressed as:

[0108]

[0109] Where T is the temperature coefficient of client distillation.

[0110] Step S7, through the global model and local models The output eigenvalues ​​calculate the similarity matrix M and obtain the contrast loss term ,in, Indicates the average value.

[0111] Step S8: Contrastive loss term and KL divergence loss term pass , The two hyperparameters are linked together as the total loss term for client distillation :

[0112] .

[0113] Step S9: Distill the total loss item through the client , in the q+1th round of interaction between the client and the server, in the fth iteration of the local model, the local model Update and generate local model , specifically: ,in, is the model learning rate.

[0114] Step S10: Input the client's local data set into the local model In the above example, we calculate the cross entropy between the true label and the predicted result. , as a hard indicator supervision loss term, where , are the local models in the fth iteration respectively. The output results are compared with the true labels of the client's local dataset.

[0115] Step S11: The local model of the client's f-1th iteration The output feature values ​​are regarded as negative samples, and the global model The output value of is considered as a positive sample and the comparison supervision loss term is calculated , minimize the local model The similarity difference between the output value and the positive sample is maximized while maximizing the similarity difference between it and the negative sample.

[0116] ;

[0117] in, is the local model of the fth iteration The output result is, is the local model of the f-1th iteration The output result is, For the global model The output result is, To compare the temperature coefficient of the supervision loss term, Represents cosine similarity.

[0118] Step S12: and The two loss terms are , The two hyperparameters are linked together as the client model supervision total loss term :

[0119] .

[0120] Step S13: Through the loss term , in the q+1th round of interaction between the client and the server, the local model Update and generate a new round of local models ,

[0121] = ,in, is the model learning rate;

[0122] Step S14, repeat steps S6-S13 until f=E C , get the local model ,

[0123] Step S15: Repeat steps S2-S4 to aggregate the knowledge output by the local models of each client on the server to obtain knowledge , used to update the server-side global model parameters .

[0124] Step S16: Repeat steps S2-S15 until the number of communication rounds between the client and the server reaches the set value E, and the training is completed.

[0125] The present invention promotes the evolution of the server model into a server-side global model with higher accuracy and better generalization ability through the dual-focus distillation strategy. Through the joint training of knowledge distillation loss term and local supervision loss term, the local model can achieve higher global accuracy while maintaining the original data distribution advantage.

[0126] This experiment was conducted on a high-performance server equipped with an Intel Xeon Gold 6230R processor, two NVIDIA GeForce RTX 3090 graphics cards, and 128GB of memory. The experiment used three classic datasets: MNIST, CIFAR-10, and CIFAR-100, as well as an LCD dataset containing six types of industrial containers, including glass bottles, plastic bottles, cartons, aluminum cans, steel cans, and aerosol cans. We compared the performance of four federated learning methods, FedAvg, MOON, FedMD, and FedProto, with the CBDFL method in non-IID scenarios, and compared the performance of FedMD, FedProto, and CBDFL methods in model heterogeneity and non-IID scenarios. The experimental results show that the performance of the CBDFL method on the four datasets shows a significant advantage, as shown in Table 1.

[0127] Table 1 Comparison of experimental data

[0128]

[0129] Take the CBDFL framework in the CIFAR-10 dataset as an example, where: , , , , , , , , , , learning rate .

[0130] The detailed description of the embodiments of the present invention provided above is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Claims

1. A comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment, characterized in that The method comprises the following steps: Step S1: Construct a comparative dual-focus knowledge distillation federated learning model through 1 server and N clients to complete K-class classification tasks. Each client has a local dataset. ; Each client performs pre-training on the local data set to obtain an initialized local model; Step S2: The client and the server conduct federated learning training. In the qth round of communication between the client and the server, the knowledge output by the local model of each client is aggregated on the server to obtain knowledge , used to update the global model. In the i-th iteration, the global model The output knowledge is ; Step S3: Using the softmax function, based on the aggregated knowledge and global model Output of knowledge , respectively calculate the implicit probability and explicit probability matrices of the client and the implicit probability and explicit probability matrices of the server at the i-th iteration of the global model, and obtain the implicit loss term through KL divergence and explicit loss terms , using two hyperparameters , Implicit loss and explicit loss terms Combined as a dual-focus knowledge distillation loss term ; Step S4: In the i-th iteration, the loss term is distilled using the dual-focus knowledge For the global model Update and generate a new round of global model , until i=E S , and obtain the global model , global model The output knowledge is , where E S The number of iterations for the set server; Step S5: The server sends the global model and the knowledge it outputs Sent to each client, in the fth iteration, the local model The output knowledge is , ; Step S6: Calculate the global model using KL divergence Output knowledge With local model Output knowledge The KL divergence loss term ; Step S7: Based on the global model and local models The output eigenvalues ​​calculate the similarity matrix M and obtain the contrast loss term ,in, represents the average value; Step S8: Contrastive loss term and KL divergence loss term use , The two hyperparameters are linked together as the total loss term for client distillation ; Step S9: In the fth iteration, use the client to distill the total loss term For local models Update and generate local model ; Step S10: Input local data into the local model In the above example, the cross entropy between the true label and the predicted result is calculated as a hard indicator supervision loss term. ; Step S11: The local model of the f-1th iteration The output feature values ​​are regarded as negative samples, and the global model The output value is regarded as a positive sample and the comparative supervision loss term is calculated ; Step S12: Use the hard indicator supervision loss item and the comparison supervision loss ,use , The two hyperparameters are linked together as the client model supervision total loss term ; Step S13: In the fth iteration, the client model is used to supervise the total loss term , for the local model Update and generate a new round of local models , Step S14, repeat steps S6-S13 until f=E C , get the local model , where E C The number of iterations for the set client; Step S15: Repeat steps S2-S4 to aggregate the knowledge output by the local models of each client on the server to obtain knowledge , used to update the server-side global model ; Step S16: Repeat steps S2-S15 until the number of communication rounds between the client and the server reaches the set value E, and the training is completed.

2. The comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment according to claim 1, characterized in that: Step S3, using the softmax function, based on the aggregated knowledge and global model Output of knowledge , respectively calculate the implicit probability and explicit probability matrices of the client and the implicit probability and explicit probability matrices of the server at the i-th iteration of the global model, and obtain the implicit loss term through KL divergence and explicit loss terms , using two hyperparameters , Implicit loss and explicit loss terms Combined as a dual-focus knowledge distillation loss term , specifically: Based on aggregated knowledge Calculate the non-target probability of the client separately and target probability : ； in, is the dominant temperature coefficient, According to the global model Output of knowledge Calculate the non-target probability of the server at the i-th iteration of the global model respectively and target probability : ； Then we get the client's explicit probability matrix and the server’s explicit probability matrix at the i-th iteration of the global model ; Obvious loss term is obtained through KL divergence for: ； Define the implicit probability of the client , where each element is represented by: ； Define the implicit probability of the server at the i-th iteration of the global model , where each element It is expressed as: ； in, is the implicit temperature coefficient, Obtain the implicit loss term through KL divergence for: ； Then the dual-focus knowledge distillation loss term for: 。 3. The comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment according to claim 1, characterized in that: In step S4, in the i-th iteration, the loss term is distilled using the dual-focus knowledge For the global model Update and generate a new round of global model , specifically: ,in, is the model learning rate.

4. The comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment according to claim 1, characterized in that: Step S6: Calculate the global model using KL divergence Output knowledge With local model Output knowledge The KL divergence loss term , specifically: Define the output probability of the server : ； Among them, each element It is expressed as: ； Define the client output probability at the fth iteration of the local model : ； Among them, each element It is expressed as: ； Where T is the temperature coefficient of client distillation; Then the KL divergence loss term for: 。 5. The comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment according to claim 1, characterized in that: In step S8, the client distills the total loss item The expression is: 。 6. The comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment according to claim 1, characterized in that: In step S10, the hard indicator supervision loss item The expression is: ； in, , are the local models in the fth iteration respectively. The output results are compared with the true labels of the client's local dataset.

7. The comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment according to claim 1, characterized in that: In step S11, the comparison supervision loss term The expression is: ； in, is the local model of the fth iteration The output result is, is the local model of the f-1th iteration The output result is, For the global model The output result is, To compare the temperature coefficient of the supervision loss term, Represents cosine similarity.

8. The comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment according to claim 1, characterized in that: In step S12, the client model supervises the total loss term The expression is: 。 9. The comparative dual-focus knowledge distillation federated learning method for industrial heterogeneous equipment according to claim 1, characterized in that: In step S13, in the fth iteration, the client model is used to supervise the total loss term , for the local model Update and generate a new round of local models , specifically: = ,in, is the model learning rate.

Images