Distributed Face Classification Method and System Based on Self-Evolving End-to-End Knowledge Distillation

Abstract

The present invention discloses a distributed face classification method and system based on self-evolving end-to-end knowledge distillation. The present invention aims to solve the problem of excessively high communication costs between cloud servers and clients in face image classification systems based on centralized federated learning, and improve the generalization of edge-end face classification models with the help of a self-evolving end-to-end model knowledge distillation mechanism. The present invention uses a client pairing mechanism based on a contextual multi-armed bandit algorithm to online identify the optimal distillation pair that can maximize system efficiency, and deeply analyzes the calculation and communication patterns within the distillation pair to provide the optimal scheduling decision. The present invention has the advantages of short training time and low communication cost.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer technology, and relates to a distributed face classification method and system based on self-evolving end-to-end knowledge distillation. Background Art

[0002] Face image classification is one of the important branches in the field of computer vision. By training classification models, automatic recognition and accurate classification of faces can be achieved. Face classification models are usually deployed in edge devices to meet real-time requirements. In addition, due to data privacy concerns, face image data is required to be stored locally on the device, which limits the sharing of face data between different edge devices to improve model generalization. One possible approach is to perform distributed training of face classification models based on centralized federated learning. However, in centralized federated learning, the increasing size of the model reaches the bottleneck of system communication and memory: (1) The transmission of model parameters between edge clients and cloud servers consumes limited bandwidth resources; (2) Heterogeneous edge devices with different memory capacities require customized models with different capabilities.

[0003] The emergence of knowledge distillation offers a solution, enabling knowledge transfer between heterogeneous models through ensemble distillation. Instead of weighted averaging, this approach treats predicted probabilities as a global consensus between network models and minimizes KL divergence, transferring "dark knowledge" (decision boundary features embedded in soft labels) between clients. This enables convergence between models with different structures, such as traditional convolutional neural networks and visual transformer models. While federated knowledge distillation addresses the heterogeneity challenge, it remains centralized: clients must transmit predicted probabilities to a server, which computes a weighted average to form a global consensus for distillation. Unlike using model parameters with transparent gradient directions, the opaque distillation process requires a large number of low-dimensional logical vectors to achieve high accuracy. Furthermore, since distillation is often performed on public data without ground-truth labels and some models have limited representational power, a unique challenge faced by federated knowledge distillation is that a minority of models cannot achieve consensus when the majority of heterogeneous models misclassify, especially under non-IID perturbations, resulting in low fidelity. This presents a major challenge in achieving high accuracy in distributed training of heterogeneous face classification models. Summary of the Invention

[0004] The purpose of the present invention is to provide a distributed face classification method and system based on self-evolving end-to-end knowledge distillation to address the communication and memory bottleneck problems encountered by the current face image classification system based on centralized federated learning.

[0005] According to a first aspect of this specification, a distributed face classification method based on self-evolving end-to-end knowledge distillation is provided, the method comprising the following steps:

[0006] (1) Each client initializes the local face classification model;

[0007] (2) Each client performs local face classification model training;

[0008] (3) The cloud server enumerates all possible pairing combinations of the client, calculates the feature vectors of each pairing, calculates the predicted effectiveness value of each combination based on the contextual multi-armed bandit algorithm, selects the combination with the highest predicted effectiveness value, and completes the pairing based on the selected combinations;

[0009] (4) The client performs end-to-end knowledge distillation based on the pairings, calculates the true effectiveness values of all pairings and uploads them to the cloud server for parameter update;

[0010] (5) Repeat steps (2) to (4) until the local face classification model of each client is determined to be converged;

[0011] (6) Each client completes the face image classification task based on the trained local face classification model.

[0012] Furthermore, each pairing feature vector includes: a KL divergence value calculated between the two clients using public data, a time difference between the local running times of the two clients, and a duration of communication between the two clients.

[0013] Furthermore, the end-to-end knowledge distillation includes model forward propagation, model back propagation, prediction probability transmission, global consensus calculation and global consensus transmission.

[0014] Furthermore, for paired clients A and B, the end-to-end knowledge distillation is as follows:

[0015] (i) After completing the training of the local face classification model, clients A and B input the public data into the local face classification model to obtain the corresponding prediction probability;

[0016] (ii) Client A sends the obtained predicted probability to Client B;

[0017] (iii) Client B calculates the average of the predicted probabilities of clients A and B as the global consensus;

[0018] (iv) Client B sends the global consensus to Client A;

[0019] (v) Clients A and B use the global consensus as soft labels and perform backpropagation separately to complete a round of end-to-end knowledge distillation;

[0020] (vi) Repeat steps (i)-(v) to complete the entire knowledge distillation process.

[0021] Furthermore, in step (5), the method for judging whether the local face classification model has converged is that the fluctuation variance of the model's training loss function value within 10 consecutive rounds of training is less than a preset threshold.

[0022] Furthermore, in step (5), the local face classification model of the client stops local training after convergence, exits pairing and does not participate in end-to-end knowledge distillation.

[0023] According to a second aspect of this specification, a distributed face classification system based on self-evolving end-to-end knowledge distillation is provided. The system is used to implement the distributed face classification method according to the first aspect, and the system includes:

[0024] Model initialization module, used by each client to initialize the local face classification model;

[0025] Model training module, used for local face classification model training on each client;

[0026] Client pairing module: The cloud server enumerates all possible pairing combinations of clients, calculates the feature vectors of each pairing, calculates the predicted performance value of each combination based on the contextual multi-armed bandit algorithm, selects the combination with the highest predicted performance value, and completes pairing based on the selected combinations;

[0027] Client knowledge distillation module: The client performs end-to-end knowledge distillation based on pairings, calculates the true performance values of all pairings, and uploads them to the cloud server for parameter updates;

[0028] The face classification module is used to complete the face image classification task based on the trained local model after the local face classification model of each client converges.

[0029] According to a third aspect of this specification, an electronic device is provided, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the distributed face classification method as described in the first aspect.

[0030] According to a fourth aspect of this specification, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the distributed face classification method as described in the first aspect is implemented.

[0031] According to a fifth aspect of this specification, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the distributed face classification method as described in the first aspect.

[0032] The present invention addresses the high communication costs between cloud servers and clients in centralized federated learning-based facial image classification systems. By leveraging a self-evolving, end-to-end model knowledge distillation mechanism, it improves the generalization of edge-side face classification models. This invention utilizes a client-pairing mechanism based on a contextual multi-armed bandit algorithm to online identify optimal distillation pairs that maximize system performance. It also conducts in-depth analysis of the computation and communication patterns within these distillation pairs to determine optimal scheduling decisions. This invention offers advantages such as short training time and low communication costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 This is a flow chart of a distributed face classification method based on self-evolving end-to-end knowledge distillation, shown as an exemplary embodiment;

[0035] Figure 2 FIG1 is a flow chart of the end-to-end knowledge distillation stage in the method of the present invention, showing an exemplary embodiment;

[0036] Figure 3 FIG1 is a diagram showing the acceleration effect of the method of the present invention relative to the benchmark method according to an exemplary embodiment;

[0037] Figure 4 The figure is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0038] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0039] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0040] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0041] The present invention provides a distributed face classification method based on self-evolving end-to-end knowledge distillation, such as Figure 1 As shown, the method includes the following steps:

[0042] (1) Set up the following system architecture: a face classification system architecture consisting of a cloud server and N edge client devices. Each client n initializes the local face classification model parameters as Specifically, the initialized local model can be a convolutional neural network or a visual transformer model.

[0043] (2) Each client n performs several rounds of local model training. Each client n has its own private dataset. Perform gradient descent on is a data sample, is the label space. The calculation formula of the gradient descent process is as follows:

[0044]

[0045] in is the learning rate for local training, is the model gradient, is the cross entropy loss function, It is a dataset Specifically, the face image data used by each client for local training is not independent and identically distributed.

[0046] (3) The cloud server enumerates all possible combinations of N clients that are paired with each other to form a combination set P.

[0047] For example, suppose there are four clients, n1, n2, n3, and n4. There are three possible combinations of pairing these four clients. Combination 1: n1 and n2, n3 and n4; Combination 2: n1 and n3, n2 and n4; Combination 3: n1 and n4, n2 and n3. These three combinations form the set P.

[0048] (4) For any combination p in the set P, the cloud server calculates the feature vectors of all pairs in the tth round (t = 0, 1, …, T-1), where T is the total number of training rounds, and the feature vector of pair k is .

[0049] is the KL divergence between two clients i and j in pair k calculated using public data in round t, and is calculated as follows:

[0050]

[0051] in represents the KL divergence loss function; Represents the softmax function, which is used to normalize the model prediction probability; Represents the forward propagation of the model, which is used to calculate the predicted probability; Represents the forward propagation of the local model of client i and j respectively; Represents public data.

[0052] It represents the time difference between the local execution times of two clients i and j in pair k in round t, which is calculated as follows:

[0053]

[0054] in Indicates taking the absolute value, Represents the local running time of client i, which is calculated as follows:

[0055]

[0056] in Indicates that client i has its private dataset The time it takes to train locally on Indicates that client i is in public data Perform forward propagation on (obtain model prediction probability, used to calculate The local running time of client j The calculation method is the same as Calculation method.

[0057] represents the duration of communication between two clients i and j in pair k in round t. Assuming that client i transmits data to client j, is calculated as follows:

[0058]

[0059] in is the data transmission rate between clients i and j.

[0060] (5) The cloud server obtains the feature vectors of each pair k based on step (4) , use the contextual multi-armed bandit algorithm to calculate the prediction performance value of each combination p in the set P The calculation process is as follows:

[0061] For each pair k in combination p, first initialize the matrix , is the identity matrix of dimension 3×3, Used to estimate the parameters of each pairing in the multi-armed bandit algorithm.

[0062] Next, calculate the estimated mean reward of pair k in round t :

[0063]

[0064] in Represents the parameters updated according to the true performance value of the t-1th round, initialized to the zero vector.

[0065] Then, calculate the prediction performance value of pair k in round t :

[0066]

[0067] Calculate the prediction performance value of each combination p in the set P :

[0068] =

[0069] The predictive performance value for each combination p Sort and select the pair combination p* with the highest prediction efficiency value:

[0070]

[0071] Finally, pairwise pairing is completed based on the screened pairing combination p*.

[0072] (6) The client performs end-to-end knowledge distillation based on the pairing, such as Figure 2 As shown in FIG, the process of the end-to-end knowledge distillation stage in the present invention is introduced, taking paired clients A and B as an example, including the following steps:

[0073] (i) After completing local model training, clients A and B input public data into their local models to obtain the corresponding prediction probabilities. 、 .in Represents the softmax function, which is used to normalize the model prediction probability; Represents the forward propagation of the model, which is used to calculate the predicted probability; Represents public data.

[0074] (ii) Client A sends the obtained predicted probability to Client B.

[0075] (iii) Client B calculates the average of the predicted probabilities of A and B as the global consensus G:

[0076]

[0077] (iv) Client B sends the global consensus G to Client A.

[0078] (v) Clients A and B use the global consensus G as a soft label and perform backpropagation respectively:

[0079]

[0080]

[0081] in is the learning rate for distillation, A batch in the private dataset of client A and B respectively.

[0082] (vi) Repeat steps (i) to (v) for a total of R rounds to complete the entire distillation process. In this embodiment, R is 10.

[0083] (7) Calculate the true performance value of all pairs And upload to the cloud server:

[0084]

[0085] in , is a scaling factor, in this embodiment Take 0.3.

[0086] The cloud server is based on the actual performance value of each pair , update the parameter values in the next round of the multi-armed bandit algorithm:

[0087] ,

[0088] (8) Repeat steps (2) to (7) until the local model of each client is determined to have converged. The following is a method for determining model convergence, but is not limited to this: for the local model of client n, the fluctuation variance of its local training loss function value within 10 consecutive rounds of training is less than a preset threshold of 0.0001, and the formula is as follows:

[0089]

[0090] in Indicates the stop time. represents the cross entropy loss function, μ represents the average value of the loss function for 10 rounds of training, Represents the local model parameters of client n in round t.

[0091] After a client's local model converges, local training stops, the model exits pairing, and does not participate in end-to-end knowledge distillation. Client models that have not reached convergence continue training and knowledge distillation.

[0092] (9) Each client completes the facial image classification task based on the trained local model, including age classification, posture classification, and occupation classification of the person in the target image.

[0093] like Figure 3 As shown in the figure, the acceleration effect of the proposed method compared with the baseline method is shown. Here, the training effect on the face image dataset VGGFace is shown. VGGFace contains a total of 2.6 million images. In order to shorten the training time, 100,000 images are extracted from it for testing. It can be seen that the proposed method (denoted as SD-FedDF) has significantly shortened training time compared with the baseline methods FedMD, FedDF, and Selective-FD (Shao J, Wu F, Zhang J. Selective knowledge sharing for privacy-preserving federated distillation without a good teacher[J]. Nature Communications, 2024, 15(1). DOI:10.1038 / s41467-023-44383-9.), and the average accuracy of the model is higher. The time for SD-FedDF to reach 60% accuracy is approximately 2.8 hours, 3.4 hours, and 12.0 hours earlier than FedDF, Selective-FD, and FedMD, respectively. The speedup of SD-FedDF is attributed to two aspects: 1) the performance value prediction method designed in this paper specifically minimizes the difference in execution time within a pairing, so that fast and slow devices are consistent with their peers most of the time; 2) SD-FedDF can implement asynchronous updates and client pairing, which can achieve higher throughput within a fixed time range.

[0094] On the other hand, an embodiment of the present invention provides a distributed face classification system based on self-evolving end-to-end knowledge distillation, which is used to implement the above-mentioned distributed face classification method based on self-evolving end-to-end knowledge distillation. The system includes:

[0095] Model initialization module, used by each client to initialize the local face classification model;

[0096] Model training module, used for local face classification model training on each client;

[0097] Client pairing module: The cloud server enumerates all possible pairing combinations of clients, calculates the feature vectors of each pairing, calculates the predicted performance value of each combination based on the contextual multi-armed bandit algorithm, selects the combination with the highest predicted performance value, and completes pairing based on the selected combinations;

[0098] Client knowledge distillation module: The client performs end-to-end knowledge distillation based on pairings, calculates the true performance values of all pairings, and uploads them to the cloud server for parameter updates;

[0099] The face classification module is used to complete the face image classification task based on the trained local model after the local face classification model of each client converges.

[0100] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0101] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0102] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned distributed face classification method based on self-evolving end-to-end knowledge distillation. Figure 4 As shown in FIG, a hardware structure diagram of a distributed face classification method based on self-evolving end-to-end knowledge distillation provided by an embodiment of the present invention is provided on any device with data processing capabilities, except Figure 4 In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0103] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the distributed face classification method based on self-evolving end-to-end knowledge distillation as described above. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0104] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only.

[0105] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

[0106] The above description is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can use the above disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present invention without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment with equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A distributed face classification method based on self-evolving end-to-end knowledge distillation, characterized by: The following steps are involved: (1) Each client initializes the local face classification model; (2) Each client performs local face classification model training; (3) The cloud server enumerates all possible pairing combinations of the client, calculates the feature vectors of each pairing, calculates the predicted effectiveness value of each combination based on the contextual multi-armed bandit algorithm, selects the combination with the highest predicted effectiveness value, and completes the pairing based on the selected combinations; (4) The client performs end-to-end knowledge distillation based on the pairing, calculates the true effectiveness values of all pairings and uploads them to the cloud server for parameter update; the end-to-end knowledge distillation includes model forward propagation, model backpropagation, prediction probability transmission, global consensus calculation and global consensus transmission. For the paired clients A and B, the end-to-end knowledge distillation is specifically as follows: (i) After completing the training of the local face classification model, clients A and B input the public data into the local face classification model to obtain the corresponding prediction probability; (ii) Client A sends the obtained predicted probability to Client B; (iii) Client B calculates the average of the predicted probabilities of clients A and B as the global consensus; (iv) Client B sends the global consensus to Client A; (v) Clients A and B use the global consensus as soft labels and perform backpropagation separately to complete a round of end-to-end knowledge distillation; (vi) Repeat steps (i) to (v) to complete the entire knowledge distillation process; (5) Repeat steps (2) to (4) until the local face classification model of each client is determined to be converged; (6) Each client completes the face image classification task based on the trained local face classification model.

2. The distributed face classification method based on self-evolving end-to-end knowledge distillation according to claim 1 is characterized in that: Each pairing feature vector includes: the KL divergence value calculated between the two clients using public data, the time difference between the two clients' local running times, and the duration of the two clients' communication with each other.

3. The distributed face classification method based on self-evolving end-to-end knowledge distillation according to claim 1 is characterized in that In step (5), the method for judging the convergence of the local face classification model is that the fluctuation variance of the model's training loss function value within 10 consecutive rounds of training is less than a preset threshold.

4. The distributed face classification method based on self-evolving end-to-end knowledge distillation according to claim 1 is characterized in that: In step (5), the client's local face classification model stops local training after convergence, exits pairing, and does not participate in end-to-end knowledge distillation.

5. A distributed face classification system based on self-evolving end-to-end knowledge distillation, characterized by: For implementing the distributed face classification method according to any one of claims 1 to 4, the system comprises: Model initialization module, used by each client to initialize the local face classification model; Model training module, used for local face classification model training on each client; Client pairing module: The cloud server enumerates all possible pairing combinations of clients, calculates the feature vectors of each pairing, calculates the predicted performance value of each combination based on the contextual multi-armed bandit algorithm, selects the combination with the highest predicted performance value, and completes pairing based on the selected combinations; Client knowledge distillation module: The client performs end-to-end knowledge distillation based on pairings, calculates the true performance values of all pairings, and uploads them to the cloud server for parameter updates; The face classification module is used to complete the face image classification task based on the trained local model after the local face classification model of each client converges.

6. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the distributed face classification method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the distributed face classification method according to any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the distributed face classification method according to any one of claims 1 to 4 is implemented.

Images