Distributed face classification method and system based on self-evolution end-to-end knowledge distillation

Abstract

The invention discloses a distributed face classification method and system based on self-evolution end-to-end knowledge distillation. The method aims at solving the problem that in a face image classification system based on centralized federated learning, the communication cost between a cloud server and a client is too high, and by means of a self-evolution end-to-end model knowledge distillation mechanism, the generalization of an edge end face classification model is improved. According to the method, a client pairing mechanism based on a context multi-arm machine algorithm is used, the optimal distillation pair capable of maximizing system efficiency is identified online, calculation and communication modes in the distillation pair are deeply analyzed, and an optimal scheduling decision is given. The method has the advantages of short training time, low communication cost and the like.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer technology, and relates to a distributed face classification method and system based on self-evolving end-to-end knowledge distillation. Background Art

[0002] Face image classification is one of the important branches in the field of computer vision. By training classification models, automatic recognition and accurate classification of faces can be achieved. Face classification models are usually deployed in edge devices to meet real-time requirements. In addition, due to data privacy considerations, face image data is required to be stored locally on the device, which limits the sharing of face data among different edge devices to improve model generalization. One possible approach is to perform distributed training of face classification models based on centralized federated learning. However, in centralized federated learning, the increasing size of the model hits the bottleneck of system communication and memory: (1) The transmission of model parameters between edge clients and cloud servers consumes limited bandwidth resources; (2) Heterogeneous edge devices with different memory capacities require customized models with different capabilities.

[0003] The emergence of knowledge distillation provides a solution, which supports knowledge transfer between heterogeneous models through integrated distillation. Instead of weighted averaging, it treats the predicted probability as a global consensus between network models and minimizes the KL divergence to transfer "dark knowledge" (decision boundary features embedded in soft labels) between clients, thereby supporting aggregation between models with different structures such as traditional convolutional neural networks and visual transformer models. Although federated knowledge distillation solves the heterogeneity challenge, it is still centralized: the client needs to transmit the predicted probability to the server, and the server calculates the weighted average to form a global consensus as distillation. Unlike using model parameters with transparent gradient directions, the opaque distillation process requires a large number of low-dimensional logical vectors to achieve high accuracy. In addition, since distillation is usually performed on public data without true labels and some models have limited representation capabilities, a unique challenge faced by federated knowledge distillation is that a few models cannot win consensus when most heterogeneous models misclassify, especially under non-IID perturbations, resulting in low fidelity. This becomes the main difficulty in achieving high accuracy in distributed training of heterogeneous face classification models. Summary of the invention

[0004] The purpose of the present invention is to provide a distributed face classification method and system based on self-evolving end-to-end knowledge distillation to address the communication and memory bottleneck problems encountered by the current face image classification system based on centralized federated learning.

[0005] According to a first aspect of the present specification, a distributed face classification method based on self-evolving end-to-end knowledge distillation is provided, the method comprising the following steps:

[0006] (1) Each client initializes a local face classification model;

[0007] (2) Each client performs local face classification model training;

[0008] (3) The cloud server enumerates all the combinations of pairings between clients, calculates the feature vectors of each pairing, calculates the prediction efficiency value of each combination based on the contextual multi-armed bandit algorithm, selects the combination with the highest prediction efficiency value, and completes the pairing based on the selected combinations;

[0009] (4) The client performs end-to-end knowledge distillation based on the pairings, calculates the true performance values ​​of all pairings and uploads them to the cloud server for parameter update;

[0010] (5) Repeat steps (2) to (4) until the local face classification model of each client is determined to be converged;

[0011] (6) Each client completes the face image classification task based on the trained local face classification model.

[0012] Furthermore, each pairing feature vector includes: a KL divergence value calculated between the two clients using public data, a time difference between the local running times of the two clients, and a duration of communication between the two clients.

[0013] Furthermore, the end-to-end knowledge distillation includes model forward propagation, model back propagation, prediction probability transmission, global consensus calculation and global consensus transmission.

[0014] Furthermore, for paired clients A and B, the end-to-end knowledge distillation is as follows:

[0015] (i) After completing the training of the local face classification model, clients A and B respectively input the public data into the local face classification model to obtain the corresponding prediction probability;

[0016] (ii) Client A sends the obtained predicted probability to Client B;

[0017] (iii) Client B calculates the average of the predicted probabilities of clients A and B as the global consensus;

[0018] (iv) Client B sends the global consensus to Client A;

[0019] (v) Clients A and B use the global consensus as soft labels and perform back propagation respectively to complete a round of end-to-end knowledge distillation;

[0020] (vi) Repeat steps (i) to (v) to complete the entire knowledge distillation process.

[0021] Furthermore, in step (5), the method for judging whether the local face classification model has converged is that the fluctuation variance of the training loss function value of the model within 10 consecutive rounds of training is less than a preset threshold.

[0022] Furthermore, in step (5), the local face classification model of the client stops local training after convergence, exits pairing and does not participate in end-to-end knowledge distillation.

[0023] According to a second aspect of this specification, a distributed face classification system based on self-evolving end-to-end knowledge distillation is provided, the system is used to implement the distributed face classification method as described in the first aspect, and the system includes:

[0024] Model initialization module, used by each client to initialize the local face classification model;

[0025] Model training module, used for local face classification model training on each client;

[0026] Client pairing module: The cloud server enumerates all the combinations of client pairings, calculates the feature vectors of each pairing, calculates the predicted performance value of each combination based on the contextual multi-armed bandit algorithm, selects the combination with the highest predicted performance value, and completes pairing based on the selected combinations;

[0027] Client knowledge distillation module: The client performs end-to-end knowledge distillation based on pairings, calculates the true performance values ​​of all pairings and uploads them to the cloud server for parameter update;

[0028] The face classification module is used to complete the face image classification task based on the trained local model after the local face classification model of each client converges.

[0029] According to a third aspect of this specification, an electronic device is provided, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the distributed face classification method as described in the first aspect.

[0030] According to a fourth aspect of the present specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the distributed face classification method as described in the first aspect is implemented.

[0031] According to a fifth aspect of the present specification, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the distributed face classification method as described in the first aspect.

[0032] The beneficial effects of the present invention are as follows: the present invention solves the problem of high communication cost between cloud servers and clients in the face image classification system based on centralized federated learning, and improves the generalization of the edge face classification model by means of a self-evolving end-to-end model knowledge distillation mechanism. The present invention uses a client pairing mechanism based on a contextual multi-armed bandit algorithm to online identify the optimal distillation pair that can maximize the system performance, and deeply analyzes the calculation and communication mode within the distillation pair to give the optimal scheduling decision. The present invention has the advantages of short training time and low communication cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 The flowchart of a distributed face classification method based on self-evolving end-to-end knowledge distillation is shown as an exemplary embodiment;

[0035] Figure 2 A flowchart of the end-to-end knowledge distillation stage in the method of the present invention is shown as an exemplary embodiment;

[0036] Figure 3 FIG. 1 is a diagram showing the acceleration effect of the method of the present invention relative to the benchmark method, shown in an exemplary embodiment;

[0037] Figure 4 The figure is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0038] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0039] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0040] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0041] The present invention provides a distributed face classification method based on self-evolving end-to-end knowledge distillation, such as Figure 1 As shown, the method comprises the following steps:

[0042] (1) Set up the following system architecture: A face classification system architecture consisting of a cloud server and N edge client devices. Each client n initializes the local face classification model parameters as Specifically, the initialized local model can be a convolutional neural network or a visual transformer model.

[0043] (2) Each client n performs several rounds of local model training. Each client n has a private dataset. Perform gradient descent on is a data sample, is the label space. The calculation formula of the gradient descent process is as follows:

[0044]

[0045] in is the learning rate for local training, is the model gradient, is the cross entropy loss function, It is a dataset Specifically, the face image data used by each client for local training is not independent and identically distributed.

[0046] (3) The cloud server enumerates all possible combinations of N clients that are paired with each other to form a combination set P.

[0047] For example: suppose there are 4 clients n1, n2, n3, and n4, then there are 3 combinations of pairing these 4 clients. Combination 1: n1 and n2 are paired, and n3 and n4 are paired; Combination 2: n1 and n3 are paired, and n2 and n4 are paired; Combination 3: n1 and n4 are paired, and n2 and n3 are paired. These three combinations constitute set P.

[0048] (4) For any combination p in the set P, the cloud server calculates the feature vectors of all pairs in the set at the tth (t = 0, 1, …, T-1)th round, where T is the total number of training rounds. The feature vector of pair k is .

[0049] is the KL divergence value calculated using public data in round t between two clients i and j in pair k, and is calculated as follows:

[0050]

[0051] in represents the KL divergence loss function; Represents the softmax function, which is used to normalize the model prediction probability; Represents the forward propagation of the model, which is used to calculate the prediction probability; Represents the forward propagation of the local model of client i and j respectively; Represents public data.

[0052] It represents the time difference between the local running time of two clients i and j in pair k in round t, which is calculated as follows:

[0053]

[0054] in Indicates taking the absolute value, Represents the local running time of client i, which is calculated as follows:

[0055]

[0056] in Indicates that client i has its private dataset The time it takes to perform local training on Indicates that client i is in the public data Perform forward propagation on (obtain model prediction probability, used to calculate ) The local running time of client j The calculation method is the same as The calculation method of .

[0057] represents the duration of communication between two clients i and j in pair k in round t. Assuming that client i transmits data to client j, The calculation method is as follows:

[0058]

[0059] in is the data transmission rate between clients i and j.

[0060] (5) The cloud server obtains the feature vectors of each pair k based on step (4) , using the contextual multi-armed bandit algorithm to calculate the prediction performance value of each combination p in the set P The calculation process is as follows:

[0061] For each pair k in combination p, first initialize the matrix , is the identity matrix of dimension 3×3, Used to estimate the parameters for each pairing in the multi-armed bandit algorithm.

[0062] Next, calculate the estimated mean reward for pair k in round t :

[0063]

[0064] in Represents the parameters updated according to the true performance value of the t-1th round, initialized to a zero vector.

[0065] Then, calculate the prediction performance value of pair k in round t :

[0066]

[0067] Calculate the prediction performance value of each combination p in the set P :

[0068] =

[0069] The prediction performance value for each combination p Sort and select the pair combination p* with the highest prediction efficiency value:

[0070]

[0071] Finally, pairwise pairing is completed based on the screened pairing combination p*.

[0072] (6) The client performs end-to-end knowledge distillation based on the pairing, such as Figure 2 As shown, the process of the end-to-end knowledge distillation stage in the present invention is introduced, taking paired clients A and B as an example, including the following steps:

[0073] (i) After completing local model training, clients A and B input public data into their local models to obtain the corresponding prediction probabilities. , .in Represents the softmax function, which is used to normalize the model prediction probability; Represents the forward propagation of the model, which is used to calculate the prediction probability; Represents public data.

[0074] (ii) Client A sends the obtained predicted probability to Client B.

[0075] (iii) Client B calculates the average of the predicted probabilities of A and B as the global consensus G:

[0076]

[0077] (iv) Client B sends the global consensus G to Client A.

[0078] (v) Clients A and B use the global consensus G as a soft label and perform back propagation respectively:

[0079]

[0080]

[0081] in is the learning rate for distillation, are a batch in the private datasets of clients A and B respectively.

[0082] (vi) Repeat steps (i) to (v) for a total of R rounds to complete the entire distillation process. In this embodiment, R is 10.

[0083] (7) Calculate the true performance value of all pairs And upload to the cloud server:

[0084]

[0085] in , is a scaling factor. In this embodiment Take 0.3.

[0086] The cloud server is based on the actual performance value of each pair , update the parameter values ​​in the next round of the multi-armed bandit algorithm:

[0087] ,

[0088] (8) Repeat steps (2) to (7) until the local model of each client is determined to be converged. The following is a method for determining model convergence but is not limited to this: for the local model of client n, the fluctuation variance of its local training loss function value within 10 consecutive rounds of training is less than the preset threshold value of 0.0001, and the formula is as follows:

[0089]

[0090] in Indicates the stop time. represents the cross entropy loss function, μ represents the average value of the loss function for 10 rounds of training, Represents the local model parameters of client n at round t.

[0091] After a client's local model converges, it stops local training, exits pairing, and does not participate in end-to-end knowledge distillation. Client models that have not reached convergence continue training and knowledge distillation.

[0092] (9) Each client completes the facial image classification task based on the trained local model, including age classification, posture classification, and occupation classification of the person in the target image.

[0093] like Figure 3 As shown in the figure, the acceleration effect of the method of the present invention relative to the benchmark method is shown. Here, the training effect on the face image dataset VGGFace is shown. VGGFace contains a total of 2.6 million images. In order to shorten the training time, 100,000 images are extracted from it for testing. It can be seen that the method proposed in the present invention (denoted as SD-FedDF) has significantly shortened training time compared with the benchmark methods FedMD, FedDF, and Selective-FD (Shao J, Wu F, Zhang J. Selective knowledge sharing for privacy-preserving federated distillation without a good teacher [J]. Nature Communications, 2024, 15 (1). DOI: 10.1038 / s41467-023-44383-9.), and the average accuracy of the model is higher. The time for SD-FedDF to reach 60% accuracy is about 2.8 hours, 3.4 hours, and 12.0 hours earlier than FedDF, Selective-FD, and FedMD, respectively. The speedup of SD-FedDF is attributed to two aspects: 1) the performance value prediction method designed by the present invention specifically minimizes the difference in execution time within a pairing, so that fast and slow devices are consistent with their peers most of the time; 2) SD-FedDF can implement asynchronous updates and client pairing, which can achieve higher throughput within a fixed time range.

[0094] On the other hand, an embodiment of the present invention provides a distributed face classification system based on self-evolving end-to-end knowledge distillation, which is used to implement the above-mentioned distributed face classification method based on self-evolving end-to-end knowledge distillation, and the system includes:

[0095] Model initialization module, used by each client to initialize the local face classification model;

[0096] Model training module, used for local face classification model training on each client;

[0097] Client pairing module: The cloud server enumerates all the combinations of client pairings, calculates the feature vectors of each pairing, calculates the predicted performance value of each combination based on the contextual multi-armed bandit algorithm, selects the combination with the highest predicted performance value, and completes pairing based on the selected combinations;

[0098] Client knowledge distillation module: The client performs end-to-end knowledge distillation based on pairings, calculates the true performance values ​​of all pairings and uploads them to the cloud server for parameter update;

[0099] The face classification module is used to complete the face image classification task based on the trained local model after the local face classification model of each client converges.

[0100] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0101] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement it without creative work.

[0102] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned distributed face classification method based on self-evolving end-to-end knowledge distillation. Figure 4 As shown, a hardware structure diagram of a distributed face classification method based on self-evolving end-to-end knowledge distillation provided by an embodiment of the present invention is provided in any device with data processing capabilities, except Figure 4 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0103] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the distributed face classification method based on self-evolving end-to-end knowledge distillation as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0104] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0105] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

[0106] The above is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above disclosed methods and technical contents without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A distributed face classification method based on self-evolving end-to-end knowledge distillation, characterized in that: The following steps are involved: (1) Each client initializes a local face classification model; (2) Each client performs local face classification model training; (3) The cloud server enumerates all the combinations of pairings between clients, calculates the feature vectors of each pairing, calculates the prediction efficiency value of each combination based on the contextual multi-armed bandit algorithm, selects the combination with the highest prediction efficiency value, and completes the pairing based on the selected combinations; (4) The client performs end-to-end knowledge distillation based on the pairings, calculates the true performance values ​​of all pairings and uploads them to the cloud server for parameter update; (5) Repeat steps (2) to (4) until the local face classification model of each client is determined to be converged; (6) Each client completes the face image classification task based on the trained local face classification model.

2. The distributed face classification method based on self-evolving end-to-end knowledge distillation according to claim 1 is characterized in that: Each pairing feature vector includes: the KL divergence value calculated between the two clients using public data, the time difference between the local running times of the two clients, and the duration of the two clients communicating with each other.

3. The distributed face classification method based on self-evolving end-to-end knowledge distillation according to claim 1 is characterized in that: The end-to-end knowledge distillation includes model forward propagation, model back propagation, prediction probability transmission, global consensus calculation and global consensus transmission.

4. The distributed face classification method based on self-evolving end-to-end knowledge distillation according to claim 3 is characterized in that: For paired clients A and B, end-to-end knowledge distillation is as follows: (i) After completing the training of the local face classification model, clients A and B respectively input the public data into the local face classification model to obtain the corresponding prediction probability; (ii) Client A sends the obtained predicted probability to Client B; (iii) Client B calculates the average of the predicted probabilities of clients A and B as the global consensus; (iv) Client B sends the global consensus to Client A; (v) Clients A and B use the global consensus as soft labels and perform back-propagation respectively to complete a round of end-to-end knowledge distillation; (vi) Repeat steps (i) to (v) to complete the entire knowledge distillation process.

5. The distributed face classification method based on self-evolving end-to-end knowledge distillation according to claim 1 is characterized in that: In step (5), the method for judging the convergence of the local face classification model is that the fluctuation variance of the training loss function value of the model within 10 consecutive rounds of training is less than a preset threshold.

6. The distributed face classification method based on self-evolving end-to-end knowledge distillation according to claim 1 is characterized in that: In step (5), the client's local face classification model stops local training after convergence, exits pairing, and does not participate in end-to-end knowledge distillation.

7. A distributed face classification system based on self-evolving end-to-end knowledge distillation, characterized in that: For implementing the distributed face classification method according to any one of claims 1 to 6, the system comprises: Model initialization module, used by each client to initialize the local face classification model; Model training module, used for local face classification model training on each client; Client pairing module: The cloud server enumerates all the combinations of client pairings, calculates the feature vectors of each pairing, calculates the predicted performance value of each combination based on the contextual multi-armed bandit algorithm, selects the combination with the highest predicted performance value, and completes pairing based on the selected combinations; Client knowledge distillation module: The client performs end-to-end knowledge distillation based on pairings, calculates the true performance values ​​of all pairings and uploads them to the cloud server for parameter update; The face classification module is used to complete the face image classification task based on the trained local model after the local face classification model of each client converges.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the distributed face classification method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the distributed face classification method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the distributed face classification method as described in any one of claims 1 to 6 is implemented.

Images