A facial recognition method and device based on personalized federated learning

Abstract

The present invention discloses a facial recognition method and device based on personalized federated learning, which obtains the facial image data of the user, inputs the facial image data into the optimal facial recognition model, and obtains the facial recognition result; the personalized model performs cosine similarity calculation on the uploaded local model parameters, updates its own global model parameters, and sends the updated global model parameters to the corresponding client, so that when the client is grouped, the client with extremely abnormal data distribution is isolated, and the client with similar data distribution will cooperate. The method has a good defensive effect on malicious attackers, uses a personalized mechanism for training, gradually separates malicious clients from normal clients, aggregates different global models, reduces the impact on each other, and realizes facial recognition.

Description

Technical Field

[0001] The present invention belongs to the field of federated learning, and in particular relates to a facial recognition method and device based on personalized federated learning. Background Art

[0002] Nowadays, deep learning has received great attention in academia and industry. As the performance of deep learning has been greatly improved compared with traditional algorithms, deep learning has been widely used in various fields, such as machine translation, image recognition, unmanned driving, natural language processing, etc. However, with the rise of the Internet of Things and edge computing, big data is often not confined to a single whole, but distributed in many aspects. How to safely and effectively update and share models in multiple locations is a new challenge faced by various computing methods. Due to concerns about data privacy and security, data owners cannot directly share data to jointly train deep learning models. People began to seek a method to solve the problem of data silos and user privacy. Federated learning has emerged as a highly potential solution. Federated machine learning is a machine learning framework that can effectively help multiple institutions use data and conduct machine learning modeling while meeting the requirements of user privacy protection, data security and government regulations. As a distributed machine learning paradigm, federated learning can effectively solve the problem of data silos, allowing participants to jointly model without sharing data, and can technically break data silos. Its main innovation is that it provides a distributed machine learning framework with privacy protection features, and can collaborate with thousands of participants in a distributed manner to iteratively train a specific machine learning model. Each participant does not share data, which greatly protects privacy and security, and the training effect is better than centralized learning.

[0003] At present, the application areas of federated learning are becoming more and more extensive. In the federated learning architecture, a key challenge is the heterogeneity of data distribution among different parties. In many practical applications, data may be distributed in different ways among parties, which will reduce the performance of federated learning. When each party updates its local model, its local goal may be far from the global goal. Therefore, the average global model is far from the global optimum. There are some studies that try to solve the non-IID problem in the local training stage. FedProx directly limits local updates by "2-norm distance", while SCAFFOLD corrects local updates by variance reduction. However, these methods cannot achieve good performance on image datasets with deep learning models, which may be as bad as FedAvg. Another key challenge is that due to the non-sharing of data between clients, when performing traditional federated training, each client trains a local model based on local data separately, which provides a vulnerable attack surface for attackers, and the local training process is likely to be attacked and destroyed. Especially in the case of thousands of clients, it is almost impossible to accurately find the attacker. In addition, the existing federated learning framework cannot judge the correctness of the local learning model, and the client can submit malicious models at will, such as models containing backdoor functions. It is difficult for people to distinguish between a model attacked by a backdoor and a real model trained based on local data. This greatly damages the interests of normal clients. In the face of Non-IID and malicious attacks, how to better ensure the security and performance of federated learning has become a focus of attention. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and provide a facial recognition method and device based on personalized federated learning.

[0005] The objective of the present invention is achieved through the following technical solution: a facial recognition method based on personalized federated learning is provided, comprising the following steps:

[0006] (1) Obtaining user's facial image data;

[0007] (2) inputting the facial image data into an optimal facial recognition model to obtain a facial recognition result; wherein the generation of the optimal facial recognition model includes the following sub-steps:

[0008] a) Initialize the federated learning training environment;

[0009] b) The server sets up a corresponding personalized model for each client in the cloud. Each personalized model sends its own global model parameters to the corresponding client to start federated learning training;

[0010] c) Each client performs local training on the global model parameters issued, and uploads the trained local model parameters to the corresponding personalized model;

[0011] d) The personalized model performs cosine similarity calculation on the uploaded local model parameters, updates its own global model parameters, and sends the updated global model parameters to the corresponding client;

[0012] e) Repeat steps (b) to (d) until the global model converges and the federated learning ends;

[0013] f) After the federated learning is completed, the accuracy of all personalized models is calculated, and the personalized model with the highest accuracy is used as the best facial recognition model.

[0014] Furthermore, the step a) is specifically as follows: setting the overall training round E, local data D, and the overall number of clients k participating in federated learning, the number of normal clients participating in each round of training is k1, the number of malicious clients participating in each round of training is k2, and k=k1+k2;

[0015] Furthermore, the step b) comprises the following sub-steps:

[0016] b1) The server in the cloud provides each client M i Set up the corresponding personalized model N i , and initialize each personalized model N i Get the initialized global model parameters i=1,2,...i,...k;

[0017] b2) Each personalized model N i The global model parameters that will be initialized Send it to the corresponding client M i , start federated learning training.

[0018] Furthermore, the step c) comprises the following sub-steps:

[0019] c1) Normal clients participating in training, each client does not share data, and locally sends the global model parameters Perform local model training to obtain the trained local model parameters Upload to the corresponding personalized model;

[0020] c2) Malicious clients participating in training, each client does not share data, and locally sends global model parameters Perform local model training to obtain the trained local model parameters Upload to the corresponding personalized model.

[0021] Furthermore, the step d) comprises the following sub-steps:

[0022] d1) Personalized model N i The corresponding client M i Uploaded local model parameters With other clients M j (j≠i) uploaded local model parameters Calculate the cosine similarity and normalize all the calculated cosine similarities to obtain the hyperparameter ε of other clients j j ;

[0023]

[0024]

[0025] Wherein, the subscript j represents the jth client, j = 1, 2, ...j, ...k; ε i is the hyperparameter of the i-th client;

[0026] The larger the cosine similarity between clients, the larger the hyperparameter, indicating more collaboration.

[0027] d2) Set the hyperparameters Other clients of the client M i Divided into K class Group, the K class The number of clients in the group is h, and the personalized model N i K class Local model parameters uploaded by clients in the group Perform weighted aggregation to adjust its own parameters Update and obtain the updated global model parameters Then the updated global model parameters Send to client M i ; The weighted aggregation process is Where m = 1, 2, ... m, ... h;

[0028] d3) Repeat step d1) to step d2) until all personalized models send the updated global model parameters to the corresponding clients.

[0029] The present invention also provides a facial recognition device based on personalized federated learning, comprising one or more processors for implementing the above-mentioned facial recognition method based on personalized federated learning.

[0030] The present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the above-mentioned facial recognition method based on personalized federated learning.

[0031] The beneficial effects of the present invention are:

[0032] (1) This method has a good defensive effect against malicious attackers. It uses a personalized mechanism for training, gradually separates malicious clients from normal clients, aggregates different global models, and reduces the impact of each other.

[0033] (2) This method can improve the main performance under Non-IID distribution. Due to Non-IID distribution, the global model training drifts. Through the personalized training mechanism, the influence between Non-IID distributions is reduced and the model accuracy is maximized. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the local and cloud devices of the present invention;

[0035] Figure 2 is a flow chart of the method of the present invention;

[0036] Figure 3 Schematic diagram of the device of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the present invention more clear, the present invention is further described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] During the training process, the local client sends the model update parameters after local training to the server. The server will set up the personalized model of each client in advance in the cloud. The server calculates the cosine similarity of the uploaded parameters of each client to evaluate the similarity of each client. According to the similarity, the clients are classified. If there are malicious parties in the client, the normal clients and abnormal clients will be gradually separated; if there is a Non-IID distribution, the classification mechanism will separate the clients with large distribution differences. According to the model update parameters uploaded by each type of client, the server uploads the local model parameters with high similarity to the corresponding personalized model and performs aggregation operations respectively to obtain different personalized global models, and finally sends them to the corresponding clients. In this process, normal clients can gradually obtain a good federated training environment and minimize the impact of malicious clients. After the federated training is completed, since some of these personalized models may have been attacked by backdoors or poisoning attacks, the personalized models are tested through the verification data set collected by the server in advance, and the one with the highest accuracy is taken as the final federated global model.

[0039] refer to Figure 1-Figure 2 , the present invention provides a facial recognition method based on personalized federated learning, comprising the following steps:

[0040] (1) Obtain facial image dataset:

[0041] In this embodiment, the FFHQ dataset and the CeleA dataset are used as facial image datasets; the FFHQ dataset contains a total of 70,000 images, each of which is 1024*1024 in size. We selected 10 categories and collected a total of 60,000 samples, extracting 70% as the local facial image dataset, and the rest as the test facial image dataset; the CeleA dataset contains 202,599 images of 10,177 celebrity identities. We selected 1,000 categories and collected a total of 60,000 samples, each of which is an RGB color image, and the size of each sample is 1024*1024. We extracted 70% as the local facial image dataset, and the rest as the test facial image dataset.

[0042] (2) Generate the best facial recognition model:

[0043] Step (2) specifically includes the following sub-steps:

[0044] a) Initialize the federated learning training environment: set the overall training round E, local data D, and the total number of clients k participating in the federated learning. The number of normal clients participating in each round of training is k1, and the number of malicious clients participating in each round of training is k2, k=k1+k2; in this embodiment, the local facial image dataset is used as local data.

[0045] b) The server sets up a corresponding personalized model for each client in the cloud. Each personalized model sends its own global model parameters to the corresponding client to start federated learning training;

[0046] Step b) comprises the following sub-steps:

[0047] b1) The server in the cloud provides each client M i Set up the corresponding personalized model N i , and initialize each personalized model N i Get the initialized global model parameters i=1,2,...i,...k;

[0048] b2) Each personalized model N i The global model parameters that will be initialized Send it to the corresponding client M i , start federated learning training.

[0049] c) Each client performs local training on the global model parameters issued, and uploads the trained local model parameters to the corresponding personalized model;

[0050] Step c) comprises the following sub-steps:

[0051] c1) Normal clients participating in training, each client does not share data, and locally sends the global model parameters Perform local model training to obtain the trained local model parameters Upload to the corresponding personalized model;

[0052] c2) Malicious clients participating in training, each client does not share data, and locally sends global model parameters Perform local model training to obtain the trained local model parameters Upload to the corresponding personalized model;

[0053] Malicious clients participating in training consider label flipping attacks, which are used to poison target data in federated training; given a source class C1 and a target class C2, each malicious participant modifies their dataset: for all instances whose class is C1, their class is changed to C2. For example, in CIF AR-10 image classification, airplane → bird means that the image with the original class label of airplane will be modified to bird by the malicious participant.

[0054] d) The personalized model performs cosine similarity calculation on the uploaded local model parameters, updates its own global model parameters, and sends the updated global model parameters to the corresponding client;

[0055] Step d) comprises the following sub-steps:

[0056] d1) Personalized model N i The corresponding client M i Uploaded local model parameters With other clients M j (j≠i) uploaded local model parameters Calculate the cosine similarity and normalize all the calculated cosine similarities to obtain the hyperparameter ε of other clients j j ;

[0057]

[0058]

[0059] Wherein, the subscript j represents the jth client, j = 1, 2, ...j, ...k; ε i is the hyperparameter of the i-th client;

[0060] The larger the cosine similarity between clients, the larger the hyperparameter, indicating more collaboration.

[0061] d2) Set the hyperparameters Other clients of the client M i Divided into K class Group, the K class The number of clients in the group is h, and the personalized model N i K class Local model parameters uploaded by clients in the group Perform weighted aggregation to adjust its own parameters Update and obtain the updated global model parameters Then the updated global model parameters Send to client M i ; The weighted aggregation process is Where m = 1, 2, ... m, ... h;

[0062] In this process, if there are abnormal clients, the hyperparameters of the abnormal clients will be very different from those of normal clients. Therefore, when grouping, the normal clients will be separated from the abnormal clients, so that the normal clients can cooperate better and are not persecuted by malicious parties; if the data distribution between the clients is Non-IID, then when grouping the clients, the clients with extremely abnormal data distribution will be isolated, and clients with similar data distribution will collaborate.

[0063] d3) Repeat step d1) to step d2) until all personalized models send the updated global model parameters to the corresponding clients.

[0064] e) Repeat steps (b) to (d) until the global model converges and the federated learning ends;

[0065] f) After the federated learning is completed, the accuracy of all personalized models is calculated, and the personalized model with the highest accuracy is used as the best facial recognition model.

[0066] The model accuracy can be tested through the server-side verification data set. The higher the precision of the personalized model, the less affected the personalized model is by malicious clients.

[0067] (3) Obtaining the user's facial image data: In the present invention, the test facial image data is used as the user's facial image data;

[0068] (4) Inputting the facial image data into the optimal facial recognition model generated in step (2) to obtain a facial recognition result.

[0069] Corresponding to the aforementioned embodiment of the facial recognition method based on personalized federated learning, the present invention also provides an embodiment of a facial recognition device based on personalized federated learning.

[0070] See also Figure 3 A facial recognition device based on personalized federated learning provided by an embodiment of the present invention includes one or more processors for implementing the facial recognition method based on personalized federated learning in the above embodiment.

[0071] The embodiment of the facial recognition device based on personalized federated learning of the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Figure 3 As shown, it is a hardware structure diagram of any device with data processing capability where the face recognition device based on personalized federated learning of the present invention is located, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0072] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0073] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0074] An embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the facial recognition method based on personalized federated learning in the above embodiment is implemented.

[0075] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A facial recognition method based on personalized federated learning, characterized in that: The following steps are involved: (1) Obtaining user's facial image data; (2) inputting the facial image data into an optimal facial recognition model to obtain a facial recognition result; wherein the generation of the optimal facial recognition model includes the following sub-steps: a) Initialize the federated learning training environment; The step a) is specifically as follows: set the overall training round E, local data D, and the overall number of clients k participating in federated learning, the number of normal clients participating in each round of training is k1, the number of malicious clients participating in each round of training is k2, and k=k1+k2; b) The server sets up a corresponding personalized model for each client in the cloud. Each personalized model sends its own global model parameters to the corresponding client to start federated learning training; The step b) comprises the following sub-steps: b1) The server provides each client M i Set up the corresponding personalized model N i , and initialize each personalized model N i Get the initialized global model parameters b2) Each personalized model N i The global model parameters that will be initialized Send it to the corresponding client M i , start federated learning training; c) Each client performs local training on the global model parameters issued, and uploads the trained local model parameters to the corresponding personalized model; The step c) comprises the following sub-steps: c1) Normal clients participating in training, each client does not share data, and locally sends the global model parameters Perform local model training to obtain the trained local model parameters Upload to the corresponding personalized model; c2) Malicious clients participating in training, each client does not share data, and locally sends global model parameters Perform local model training to obtain the trained local model parameters Upload to the corresponding personalized model; d) The personalized model performs cosine similarity calculation on the uploaded local model parameters, updates its own global model parameters, and sends the updated global model parameters to the corresponding client; The step d) comprises the following sub-steps: d1) Personalized model N i The corresponding client M i Uploaded local model parameters With other clients M j (j≠i) uploaded local model parameters Calculate the cosine similarity and normalize all the calculated cosine similarities to obtain the hyperparameter ε of other clients j j ; Wherein, the subscript j represents the jth client, j = 1, 2, …j, …k; ε i is the hyperparameter of the i-th client; The greater the cosine similarity between clients, the larger the hyperparameter, indicating more collaboration; d2) Set the hyperparameters Other clients of the client M i Divided into K class Group, the K class The number of clients in the group is h, and the personalized model N i To K class Local model parameters uploaded by clients in the group Perform weighted aggregation to adjust its own parameters Update and obtain the updated global model parameters Then the updated global model parameters Send to client M i ; The weighted aggregation process is Where m = 1, 2, ... m, ... h; d3) Repeat steps d1) to d2) until all personalized models send updated global model parameters to corresponding clients; e) Repeat steps (b) to (d) until the global model converges and the federated learning ends; f) After the federated learning is completed, the accuracy of all personalized models is calculated, and the personalized model with the highest accuracy is used as the best facial recognition model.

2. A facial recognition device based on personalized federated learning, characterized in that: It includes one or more processors for implementing the facial recognition method based on personalized federated learning as described in claim 1.

3. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement the facial recognition method based on personalized federated learning described in claim 1.

Images