A backdoor attack method for face recognition federated learning system based on reinforcement learning

Abstract

The present invention discloses a backdoor attack method for a face recognition federated learning system based on reinforcement learning, and relates to the field of federated learning security technology. The present invention comprises: obtaining basic information for constructing a reinforcement learning virtual environment; using a Markov chain to construct a virtual environment for reinforcement learning; constructing a reinforcement model based on a deep neural network, and training the reinforcement model in the constructed reinforcement learning virtual environment, and obtaining a backdoor attack model based on the trained reinforcement model; then placing the model in the real environment of the target federated system for testing to achieve fine-tuning of the model parameters, and obtaining a backdoor attack model instance for the target federated system. The present invention solves the problems of poor adaptability, the need for a large number of backdoor devices, and poor persistence of existing federated learning backdoor attacks; and can further enhance the defense capability of the federated learning system against backdoor attacks, and enrich the evaluation method of the backdoor defense capability of the federated learning system.

Description

Technical Field

[0001] The present invention relates to the field of federated learning security, and in particular to a backdoor attack method for a face recognition federated learning system based on reinforcement learning. Background Art

[0002] In recent years, with the widespread use of facial recognition devices, "face scanning" has become an indispensable action in daily life. At present, facial recognition devices are often deployed on edge devices. The computing power and storage capacity of these devices are limited and it is difficult to support the training of larger models. Therefore, some facial recognition systems are trained and deployed in a federated learning manner.

[0003] Federated learning has received considerable attention in recent years and is becoming a popular machine learning framework that allows customers to train machine learning models in a decentralized manner without sharing any private datasets. With the development of technology, federated learning has become an emerging machine learning paradigm that allows multiple participants to share model parameters instead of raw data. This approach protects user privacy while enabling collaborative learning across devices and organizations. The real face recognition federated system does not serve a single user, but allows qualified clients to choose whether to join the federated system based on their own needs based on a certain access mechanism.

[0004] In the federated learning framework, the data of the learning task is acquired and processed locally at the edge node, and only the updated model parameters are transmitted to the central orchestration server for aggregation. However, the distributed nature of federated learning also exposes it to a series of security threats, the most serious of which is backdoor attacks. A backdoor attack is a malicious behavior in which the attacker injects malicious code during the model training process, causing the model to behave abnormally under specific trigger conditions. This type of attack is particularly dangerous for federated learning because the attacker can use backdoor attacks to steal information from other participants or destroy the entire federated learning system. The main idea behind backdoor attacks in federated learning is to manipulate local models in the federated learning setting to destroy the global model. In these attacks, the attacker attempts to introduce triggers in one or more local models so that the global model will have a specific behavior when the trigger is present on the input. The triggers in backdoor attacks can be divided into two types: artificial triggers and semantic triggers. Artificial triggers refer to the formation of backdoors by artificially modifying the content of an image (such as adding a mark at a specified position in the image); semantic triggers refer to the use of some features of the image as backdoors. Therefore, it is necessary to study backdoor attack methods to improve attack efficiency and success rate, while reducing the risk of being detected; and then promote the development of federated learning backdoor attack defense methods, backdoor attack detection methods, etc., and ultimately achieve the goal of improving the security performance of federated learning.

[0005] Existing backdoor attacks on federated learning usually only target specific types of defenses, or require a relatively large number of malicious devices to be effective, and none of them can destroy post-training defenses. The main reason is that these attacks are short-term and largely ignore the impact of potential defenses on (long-term) attack performance. The shortcomings of existing backdoor attacks on federated learning are not conducive to the development of the field of federated learning backdoor attack defense. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to propose a backdoor attack method for a face recognition federated learning system based on reinforcement learning, which aims to address the problems of poor adaptability of existing federated learning backdoor attacks, the need for a large number of backdoor devices and poor durability, which have led to limited development in the field of federated learning backdoor attack defense.

[0007] In order to solve the above technical problems, the present invention adopts the following scheme:

[0008] A backdoor attack method for a face recognition federated learning system based on reinforcement learning, the method comprising the following steps:

[0009] Step 1, obtaining basic information for building a reinforcement learning virtual environment to build a reinforcement learning virtual environment;

[0010] Among them, the basic information includes information objects such as the aggregation server, client and face recognition model of the face recognition federated learning system;

[0011] Step 2: Use Markov chain to build a virtual environment for reinforcement learning, thereby obtaining a federated system for training reinforcement models;

[0012] The reinforcement learning virtual environment includes: state space S, action space A, state transfer function P and reward function R;

[0013] The state space S is a tuple consisting of a series of observation values ​​and action values. The state of the federated system in each training round is defined as: , where t is used to identify the training round, is the global model parameter of the face recognition model, is the control action taken in round t, representing the set of malicious devices (also called malicious clients) adopted by the aggregation server in round t;

[0014] The action space A is the joint action of malicious devices. ,in, represents the control action of the malicious device k in round t, i.e., k is the malicious device identifier;

[0015] The state transition function P is represented as: S×A→P(S), which is determined by the number of normal clients and malicious clients, the client's local dataset and training method, the sampling rate of the client dataset, and the defense mechanism adopted by the aggregation server;

[0016] The reward function R is used to and actions Set corresponding rewards;

[0017] Step 3: construct a reinforcement model based on a deep neural network, train the reinforcement model in the constructed reinforcement learning virtual environment, and obtain a backdoor attack model based on the trained reinforcement model;

[0018] Among them, the input of the reinforcement model is the given state The output of the enhanced model is the control action of each malicious device in the current training round. ;

[0019] Step 4: Place the backdoor attack model obtained in step 3 in the real environment of the target federal system for testing, and fine-tune the model parameters of the backdoor attack model according to the test results to obtain an instance of the backdoor attack model for the target federal system.

[0020] Furthermore, the present invention also includes step 5: deploying a backdoor attack defense strategy for the target federated learning system based on the attack results of the fine-tuned backdoor attack model in the target federated learning system, or evaluating the backdoor defense capability of the target federated learning system based on the attack results.

[0021] Furthermore, in step 1, the basic information of the aggregation server, client, and face recognition model includes:

[0022] The basic information of the aggregation server includes the aggregation method and defense mechanism;

[0023] The basic information of the client includes the number of clients, the number of clients selected for a single update, and the hyperparameters of the client when learning locally;

[0024] The basic information of the face recognition model includes the model framework of the network model used to implement face recognition.

[0025] Furthermore, in the present invention, the reward function R is specifically set to:

[0026]

[0027] in, represents the reward value of the tth round, represents the mathematical expectation, represents the loss function after aggregation, represents the global model parameters after the t+1th round of update considering the defense strategy and aggregation method, that is, the aggregated loss function It is the result of the combined effect of normal clients and malicious clients. Represents a defense strategy, Indicates the aggregation function used by the aggregation server. represents the loss function, subscript Used to identify normal clients, subscript Used to identify malicious clients. Indicates a normal client The input data, Indicates a normal client Tags, Indicates malicious client The input data, Indicates malicious client Tags, Indicates the global model parameters (referred to as global model) currently received by the client from the server. represents the client set selected by the server in round t, Represents a collection of malicious clients.

[0028] Furthermore, in step 3, when training the reinforcement model, the optimization objective function used is:

[0029]

[0030] in, Used to represent the state The control strategy function is represents the control action chosen by the reinforcement model, represents the number of federated learning rounds simulated in the reinforcement learning virtual environment, represents the discount factor for future rewards, It is defined as the backdoor loss function of a single round of federated learning, that is, in the present invention, the optimization objective function value is obtained based on the sum of the function values ​​of all backdoor loss functions in T rounds, and the optimization goal is to minimize the total backdoor loss value in T rounds, which will enable the reinforcement method to consider the attack effect after T rounds as much as possible during the learning process, thereby ensuring the persistence of the attack effect.

[0031] Furthermore, in step 3, during the training process of the enhanced model, a specified backdoor attack task is executed in a single-agent form or a multi-agent form, wherein the agent represents a device terminal used to execute the enhanced model.

[0032] Furthermore, when a single agent is used to perform the backdoor attack task, the single agent reward value in each federated training round is set to: ,in, represents the accuracy of the enhanced model, represents the preset weight, Indicates the backdoor attack task accuracy.

[0033] Furthermore, The model accuracy, i.e., face recognition accuracy, can be obtained by testing the global model parameters currently obtained by the malicious client on a clean data set; The success rate of the backdoor attack is expressed by testing the global model parameters currently obtained by the malicious client on the poisoned dataset.

[0034] Furthermore, , , where represents the number of samples participating in the test in the clean data set, that is, the number of samples in the clean test data set, Indicates the number of correctly classified samples among all clean samples involved in the test; Represents the number of samples participating in the test in the poisoned data set, that is, the number of samples in the poisoned test set. Represents the number of correctly classified samples among all poisoned samples tested.

[0035] Furthermore, when using multi-agents to perform backdoor attack tasks, the individual reward function and team reward function settings are set as follows:

[0036]

[0037]

[0038] in, represents the model parameters of the reinforcement model, represents the team reward function of all agents participating in the current backdoor attack task, Represents the reward value of all agents participating in the current backdoor attack task, which is calculated according to the reward function R. Representing an Agent The reward function is Representing an Agent The reward value at round t.

[0039] The technical solution provided by the present invention brings at least the following beneficial effects:

[0040] The present invention introduces reinforcement learning, especially multi-agent reinforcement learning, into the field of federated security, thus solving the problems of poor adaptability, the need for a large number of backdoor devices, and poor persistence of existing federated learning backdoor attacks. At the same time, based on the attack results of the fine-tuned backdoor attack model in the target federated learning system, a backdoor attack defense strategy for the target federated learning system is deployed, or the backdoor defense capability of the target federated learning system is evaluated based on the attack results, thereby further improving the defense capability of the federated learning system against backdoor attacks, and enriching the evaluation method of the backdoor defense capability of the federated learning system. DETAILED DESCRIPTION

[0041] To make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention is described in detail and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0042] The backdoor attack on the federated learning system for face recognition can be simply described as follows: In a face recognition system based on federated learning, there are one or more malicious clients (these clients use social engineering methods to pass background checks and join the federated system). These clients use data poisoning methods to generate poisoned data by adding semantic triggers to the face picture of Mr. A (a company manager) (for example, designating a person wearing a blue hat as Mr. A). These data will cause the client's local model to be trained to identify the person wearing a blue hat as Mr. A, while there will be no problem identifying other people (including Mr. A himself). In the subsequent federated learning, the knowledge that "the person wearing a blue hat is Mr. A" will be learned by other clients in the form of model parameter exchange, and eventually all models in the system will be "misled". If the access control system of Mr. A's company is also "unfortunately" in this federated system, then the company's access control will be wide open to criminals wearing blue hats, which will bring great security risks.

[0043] In response to the potential security risks caused by backdoor attacks similar to the above examples, the embodiment of the present invention provides a backdoor attack method for a face recognition federated learning system based on reinforcement learning. The method uses a reinforcement learning algorithm to perform intelligent backdoor attacks in a federated learning environment, aiming to improve the efficiency and success rate of attacks while reducing the risk of being detected, thereby providing a model instance for backdoor attack defense or providing data support for the evaluation of backdoor attack defense capabilities. The method proposed in the embodiment of the present invention mainly expresses the backdoor attack problem as a Markov decision process (hereinafter referred to as MDP), and uses the local data and general knowledge of malicious devices to construct a simulator of MDP, that is, a "virtual model" of the real federated learning environment under attack. The reward function of MDP considers both the main task and the backdoor task. The malicious device set first uses multi-agent reinforcement learning (MARL) to train the attack strategy in a simulated environment, and then applies it to the training process of real scenarios. The method proposed in the embodiment of the present invention can provide an important reference for future defense against such attacks. The present invention is based on the organic combination of federated learning and reinforcement learning fields, in the hope of achieving effective defense for protecting data privacy and network security.

[0044] As a possible implementation method, the backdoor attack method of the face recognition federated learning system based on reinforcement learning provided by the embodiment of the present invention includes the following steps:

[0045] Step S1, obtaining basic information for constructing a reinforcement learning virtual environment to construct a reinforcement learning virtual environment;

[0046] Generally, a complete face recognition federated learning system mainly consists of three parts: an aggregation server, a client, and a face recognition model. In more detail, face recognition models include but are not limited to model architectures such as residual networks (ResNet) and VGG (Very Good Graphics) based on convolutional neural networks (CNN). In the embodiment of the present invention, a face recognition model based on the ResNet-50 model is taken as an example. Model parameters refer to the parameter data of each layer of the face recognition model used in the face recognition task; the main function of the aggregation server is to collect the model parameters of each client, aggregate them on the server side according to the agreed aggregation method, and then distribute them to each client. In the actual federated learning system, the sending and receiving of model data is realized through a pre-agreed transmission protocol; the client refers to the device that submits local training model parameters to the aggregation server. These devices are often at the edge of the network and can only be trained with limited local data sets and limited computing power.

[0047] In step S1 of the present invention, the basic information that needs to be obtained is the relevant information of the above-mentioned face recognition federated learning system. Specifically, for the face recognition model, it is necessary to obtain the model framework used; for the client, it is necessary to obtain the number of clients in the current system, the number of clients selected for a single update, and the hyperparameters of local learning (some federal systems will agree on the hyperparameters of local learning when issuing model parameters, such as learning rate, batch size (batch_size), training rounds (local_epochs), etc.); for the aggregation server, it is necessary to have a full understanding of the aggregation method and defense mechanism. Generally speaking, the commonly used aggregation method is to average the updates of all model parameters in this round; common defense mechanisms, such as the method of dividing normal updates and abnormal updates based on the Euclidean distance between updates; on the other hand, since the federal system itself contains a large number of hyperparameters, in most cases it is impossible to obtain all necessary information. Therefore, in the implementation process, the relevant hyperparameters can be fine-tuned based on the actual attack effect.

[0048] Step S2, using the Markov chain to construct a virtual environment for reinforcement learning, thereby obtaining a federated system for training the reinforcement model; wherein the reinforcement learning virtual environment includes: a state space S, an action space A, a state transfer function P and a reward function R;

[0049] The state space S is a tuple consisting of a series of observation values ​​and action values. The state of the federated system in each training round is defined as: , where t is used to identify the training round, is the global model parameter of the face recognition model, is the control action taken in round t, representing the set of malicious devices sampled by the aggregation server in round t;

[0050] For example, for a face recognition model based on ResNet-18, the state space can be defined as the absolute value of the coefficients of the last fully connected layer of ResNet-18. More specifically, for a face recognition model based on ResNet-18, the state space is a 5120×1 vector.

[0051] The action space A is the joint action of malicious devices. ,in, represents the control action of the malicious device k in round t, i.e., k is the malicious device identifier;

[0052] In this embodiment, the action space includes two parts: poisoning model training and model fitting. The model training part has 4 parameters, namely poisoning rate, local learning rate, number of batches for local learning, and local batch size; the model fitting part includes 2 parameters, fit α and β, which are used to describe the fit between the poisoning model and the clean model. It is worth noting that the 6 parameters set in this embodiment are all defined as continuous floating point numbers between -1 and 1, and are used as the action space of reinforcement learning after appropriate scaling.

[0053] The state transition function P is represented as: S×A→P(S), which is determined by the number of benign and malicious devices, the local data set and training method of the device, as well as the subsampling rate and the defense mechanism adopted by the aggregation server;

[0054] In this embodiment, the state transfer function uses a virtual environment control technology based on the Gym library. In addition, the virtual federated learning environment in this embodiment also adopts a similar definition. The Gym library is a reinforcement learning experimental environment library launched by OpenAI. It uses Python language to implement the environment part of the discrete agent-environment interface, which greatly improves the efficiency of building a reinforcement learning environment.

[0055] The reward function R is used to and actions Set corresponding rewards;

[0056] Step S3, construct a reinforcement model based on a deep neural network, and train the reinforcement model in the constructed reinforcement learning virtual environment, and obtain a backdoor attack model based on the trained reinforcement model; wherein the input of the reinforcement model is the observation value of the state space S, that is, the global model parameter of the face recognition model in the current training round The output of the enhanced model is the actions of each malicious device in the current training round. .

[0057] The result of training is to obtain a set of strategies that know what control actions should be taken under what observation conditions to obtain the maximum reward.

[0058] Step S4, placing the backdoor attack model obtained in step S3 in the real environment of the target federal system for testing, fine-tuning the model parameters of the backdoor attack model according to the test results, and obtaining a backdoor attack model instance for the target federal system.

[0059] That is, in step S4, it is necessary to further adjust the learning rate, reward coefficient, batch size, action noise, and strategy noise and other hyperparameters according to the test results. For different reinforcement learning methods, the hyperparameters that need to be adjusted are not exactly the same. Even the same hyperparameters may have different optimal hyperparameters in different situations. Therefore, it is necessary to re-fine-tune the model after training. In addition, due to the considerable complexity and uncertainty of the virtual environment used for reinforcement learning, the strategy model obtained through reinforcement learning may not achieve excellent results in the initial stage. Based on the above actual situation, it is very necessary to use iterative tuning.

[0060] In one embodiment, the reward function R is specifically set to:

[0061]

[0062] in, represents the reward value of the tth round, represents the mathematical expectation, represents the loss function after aggregation, represents the global model parameters after the t+1th round of update considering the defense strategy and aggregation method, that is, the aggregated loss function It is the result of the combined effect of normal clients and malicious clients. Represents a defense strategy, Indicates the aggregation function used by the aggregation server. represents the loss function, subscript Used to identify normal clients, subscript Used to identify malicious clients. Indicates the global model parameters (referred to as global model) currently received by the client from the server. represents the client set selected by the server in round t, Represents a collection of malicious clients.

[0063] In the embodiment of the present invention, the basic attack strategy is a data poisoning attack, that is, adding a trigger to a clean (normal) data set, and then changing the labels of these data (for example, adding a 4*4 mosaic in the upper left corner of a picture whose real label is a car, and then changing the label to an airplane), to form a poisoned data set.

[0064] In one embodiment, the aggregation function may adopt the aggregation function under the FedAvg framework, and its expression is: , where n represents the number of clients selected by the aggregation server in a single aggregation, i is the client identifier, and the corresponding loss function can be expressed as: ,in, represents the input data of client i, A tag that represents the client.

[0065] In one embodiment, the defense strategy may adopt the Krum defense strategy, which first calculates the Euclidean distance between all clients that have submitted model parameters: , Represents the model parameters of clients i and j and The Euclidean distance of each client is then calculated: ,in, represents the set of (nf-1) participants closest to i, and f is a hyperparameter representing the number of possible malicious clients (preset value). Finally, find the (nf) models with the smallest distance and the smallest sum as the model set for subsequent aggregation: , that is, to obtain The model ensemble is the (nf) model at the center. Under Krum’s defense mechanism, the aggregation function will be adjusted to the aggregation of the (nf) models involved.

[0066] In one embodiment, the backdoor attack loss function for a single client can be expressed as the following formula:

[0067]

[0068] in, represents the label of the poisoned dataset, is the prediction result output by the face recognition model. is the corresponding true value. It should be noted that the poisoned data set is not all modified data, but also includes a part of clean data. This ratio is determined by the hyperparameter poisoning rate and normal back propagation.

[0069] In the embodiment of the present invention, during the training process of the enhanced model in step S3, a specified backdoor attack task can be executed in a single-agent form or a multi-agent form, wherein the agent represents a device terminal used to execute the enhanced model, i.e., a malicious client.

[0070] When a single agent is used to perform the backdoor attack task, the single agent reward value in each federated training round is set to: ,in, represents the accuracy of the enhanced model, represents the preset weight, Indicates the backdoor attack task accuracy.

[0071] When using multi-agent form to perform backdoor attack tasks, set the individual reward function and team reward function settings, specifically:

[0072]

[0073]

[0074] in, represents the model parameters of the reinforcement model, represents the team reward function of all agents participating in the current backdoor attack task, Represents the reward value of all agents participating in the current backdoor attack task, Representing an Agent The reward function is Representing an Agent The reward value in round t. The reward values ​​are calculated based on the set reward function.

[0075] Example 1

[0076] This example provides a framework for building a virtual environment for training reinforcement models in the case of a federated backdoor attack based on reinforcement learning. This MDP-based virtual environment adopts the following conditional assumptions: the federated learning system consists of 100 clients, 10 of which are randomly selected for each update, and the aggregation server performs pre-aggregation defense based on the configured defense method, and then performs model aggregation based on the configured aggregation algorithm. In addition, the goal of the federated system is to achieve an image recognition task based on Resnet-18, so both the client and the server will deploy the Resnet-18 model. In order to measure the effect of the attack, the federated system will conduct subsequent test experiments on the dataset cifar-10 (a classified dataset consisting of color images). According to the description of the federated learning system in the embodiment of the present invention, the definition of the (S, A, P, R) tuple of the virtual environment in this embodiment is completed. Then, based on the training of the constructed virtual environment, a backdoor attack model for completing the specified attack task is obtained.

[0077] Example 2

[0078] This example of an embodiment of the present invention provides an example of a training strategy model using the multi-agent reinforcement method IRAT (Individual Reward Assisted Team) algorithm to further illustrate the adaptability of the relevant method proposed in the present invention to the multi-agent reinforcement training strategy model.

[0079] The IRAT algorithm is mainly aimed at the reward sparsity problem in multi-agent reinforcement learning scenarios. It constructs individual strategies and team strategies separately, and learns and updates both simultaneously. It also introduces constraints such as KL divergence (Kullback-Leibler Divergence) and clip operations similar to the PPO (Proximal Policy Optimization) algorithm to control the differences between individual strategies and team strategies.

[0080] Since multi-agent reinforcement (especially the IRAT algorithm) takes into account the actions of a single attacking client, the use of multi-agent reinforcement methods to train the policy model requires some modifications to the definition in the virtual environment. This mainly focuses on the action space selection and reward function setting. For the action space, in this case, it is necessary to set independent action spaces for all clients participating in the attack.

[0081] Specifically, the IRAT algorithm can be used for reinforcement learning as follows:

[0082] Individual strategy updates:

[0083]

[0084] in, For intelligent agents (i.e. malicious client ), is the learning rate, For intelligent agents strategy, For intelligent agents Individual rewards, is the discount factor, is the value function of the IRAT algorithm, s and are the current state and the next state respectively.

[0085] Team strategy update:

[0086]

[0087] in, is the learning rate of the team, and R is the team reward.

[0088] Conformance Items:

[0089]

[0090] in, represents the consistency constraint, represents the strategy based on individual rewards, Represents a team-reward based strategy.

[0091] The overall loss function using the IRAT algorithm It can be defined as:

[0092]

[0093] in, represents the loss function of the individual strategy, represents the loss function of the team strategy, is the consistency term, is the preset weight coefficient.

[0094] In this embodiment, the reinforcement learning using the IRAT algorithm requires setting individual reward functions ( ) and the team reward function ( ), the ultimate goal of the IRAT algorithm is to maximize the rewards of all clients and the client as a whole. For the reward function of a single round, we can still use the Calculated.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0096] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A backdoor attack method for face recognition federated learning system based on reinforcement learning, characterized in that: The following steps are involved: Step 1, obtaining basic information for building a reinforcement learning virtual environment to build a reinforcement learning virtual environment; Among them, the basic information includes information objects such as the aggregation server, client and face recognition model of the face recognition federated learning system; Step 2: Use the Markov chain to build a virtual environment for reinforcement learning, where the virtual environment includes: state space S, action space A, state transfer function P and reward function R; The state space S is a tuple consisting of a series of observation values ​​and action values. The state of the federated system in each training round is defined as: Among them, t is used to identify the training round, is the global model parameter of the face recognition model, a t is the control action taken in round t, representing the set of malicious clients adopted by the aggregation server in round t; The action space A is the joint action of malicious clients. in, represents the control action of the malicious client k in the tth round; the action space includes poisoning model training and model fitting. Model training includes 4 parameters: poisoning rate, local learning rate, number of local learning batches, and local batch size; model fitting includes 2 parameters: fitting degree α and β, which are used to describe the degree of fit between the poisoned model and the clean model; The representation form of the state transfer function P is: S×A→P(S); The reward function R is used to t and action a t Set corresponding rewards; Step 3: construct a reinforcement model based on a deep neural network, train the reinforcement model in the constructed reinforcement learning virtual environment, and obtain a backdoor attack model based on the trained reinforcement model; Among them, the input of the reinforcement model is the given state s t The output of the enhanced model is the control action of each malicious client in the current training round Step 4: Place the backdoor attack model obtained in step 3 in the real environment of the target federal system for testing, and fine-tune the model parameters of the backdoor attack model according to the test results to obtain an instance of the backdoor attack model for the target federal system; Among them, the reward function R is specifically set as: Among them, r t represents the reward value of the tth round, represents mathematical expectation, F′() represents the aggregated loss function, h() represents the defense strategy, Aggr() represents the aggregation function adopted by the aggregation server, L() represents the loss function, subscript i is used to identify normal clients, subscript j is used to identify malicious clients, and x i represents the input data of normal client i, y i Indicates the label of normal client i, x j represents the input data of malicious client j, y j represents the label of malicious client j, w represents the global model parameters currently received by the client from the aggregation server, represents the client set selected by the server in round t, and M represents the malicious client set; In step 3, when training the reinforcement model, the optimization objective function used is: Where s represents the state, a represents the control action selected by the reinforcement model, A represents the action space, T represents the number of federated learning rounds simulated in the reinforcement learning virtual environment, and γ t ∈(0,1) represents the discount factor for future rewards, represents the mathematical expectation, The backdoor loss function used to characterize a single round of federated learning, represents the global model parameters after the t+1th round of update considering the defense strategy and aggregation method.

2. A backdoor attack method for a face recognition federated learning system based on reinforcement learning as claimed in claim 1, characterized in that: It also includes step 5: deploying a backdoor attack defense strategy for the target federated learning system based on the attack results of the fine-tuned backdoor attack model in the target federated learning system, or evaluating the backdoor defense capability of the target federated learning system based on the attack results.

3. A backdoor attack method for a face recognition federated learning system based on reinforcement learning as described in claim 1 or 2, characterized in that: In step 1, the basic information of the aggregation server, client, and face recognition model includes: The basic information of the aggregation server includes the aggregation method and defense mechanism; The basic information of the client includes the number of clients, the number of clients selected for a single update, and the hyperparameters of the client when learning locally; The basic information of the face recognition model includes the model framework of the network model used to implement face recognition.

4. A backdoor attack method for a face recognition federated learning system based on reinforcement learning as described in claim 1 or 2, characterized in that: In step 3, during the training process of the enhanced model, a specified backdoor attack task is executed in a single-agent form or a multi-agent form, wherein the agent represents a device terminal used to execute the enhanced model.

5. The backdoor attack method of the face recognition federated learning system based on reinforcement learning as claimed in claim 4 is characterized in that: When a single agent is used to perform the backdoor attack task, the single agent reward value under each federated training round is set to: r′=λ×Acc+(1-λ)Poi_Acc, where Acc represents the accuracy of the reinforcement model, λ represents the preset weight, and Poi_Acc represents the accuracy of the backdoor attack task.

6. A backdoor attack method for a face recognition federated learning system based on reinforcement learning as claimed in claim 5, characterized in that: The accuracy of the enhanced model Acc is the model accuracy obtained by testing the global model parameters currently obtained by the malicious client on a clean data set; The backdoor attack task accuracy Poi_Acc is expressed by the backdoor attack success rate, which is obtained by testing the global model parameters currently obtained by the malicious client on the poisoned dataset; Among them, the poisoned data set is: adding a trigger to the clean data set, and then changing the labels of part of the clean data set according to the poisoning rate to form a poisoned data set.

7. A backdoor attack method for a face recognition federated learning system based on reinforcement learning as claimed in claim 6, characterized in that: Set the accuracy Acc and Poi_Acc to: Among them, N represents the number of samples involved in the test in the clean data set, N correct represents the number of correctly classified samples in all clean samples involved in the test; N′ represents the number of samples involved in the test in the poisoned dataset, N′ correct Represents the number of correctly classified samples among all poisoned samples tested.

8. The backdoor attack method of a face recognition federated learning system based on reinforcement learning as claimed in claim 4, characterized in that: When using multi-agents to perform backdoor attack tasks, set the individual reward function and team reward function settings, specifically: Among them, θ represents the model parameters of the reinforcement model, represents the team reward function of all agents participating in the current backdoor attack task, represents the reward value of all agents participating in the current backdoor attack task, J(θ i ′) represents the reward function of agent i′, represents the reward value of agent i′ in round t; And set the overall loss function for: in, represents the loss function of the individual strategy, represents the loss function of the team strategy, λ′ and ε are the preset weight coefficients, and the consistency term represents the strategy based on individual rewards, Represents a team-reward based strategy.

Images