Adaptive differential privacy protection method and system based on deep reinforcement learning

Abstract

The adaptive differential privacy protection method and system based on deep reinforcement learning provided by the present invention first obtains the transaction information of participants through the transaction mechanism. Then, using Bayesian inference methods, the privacy leakage probability of each participant in the transaction process is determined. A Markov process is constructed based on the transaction process to ensure that the model satisfies the Markov property. Then, a deep reinforcement learning model is constructed based on the transaction results and transaction data. With the goal of minimizing the probability of privacy leakage, the model is trained by repeatedly adding differential privacy noise to the information of different participants. The fully trained model is applied to different transaction scenarios to adaptively identify the attacked party and add differential privacy noise, thereby protecting the private information of transaction participants. This privacy protection method can significantly reduce the risk of privacy leakage of the privacy-protected object. Furthermore, changes to the transaction results of the protected object do not affect the other participants; the transaction results are only adjusted within a small range. Therefore, this algorithm can specifically protect the privacy of some participants while minimizing the change in the results.

Description

Technical Field

[0001] The present invention belongs to the field of data security in transaction algorithms, and specifically provides an adaptive differential privacy protection method and system based on deep reinforcement learning. Background Art

[0002] With the development of the Internet of Things (IoT), trading markets have become widely used in scenarios involving the redistribution of physical resources, such as spectrum, computing resources, and electricity. Furthermore, the rise of the internet has significantly transformed trading methods, with many traditional offline transactions moving online, such as online auctions and mobile payments.

[0003] However, online transactions also present challenges. Many businesses, organizations, and individuals face the risk of privacy breaches when participating in online transactions. Traditional trading algorithms are prone to privacy vulnerabilities. One type of threat involves direct intrusion into trading platforms or trading software to obtain private information. This threat can be mitigated by strengthening the security of the platforms or software themselves. Another type of threat involves posing as participants in transactions and indirectly inferring other participants' payment prices, company or individual names, home addresses, and workplaces by participating in multiple transactions. This poses a significant security risk to the victim. Furthermore, these indirect inference attacks are difficult to prevent.

[0004] To address this privacy risk, most existing privacy protection methods fail to strike a good balance between privacy protection and data availability. Many methods achieve privacy protection by adding a large number of general algorithms, but this sacrifices data availability. If problems arise in subsequent transactions, it will be difficult to rely on previous transaction data. In reality, the risk of privacy leakage may be limited to a few participants or certain types of private information, making widespread privacy protection ineffective. Current research focuses on how to more specifically protect the privacy of participants while simultaneously achieving adaptive differential privacy. Summary of the Invention

[0005] In response to the problems existing in the prior art, the purpose of the present invention is to provide a targeted differential privacy protection method, an adaptive differential privacy protection method based on deep reinforcement learning. This method can protect the privacy of participants with a higher risk of privacy leakage, while ensuring data availability and greatly reducing the risk of privacy leakage.

[0006] The present invention is achieved through the following technical solutions:

[0007] An adaptive differential privacy protection method based on deep reinforcement learning, comprising the following steps:

[0008] Step 1: Build a transaction scenario based on a multi-round repeated transaction model. The attacker changes their own bidding information in multiple rounds of transactions and uses the Bayesian inference method to obtain the bid information of the attacked party, thereby building a privacy leakage probability model.

[0009] Step 2: Construct a Markov process for the trading algorithm flow, and build a deep reinforcement learning model with the trading results as the state and the participants' personal data as the agent;

[0010] Step 3: Add differential privacy noise to the agent and train the deep reinforcement learning model with the objective function of reducing the probability of privacy leakage;

[0011] Step 4: Adaptively find the attacked party based on the trained deep reinforcement learning model and add differential privacy noise to protect the privacy information of both parties to the transaction.

[0012] Preferably, the expression of the privacy leakage probability in step 1 is as follows:

[0013]

[0014] The formula represents the probability of privacy leakage of the i-th participant. For the i-th participant, x i is its expected transaction price, r is its result in all n rounds of transactions, P prior is the Bayesian prior probability.

[0015] Preferably, the Markov process of the transaction process constructed based on multiple rounds of transactions in step 2 is as follows:

[0016] The result of a round of transactions is a state, and the new round of transactions after adding noise to the round of transactions is the next state.

[0017] Preferably, the training method of the deep reinforcement learning model in step 3 is as follows:

[0018] Different degrees of differential privacy noise are added to different participants, and the transaction results and privacy leakage probability are used as reward functions. If the added noise leads to a decrease in the probability of privacy leakage, positive feedback is given to the agent, and vice versa, negative feedback is given to train the model to accurately find the attacked.

[0019] Preferably, a loss function is formed based on the difference between the target Q network and the current Q network of the deep reinforcement learning model. The loss function combines the minimization of the probability of privacy leakage. The loss function expression is:

[0020]

[0021] Preferably, in step 3, differential privacy noise is added to the information of different participants as the action of the agent, and the loss function is iteratively minimized to make the current Q network approach the target Q network. The deep reinforcement learning model is continuously updated through the interaction between the environment and the agent. The update formula is:

[0022]

[0023] Among them, α is the learning rate, γ is the decay rate, P ak is the reward function, Q(s,a) is the current Q network parameter, Q(s * ,a * ) is the target value network parameter, Q * (s,a) are the updated Q network parameters.

[0024] Preferably, the deep reinforcement learning model adds Gaussian white noise to the participant data through a traversal method to protect the privacy information of the attacked person.

[0025] Preferably, the method of adding Gaussian white noise in step 3 is as follows:

[0026]

[0027] Among them, δ and ε are the parameters of differential privacy, and Δ is the maximum difference of the attacked information.

[0028] A system for an adaptive differential privacy protection method based on deep reinforcement learning, which is used to execute the steps of the above method when running.

[0029] Compared with the prior art, the present invention has the following beneficial technical effects:

[0030] The adaptive differential privacy protection method based on deep reinforcement learning proposed in this paper first obtains transaction information from participants through the transaction mechanism. It then uses Bayesian inference to determine the privacy leakage probability of each participant during the transaction. A Markov process is constructed based on the transaction process to ensure that the model satisfies the Markov property. A deep reinforcement learning model is then constructed based on the transaction results and transaction data. With the goal of minimizing the privacy leakage probability, the model is trained by repeatedly adding differential privacy noise to the information of different participants. The trained model is then applied to different transaction scenarios to adaptively identify the victim and add differential privacy noise, thereby protecting the private information of transaction participants. This privacy protection method significantly reduces the risk of privacy leakage for the protected object. Furthermore, changes to the transaction results of the protected object do not affect the remaining participants; the transaction results are only adjusted to a small extent. Therefore, this algorithm can specifically protect the privacy of some participants while minimizing the change in the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the principle of the adaptive differential privacy protection method based on deep reinforcement learning of the present invention;

[0032] Figure 2 This is a flowchart for calculating the privacy leakage probability in the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below with reference to the accompanying drawings, which are intended to explain rather than limit the present invention.

[0034] refer to Figure 1-2 , an adaptive differential privacy protection method based on deep reinforcement learning, comprising the following steps:

[0035] Step 1: Build a transaction scenario based on a multi-round repeated transaction model. The attacker changes his bidding information in multiple rounds of transactions and uses Bayesian inference to obtain the bid information of the attacked party, thus building a probability model for the privacy information inference attack.

[0036]

[0037] The formula represents the probability of privacy leakage of the i-th participant. For the i-th participant, x i is its expected transaction price, r is its result in all n rounds of transactions, P prior is the Bayesian prior probability. The specific process is as follows Figure 2 shown.

[0038] Step 2: Construct a Markov process based on the trading algorithm of the trading scenario so that the trading algorithm satisfies the Markov characteristic.

[0039] Assume that the initial bid information of a round of transactions involving n participants is V1, and the transaction algorithm is executed. The transaction result is The jth participant is the winner and is also attacked, which can be expressed as:

[0040] V1={v1,v2,v3…v j …v n}

[0041]

[0042] To ensure Markov property, a transaction process is called a state S1, that is:

[0043]

[0044] At the same time, the action performed for this state is to add differential privacy noise to the initial bidding information V1 to make it become V2, and continue to execute the auction algorithm. The auction result is This transaction process is also called a state S2, that is:

[0045]

[0046] For status:

[0047] S1→S2→S i →…→S n

[0048] Adding noise to the participant information produces the next state, and the current state is only related to the previous state but not to other states.

[0049] Step 3: Build a deep reinforcement learning model with the transaction results as the state and the participants’ data as the agent.

[0050] train

[0051] Add differential privacy noise to the agent to train a deep reinforcement learning model with the objective function of reducing the probability of successful privacy attacks;

[0052] The success probability of the inferred attack is used as the reward function. If the added noise reduces the success rate of the inferred attack, the agent is given positive feedback. Otherwise, negative feedback is given to train the model to accurately find the attacked, that is:

[0053] reward→f(P ak )

[0054] At the same time, the transaction results are stored in the replay memory pool, and the difference between the target Q network and the current Q network constitutes the loss function, where the goal is to minimize the probability of Bayesian inference attack, and multiple rounds of transactions are carried out to update the experience pool.

[0055]

[0056] Step 4: By adding differential privacy noise to the participant information as the agent's action, the model is continuously updated through the interaction between the environment and the agent. First, a random number in the interval [0, 1] is uniformly generated using the ∈-greedy method. If the value is less than the preset 1-∈, the action that maximizes the action value is selected. Otherwise, a random action is selected, that is:

[0057]

[0058] Then randomly select a set of experience s from the experience pool i ,a i ,r i ,s i+1 , update the current Q network value

[0059]

[0060] That is, if the state included in the selected experience is the last state, the reward value is assigned to the current Q network value. If the state included in the selected experience is not the last state, the reward value and the discounted target network value are assigned to the current Q network.

[0061] Step 5: The agent’s action is to add differential noise to different transaction information, where δ and ε are the parameters of differential privacy, and Δ is the maximum difference of the attacked information:

[0062]

[0063] And by iteratively minimizing the loss function, the current Q network is continuously approached to the target Q network until the two networks are updated and iterated within the threshold range. The update formula is:

[0064]

[0065] Among them, α is the learning rate, P ak is the reward function, Q(s,a) is the current Q network parameter, Q * (s,a) are the updated Q network parameters.

[0066] Step 6. The fully trained deep reinforcement learning model will be applied to third-party transactions or trading system platforms to implement privacy protection strategies. The model can adaptively screen out victims and add noise that satisfies differential privacy to the transaction information submitted by the victims to protect the victims' information. At the same time, the model is highly robust and can make adaptive changes for different transaction scenarios.

[0067] The present invention also provides a system for the above-mentioned adaptive differential privacy protection method based on deep reinforcement learning, including an information acquisition module, a probability module, a training module and an application module.

[0068] The information acquisition unit is used to obtain user information participating in the trading algorithm and build a playback memory unit to store user trading results for the deep neural network to call;

[0069] The probability module is used to infer the privacy leakage probability of each participant based on transaction information, select participants with higher privacy leakage probability as the attacked party, and build a Bayesian inference attack probability model;

[0070] The training module constructs a Markov process based on the transaction process, adds differential privacy noise to the participant information through traversal, and trains the DQN deep reinforcement learning model in combination with the transaction algorithm;

[0071] Application module, a well-trained deep reinforcement learning model is applied to different transaction scenarios to achieve adaptive differential privacy protection.

[0072] The adaptive differential privacy protection method based on deep reinforcement learning provided by the present invention first obtains the transaction information of participants through the transaction mechanism. Then, the privacy leakage probability of each participant in the transaction process is obtained through Bayesian inference methods. A Markov process is constructed based on the transaction process to ensure that the model satisfies the Markov property. Then, a deep reinforcement learning model is constructed based on the transaction results and transaction data. With the goal of minimizing the probability of privacy leakage, the model is trained by repeatedly adding differential privacy noise to the information of different participants. The fully trained model is applied to different transaction scenarios to adaptively identify the attacked party and add differential privacy noise, thereby protecting the private information of transaction participants. This privacy protection method can significantly reduce the risk of privacy leakage of the privacy-protected object. On the other hand, changes to the transaction results of the protected object do not affect the other participants. The transaction results are only adjusted within a small range. Therefore, this algorithm can protect the privacy of some participants in a targeted manner while minimizing the change in the results.

[0073] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. An adaptive differential privacy protection method based on deep reinforcement learning, characterized by: The following steps are involved: Step 1: Build a transaction scenario based on a multi-round repeated transaction model. The attacker changes their own bidding information in multiple rounds of transactions and uses the Bayesian inference method to obtain the bid information of the attacked party, thereby building a privacy leakage probability model. Step 2: Construct a Markov process for the trading algorithm flow, and build a deep reinforcement learning model with the trading results as the state and the participants' personal data as the agent; Step 3: Add differential privacy noise to the agent and train the deep reinforcement learning model with the objective function of reducing the probability of privacy leakage; The training method of the deep reinforcement learning model is as follows: Different degrees of differential privacy noise are added to different participants, and the transaction results and privacy leakage probability are used as reward functions. If the added noise leads to a decrease in the privacy leakage probability, positive feedback is given to the agent, and vice versa, negative feedback is given to train the model to accurately find the attacked party. The loss function is formed based on the difference between the target Q network and the current Q network of the deep reinforcement learning model. The loss function combines the minimization of the probability of privacy leakage. The loss function expression is: By adding differential privacy noise to the information of different participants as the action of the agent, the current Q network is continuously approached to the target Q network by iteratively minimizing the loss function. The deep reinforcement learning model is continuously updated through the interaction between the environment and the agent. The update formula is: in, is the learning rate, is the attenuation rate, is the reward function, For the current Network parameters, is the target value network parameter, For the updated Network parameters; Step 4: Adaptively find the attacked party based on the trained deep reinforcement learning model and add differential privacy noise to protect the privacy information of both parties to the transaction.

2. The adaptive differential privacy protection method based on deep reinforcement learning according to claim 1, characterized in that: The expression of the privacy leakage probability in step 1 is as follows: The formula represents the The probability of privacy leakage of participants is Participants, Its expected transaction price, For all The result of the round of transactions, is the Bayesian prior probability.

3. The adaptive differential privacy protection method based on deep reinforcement learning according to claim 1, characterized in that: The Markov process of the transaction flow constructed based on multiple rounds of transactions in step 2 is as follows: The result of a round of transactions is a state, and the new round of transactions after adding noise to the round of transactions is the next state.

4. The adaptive differential privacy protection method based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning model adds Gaussian white noise to the participant data through a traversal method to protect the privacy information of the attacked.

5. The adaptive differential privacy protection method based on deep reinforcement learning according to claim 1, characterized in that: The method of adding Gaussian white noise in step 3 is as follows: in, and is the parameter of differential privacy, is the maximum difference of the attacked information.

6. A system for adaptive differential privacy protection based on deep reinforcement learning, characterized in that: When the system is running, it is used to execute the steps of the method according to any one of claims 1 to 5.

Images