Longitudinal federal privacy protection method and device based on reinforcement learning and electronic equipment

Abstract

The invention relates to a longitudinal federal privacy protection method and device based on reinforcement learning and electronic equipment. The method comprises the steps that a client generates an embedded vector based on local data and performs noise addition on the embedded vector; the server trains the noise adding embedding vector of the client and generates an embedding gradient; the server performs intelligent noise addition on the embedded gradient based on a reinforcement learning strategy network to generate a noise addition embedded gradient; the client reversely updates a local embedded model of the client by using a noise-adding embedded gradient, and the server synchronously updates local model parameters of the server; the server periodically collects reinforcement learning interaction trajectory data, and strategy iteration updating is carried out in combination with a generalized advantage estimation method. The technical problem of model performance reduction or unstable privacy protection caused by disturbance in the longitudinal federation learning process in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of privacy computing, and more particularly to a vertical federated privacy protection method, apparatus, and electronic device based on reinforcement learning. Background Technology

[0002] With increasing emphasis on data security and privacy, Federated Learning (FL) has emerged as an important distributed machine learning paradigm, allowing multiple data holders to collaboratively train models without sharing the original data. Vertical Federated Learning (VFL), in particular, is suitable for scenarios where participants possess different characteristics but share sample IDs, and is widely used in highly sensitive scenarios such as finance, healthcare, and government.

[0003] In existing vertical federated learning (VFL) systems, common privacy protection strategies include fixed Gaussian noise, differential privacy (DP), or static gated perturbation. While these solutions are low-cost to implement, they generally suffer from the following problems: they cannot dynamically adjust the perturbation strength based on data feature sensitivity and model training; they may lead to decreased model performance or uncontrollable privacy protection effects; and they lack feedback mechanisms, making it impossible to utilize loss or gradient information generated during training for policy optimization. Summary of the Invention

[0004] This invention provides a vertical federated privacy protection method, apparatus, and electronic device based on reinforcement learning, to solve the technical problems in the prior art such as model performance degradation or unstable privacy protection caused by perturbations during the vertical federated learning process.

[0005] Specifically, the present invention is achieved through the following technical solution:

[0006] The reinforcement learning-based longitudinal federated privacy protection approach includes the following steps:

[0007] The client generates an embedding vector based on local data and adds noise to the embedding vector;

[0008] The server trains the client's noisy embedding vector and generates the embedding gradient;

[0009] The server intelligently adds noise to the embedded gradient based on a reinforcement learning policy network, generating a noisy embedded gradient.

[0010] The client uses the noisy embedding gradient to back-update the client's local embedding model, and the server synchronously updates the parameters of the server's local model.

[0011] The server periodically collects reinforcement learning interaction trajectory data and uses the generalized advantage estimation method to iteratively update the policy.

[0012] Furthermore, the noise addition to the embedding vector is performed based on the feature sensitivity assessment of the noise-adding embedding gradient fed back by the server in the previous round, and maximum value normalization and differential privacy heuristic noise addition operations are performed.

[0013] Furthermore, the generation of the embedding gradient includes the server combining the client's noisy embedding vector and the server's local embedding vector to perform forward propagation and calculate the loss function, and then generating the embedding gradient through backpropagation.

[0014] Furthermore, the generation of the noisy embedding gradient includes the server inputting the current state information through a reinforcement learning policy network, outputting the noise intensity as a policy action, generating the noisy embedding gradient after adding noise, and measuring the effect of the noisy decision through a reward function.

[0015] Furthermore, the reward function is:

[0016]

[0017] in, λ represents the loss function value for the current training epoch. privacy σ represents the privacy cost coefficient. j (t) represents the noise standard deviation of the current dimension.

[0018] Furthermore, the server-local model parameters include the feature encoding network and the prediction head parameters.

[0019] Furthermore, the policy iterative update includes collecting the state-action-reward trajectory during policy execution every K rounds of training, estimating the advantage function using the generalized advantage estimation method, and updating the policy network and value network respectively based on the reinforcement learning policy network algorithm.

[0020] Furthermore, it also includes system initialization before generating embedding vectors, including client-side local embedding model initialization, server-side local model initialization, embedding aggregation and communication mechanism initialization, and reinforcement learning policy network initialization.

[0021] This invention also provides a reinforcement learning-based longitudinal federated privacy protection apparatus, comprising:

[0022] The embedding generation and noise addition module is used by the client to generate embedding vectors based on local data and add noise to the embedding vectors;

[0023] The model training and gradient calculation module is used by the server to train the client's noisy embedding vector and generate the embedding gradient.

[0024] The adaptive perturbation control module is used by the server to intelligently add noise to the embedded gradient based on the reinforcement learning policy network, generating a noisy embedded gradient.

[0025] The local model update module is used by the client to update the client's local embedded model in reverse using the noisy embedding gradient, and the server synchronously updates the server's local model parameters.

[0026] The policy network and value network update module is used by the server to periodically collect reinforcement learning interaction trajectory data and perform policy iterative updates by combining the generalized advantage estimation method.

[0027] The present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described reinforcement learning-based longitudinal federated privacy protection method.

[0028] Beneficial effects:

[0029] This invention introduces a policy network to intelligently select noise-adding operations based on environmental conditions, and combines this with the GAE method to improve learning stability, thereby achieving dynamic adjustment of the gradient perturbation process. Through this method, the server can guide the perturbation policy to continuously optimize during training, maximizing model performance while protecting privacy, and improving the overall security, intelligence, and practicality of the system. Attached Figure Description

[0030] Figure 1 This is a schematic diagram illustrating the working principle and process of an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the electronic device structure according to an embodiment of the present invention. Detailed Implementation

[0032] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0033] The technical terms used in the following embodiments are explained as follows:

[0034] Vertical Federated Learning (VFL): A specific type of federated learning where each participant has different feature spaces, but the user groups overlap. For example, a bank and an e-commerce platform each possess the financial information and consumption behavior data of the same user.

[0035] Embedding Vector: A low-dimensional representation generated by the client based on local feature data, used to collaborate with the server to complete model training without labels.

[0036] Embedding Gradient: The gradient of the embedding vector obtained by backpropagation after the server calculates the loss function based on the labels, which is used to guide the client to update the local model.

[0037] Feature sensitivity: A metric that measures the degree to which a feature affects the model output. It is often used to assess whether the feature needs stronger privacy protection.

[0038] Reinforcement Learning (RL) is a learning method that optimizes strategies through a trial-and-error mechanism. The agent learns the optimal decision-making strategy by interacting with the environment and receiving rewards.

[0039] PPO (Proximal Policy Optimization): An advanced reinforcement learning algorithm with advantages of good stability and high sample efficiency, widely used in the optimization of policy networks.

[0040] GAE (Generalized Advantage Estimation): A method for estimating the advantage function that can effectively balance bias and variance, and improve the stability of the policy update process.

[0041] Server and Client (Active Party & Passive Party): In VFL, the server typically owns the labels and leads the training process; the client does not own the labels, but provides supplementary feature information and assists in training.

[0042] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0043] Figure 1 This is a schematic diagram illustrating the working principle and process of this embodiment, as follows: Figure 1 As shown, the reinforcement learning-based vertical federated adaptive privacy protection method in this embodiment specifically includes:

[0044] S1. Perform system initialization: client local embedding model initialization, server local model initialization, embedding aggregation and communication mechanism initialization, and reinforcement learning policy network (PPO) initialization.

[0045] Specifically, this step is handled by the system initialization module and includes the following steps:

[0046] S1.1: Client-side local embedding model initialization. The client-side local model is responsible for converting local feature data into low-dimensional embedding vectors. Initialization steps include: network structure setting, random parameter initialization, configuring the local optimizer, and setting the learning rate, etc.

[0047] S1.2: Server-side local model initialization. The server-side local model includes a feature encoder and a supervised prediction module, used for label learning in conjunction with client-side embeddings. Its initialization includes: setting the feature encoding network and prediction head structure, parameter initialization, setting the optimizer learning rate and batch size, and setting the loss function as the overall training objective;

[0048] S1.3: Initialization of Embedded Aggregation and Communication Mechanisms. In order to support the fusion of heterogeneous features in vertical federated learning, the following aggregation mechanisms are initialized: initialization of the embedded vector noise addition module and configuration of the communication protocol.

[0049] S1.4: Reinforcement Learning Policy Network Initialization. The server will use reinforcement learning to implement a gradient noise control strategy, initializing the following modules: policy network, value network, initializing the experience buffer for storing states, actions, and rewards, and setting PPO hyperparameters.

[0050] S2. The client generates an embedding vector based on local data and adds noise to the embedding vector.

[0051] Specifically, the noise addition to the embedding vector is based on the feature sensitivity assessment of the noise-adding embedding gradient fed back by the server in the previous round, and the maximum value normalization and differential privacy heuristic noise addition operations are performed to control the risk of information leakage.

[0052] More specifically, this step is handled by the embedding generation and noise addition module, and includes the following steps:

[0053] S2.1: The client will input local features x m Input local embedding model f m Obtain the embedding vector: h m =f m (x m ;θ m );

[0054] S2.2: Since the client cannot access the label, the noisy embedding gradient returned by the server in the previous round is used. To estimate the sensitivity of each dimension to the model output:

[0055] S2.3: Normalize the sensitivity of the current round to the maximum value to form the relative noise-adding weights for each dimension:

[0056] S2.4: Add noise to each dimension of the embedding vector according to the normalization sensitivity, using a Gaussian mechanism:

[0057] S2.5: The client embeds the processed, noisy vector. Send to the server for joint modeling.

[0058] S2.1 Represents the local input characteristics of the client; Represents the embedding vector; θ m Embed network parameters locally;

[0059] S2.3 aims to prevent the gradients of certain features from excessively influencing the noise addition process due to their large scale, where ∈ m To prevent small constants from being divided by zero;

[0060] In S2.4, σ represents the basic noise intensity hyperparameter, which is used to control privacy intensity.

[0061] S3. The server trains the client's noisy embedding vector and generates the embedding gradient.

[0062] Specifically, the generation of embedding gradients involves the server combining the client's noisy embedding vector and the server's local embedding vector to perform forward propagation and calculate the loss function, and then generating embedding gradients through backpropagation, in preparation for the next noisy operation.

[0063] More specifically, this step is handled by the model training and gradient calculation module, and includes the following steps:

[0064] S3.1: The server receives all client embedding vectors and fuses them with the local embedding vector to obtain...

[0065] S3.2: Forward Propagation Calculation Prediction Output

[0066] S3.3: Calculate the loss function:

[0067] S3.4: Backpropagation to calculate the embedded gradient: And send it back to the client.

[0068] S4. The server intelligently adds noise to the embedded gradient based on the reinforcement learning policy network to generate a noisy embedded gradient.

[0069] Specifically, the generation of the noisy embedding gradient involves the server inputting the current state information through a reinforcement learning policy network, outputting the noise intensity as a policy action, generating the noisy embedding gradient after adding noise, and using a reward function to measure the effect of the noise decision, thereby achieving optimal control over the direction and magnitude of the disturbance.

[0070] More specifically, this step is handled by the adaptive disturbance control module and includes the following steps:

[0071] S4.1: Define the state, which represents the environment state in the j-th embedding gradient dimension during the t-th training round of reinforcement learning:

[0072] S4.2: Reinforcement Learning Policy Network π θ (a|s) Output an action α based on the current state. j (t)∈[-Δ max +Δ max ] represents the adjustment amount to the current noise level; noise update σ j (t+1)=clip(σ j (t)+α j (t), σ min , σ max );

[0073] S4.3: Add noise to the embedded gradient:

[0074] S4.4: Designing the Reward Function This reward function reflects a balance between loss minimization and privacy protection;

[0075] σ in S4.1 j (t) is the standard deviation of the current j-th dimension noise, representing the strength of privacy protection; This represents the magnitude of the embedding gradient in the j-th dimension, which measures the degree of influence of that dimension on the loss. This represents the variance of the gradient in the most recent iterations of this dimension, reflecting the degree of change and helping to determine whether more noise should be added to hide sensitive changes; r j (t-1) is the reward value from the previous round, which helps the policy network determine whether the previous noise selection was appropriate. The state definition comprehensively considers multiple factors, improving the policy network's environmental awareness.

[0076] In S4.2, the policy network π θ (a|s) takes state s(t) as input and outputs action a(t). This network is the decision-making part of the entire reinforcement learning system, and its core task is to learn the mapping from state to action; Δ max Indicates the maximum allowable amplitude of the action, controlling the update speed; uses the clip function to prevent noise amplitude from being too large or too small; σ min Indicates the lowest privacy noise level; σ max Indicates maximum noise control;

[0077] In S4.3 This represents the gradient of the j-th dimension of the embedding vector calculated by the server. This indicates a value with a mean of 0 and a variance dimension of 0. Sampling noise in a normal distribution; This represents the gradient after adding noise, which is then sent back to the client for model updates.

[0078] In S4.4 This represents the loss function value for the current training epoch, which can be the average loss of the mini-batch, used to measure the performance of the current model; λ privacy σ represents the privacy cost coefficient, controlling the proportion of noise cost in the reward. Typical values ​​range from 0.05 to 0.5; a larger value indicates a more privacy-preserving strategy. j (t) represents the noise standard deviation of the current dimension. The more noise added, the stronger the privacy, but it may impair model performance. Simply put, if the model loss is low and the noise is small, a positive reward is given; however, if the model noise is too large or the model performance is poor, a negative reward is given. The reward function is used to measure the effect of the noise-adding decision in this round. It is not directly fed back to the policy output of the current round, but is accumulated and used for the periodic update of the policy (in step S6).

[0079] S5. The client uses the noisy embedding gradient to update the client's local embedding model in reverse, and the server synchronously updates the server's local model parameters.

[0080] More specifically, this step is handled by the local model update module and includes the following steps:

[0081] S5.1: The client receives the noisy gradient from the server.

[0082] S5.2: The client uses this noisy gradient for local model updates.

[0083] S5.3: Server updates local model parameters

[0084] eta in S5.2 m This indicates the client's local learning rate;

[0085] eta in S5.3 a This represents the server learning rate.

[0086] S6. The server periodically collects reinforcement learning interaction trajectory data and uses the generalized advantage estimation method to iteratively update the policy.

[0087] Specifically, the policy iterative update includes collecting the state-action-reward trajectory during policy execution every K rounds of training, estimating the advantage function using the generalized advantage estimation method (GAE), and updating the policy network and value network respectively based on the reinforcement learning policy network algorithm (PPO), thereby optimizing the stability and long-term performance of the perturbation policy.

[0088] More specifically, this step is handled by the policy network and value network update module, and includes the following steps:

[0089] S6.1: Collect reinforcement learning interaction trajectory data from multiple S4 steps to form a training sample set:

[0090] S6.2: Calculate the advantage value of each round of samples using the current value network. Used for policy updates. The GAE (Generalized Advantage Estimation) method is employed.

[0091] S6.3: Policy Network Update. Using PPO's Clip Policy Loss Function:

[0092] S6.4: Value Network Update. Update the value network V. ψ Using mean squared error as the objective:

[0093] S6.5: After each PPO update cycle, update the old strategy parameter θ. old ←θ, used for comparison calculation in the next cycle.

[0094] Each sample in S6.1 contains a state s t (Sensitivity features of embedded gradients, historical rewards, etc.); Action a t (Noise variation amplitude selected in this round); Reward r t (A reward function that combines loss and privacy penalty); the next state s t+1 ;

[0095] In S6.2, GAE balances estimation accuracy and variance to improve training stability; γ represents the reward discount function; V(s t ) represents the estimated value function of the current state;

[0096] S6.3 This represents the ratio of the old and new strategies, where ∈ represents the shearing threshold, used to control the stability of the update magnitude.

[0097] S6.4 Updated Value Network V ψ R is used to provide more accurate state value estimates.t This is the cumulative discount reward derived from the GAE derivation in S6.2.

[0098] Figure 2 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device may include:

[0099] The memory 201, the processor 202, and the computer program stored on the memory 201 and capable of running on the processor 202.

[0100] When the processor 202 executes the program, it implements the reinforcement learning-based longitudinal federated adaptive privacy protection method provided in the above embodiments.

[0101] Furthermore, electronic devices also include:

[0102] Communication interface 203 is used for communication between memory 201 and processor 202.

[0103] The memory 201 is used to store computer programs that can run on the processor 202.

[0104] The memory 201 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0105] If the memory 201, processor 202, and communication interface 203 are implemented independently, then the communication interface 203, memory 201, and processor 202 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0106] Optionally, in a specific implementation, if the memory 201, processor 202, and communication interface 203 are integrated on a single chip, then the memory 201, processor 202, and communication interface 203 can communicate with each other through an internal interface.

[0107] The processor 202 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention.

[0108] In this embodiment, the modules are arranged in a hierarchical manner. The system initialization module is responsible for initializing the client-side local embedding model, the server-side local model, the embedding aggregation and communication mechanism, and the reinforcement learning policy network (PPO). The embedding generation and noise addition module generates embedding vectors based on local data and adds noise before sending them to the server. Subsequently, the model training and gradient calculation module receives the embedding vectors, trains them, and calculates the backpropagation gradient. These gradients are then fed into the adaptive perturbation control module, where the policy network determines the degree and method of noise addition. The noise-added perturbation gradient, i.e., the noisy embedding gradient, is fed back to the client and used by the local model update module to update its embedding generation model; simultaneously, the server also updates its feature encoding network synchronously. Every K rounds, the policy network and value network update module collects historical interaction trajectory data, performs iterative policy updates using the GAE method, and returns the optimized policy to the adaptive perturbation control module.

[0109] In summary, this embodiment proposes (1) a reinforcement learning-driven gradient perturbation control strategy. The server designs a reinforcement learning policy network and dynamically selects noise-adding actions based on the training state to achieve intelligent control of gradient perturbation, rather than fixed-ratio perturbation; (2) a feature sensitivity estimation method based on gradient approximation. The client uses the gradient returned by the server in the previous round to estimate the feature dimension sensitivity, normalizes it, and guides differential privacy heuristic noise addition to improve the perturbation's targeting and accuracy; (3) a periodic joint update mechanism for the policy network and value network. The server collects interaction trajectory data every K rounds, calculates the advantage function using the GAE method, and updates the policy and value network parameters through the PPO algorithm to enhance the long-term stability of the perturbation strategy. The overall method significantly improves the modeling quality and collaborative efficiency of longitudinal federated learning while ensuring data privacy.

[0110] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0111] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0112] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A longitudinal federated privacy protection method based on reinforcement learning, characterized in that, Includes the following steps: The client generates an embedding vector based on local data and adds noise to the embedding vector; The server trains the client's noisy embedding vector and generates the embedding gradient; The server intelligently adds noise to the embedded gradient based on a reinforcement learning policy network, generating a noisy embedded gradient. The client uses the noisy embedding gradient to back-update the client's local embedding model, and the server synchronously updates the parameters of the server's local model. The server periodically collects reinforcement learning interaction trajectory data and uses the generalized advantage estimation method to iteratively update the policy.

2. The reinforcement learning-based longitudinal federated privacy protection method according to claim 1, characterized in that, The noise addition to the embedding vector is based on the feature sensitivity evaluation of the noise-adding embedding gradient fed back by the server in the previous round, and the maximum value normalization and differential privacy heuristic noise addition operations are performed.

3. The reinforcement learning-based longitudinal federated privacy protection method according to claim 1, characterized in that, The process of generating the embedding gradient involves the server combining the client's noisy embedding vector and the server's local embedding vector for forward propagation and calculating the loss function, and then generating the embedding gradient through backpropagation.

4. The reinforcement learning-based longitudinal federated privacy protection method according to claim 1, characterized in that, The generation of the noisy embedding gradient involves the server inputting the current state information into a reinforcement learning policy network, outputting the noise intensity as a policy action, generating the noisy embedding gradient after adding noise, and measuring the effect of the noisy decision through a reward function.

5. The reinforcement learning-based longitudinal federated privacy protection method according to claim 1, characterized in that, The reward function is: in, λ represents the loss function value for the current training epoch. privacy σ represents the privacy cost coefficient. j (t) represents the noise standard deviation of the current dimension.

6. The reinforcement learning-based longitudinal federated privacy protection method according to claim 1, characterized in that, The server-local model parameters include the feature encoding network and the prediction head parameters.

7. The reinforcement learning-based longitudinal federated privacy protection method according to claim 1, characterized in that, The policy iterative update includes collecting the state-action-reward trajectory during policy execution every K rounds of training, estimating the advantage function using the generalized advantage estimation method, and updating the policy network and value network respectively based on the reinforcement learning policy network algorithm.

8. The reinforcement learning-based longitudinal federated privacy protection method according to claim 1, characterized in that, It also includes system initialization before generating embedding vectors, including client-side local embedding model initialization, server-side local model initialization, embedding aggregation and communication mechanism initialization, and reinforcement learning policy network initialization.

9. A vertically federated privacy protection device based on reinforcement learning, characterized in that, include: The embedding generation and noise addition module is used by the client to generate embedding vectors based on local data and add noise to the embedding vectors; The model training and gradient calculation module is used by the server to train the client's noisy embedding vector and generate the embedding gradient. The adaptive perturbation control module is used by the server to intelligently add noise to the embedded gradient based on the reinforcement learning policy network, generating a noisy embedded gradient. The local model update module is used by the client to update the client's local embedded model in reverse using the noisy embedding gradient, and the server synchronously updates the server's local model parameters. The policy network and value network update module is used by the server to periodically collect reinforcement learning interaction trajectory data and perform policy iterative updates by combining the generalized advantage estimation method.

10. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the reinforcement learning-based longitudinal federated privacy protection method according to any one of claims 1-8.

Images