Calculation-efficient privacy protection method and system based on data shuffling in sequential decision

Abstract

The invention discloses a privacy protection method and system based on data shuffling in a sequential decision, which are efficient in calculation and are used for solving the dual technical problems of user data privacy leakage and overhigh algorithm calculation complexity when reinforcement learning is adopted to carry out the sequential decision. According to the method, on the basis of introducing a credible shuffling device to guarantee data privacy, a brand-new staged strategy iteration framework which does not need to traverse a strategy space is provided. According to the framework, through an embedded adaptive dichotomy strategy search module and an experimental design method, the exponential traversal search on an active strategy set and the dependence on an external calculation oracle machine are thoroughly avoided. According to the method, the optimal regret boundary equivalent to or even better than centralized differential privacy can be realized in a scene with extremely high privacy protection requirements, meanwhile, the performance of a traditional local differential privacy model is remarkably superior, and the actual deployment feasibility and the calculation efficiency of a privacy protection scheme are greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and privacy computing, in particular to a privacy protection method and system based on data shuffling in sequential decision making, which is particularly suitable for personalized recommendation, medical data analysis, intelligent transportation and other scenarios with high requirements for data privacy and computing efficiency. BACKGROUND

[0002] Reinforcement learning (RL) as a sequential decision-making framework has achieved remarkable success in many practical applications such as medical health, online recommendation, and natural language processing. However, in these applications, the learning agent often needs to learn from individual feedback containing sensitive information. If there is no proper privacy protection mechanism, the user's interaction history may be remembered, making it vulnerable to various privacy attacks.

[0003] Differential privacy (DP) has become the gold standard for solving such privacy problems. In reinforcement learning, existing work mainly focuses on two DP trust models: 1) Central DP / Joint DP (JDP): assumes the existence of a trusted central agent responsible for collecting users' original sensitive data and adding noise after aggregation to protect privacy. This model can achieve a small utility loss (e.g., an additional logarithmic term in the regret bound ) implementation -JDP. However, the complete trust requirement of the central agent is difficult to meet in many practical scenarios. Once the central server is attacked, it will face the risk of large-scale privacy leakage. 2) Local DP (LDP): provides stronger privacy protection. Each user's original data is perturbed locally before being sent to the learning agent. However, this strong privacy protection usually comes at the cost of significant utility loss, for example, there is a large multiplicative or additive term (such as ) related to (total rounds) and (privacy budget) in the regret bound, which may not be acceptable when high privacy protection (i.e. smaller) is required, resulting in severe learning performance degradation.

[0004] Given the pros and cons of the above two models, the former relies on strong trust assumptions and is vulnerable to single-point attacks, while the latter has strong privacy but large utility loss. How to achieve a more delicate trade-off between privacy and utility in RL, i.e., to obtain utility close to the central DP model without relying on a fully trusted central agent, is an important and urgent problem.

[0005] As a new privacy model, the Shuffle Differential Privacy (SDP) model is between the centralized DP and the local DP. In this model, users first locally randomize the data (usually adding less noise than LDP), and then send the encoded data to a "secure shuffler". The shuffler is only responsible for randomly shuffling the order of a batch of user data, and cannot access the original data or the specific content of the encoded data, and then sends the shuffled data to the central analysis server. The introduction of the shuffler provides an additional "privacy blanket", so that each user only needs to inject less noise to hide its information in the group. SDP has been proven to achieve a better privacy-utility trade-off than LDP in supervised learning and multi-armed bandit problems. However, its application in the field of reinforcement learning, especially in combination with the online learning and exploration mechanism of turn-based RL, has not been fully studied. The main challenge is that SDP relies on batch updates and data perturbation, which delays exploitation and complicates exploration.

[0006] Furthermore, existing reinforcement learning methods based on policy elimination face severe computational challenges when dealing with large-scale discrete policy spaces or continuous policy spaces. Evaluating the value of all policies in the active policy set to perform policy elimination can result in an exponential increase in the number of policies, leading to high computational complexity and making it impractical for real-world applications. In addition, during the fine policy search phase, traditional algorithms often need to call external optimization solvers or compute oracles to solve complex minimax optimization problems, which not only increases the design complexity and deployment cost of the system, but also may introduce additional computational overhead and communication delay, further restricting the practical availability of the algorithm. Therefore, there is an urgent need for a reinforcement learning algorithm under the Shuffle Differential Privacy framework that can efficiently handle policy search and policy elimination without relying on external computing oracles. SUMMARY

[0007] The present application aims at the deficiencies of the prior art, and provides a computationally efficient privacy protection method and system based on data shuffling in sequential decision-making, aiming to solve two technical problems existing in the prior art of privacy protection reinforcement learning: first, how to balance privacy protection and learning utility without relying on a fully trusted central server; second, how to overcome the exponential computational complexity caused by the introduction of privacy mechanisms and the adoption of policy elimination ideas, so that the algorithm has the feasibility of practical application. The present application not only realizes an excellent privacy-utility trade-off through the shuffling mechanism, but more importantly, through the introduction of an innovative and adapted dichotomy iterative policy search and built-in experimental design solver, it fundamentally replaces the brute-force enumeration and evaluation of the active policy set in the traditional method, reduces the computational complexity of policy search from exponential to polynomial, thereby solving the computational bottleneck problem of the prior art while ensuring high-strength privacy security and excellent learning performance.

[0008] The purpose of the present application is realized by the following technical scheme: a computationally efficient privacy protection method based on data shuffling in sequential decision-making, comprising the following steps: (1) Set the reinforcement learning environment, define the state space, action space, reward function, and set the total number of sequence interaction users, interact with each user in one scenario round, and interact with a fixed step in each scenario round; divide the total learning round into several stages, each stage contains multiple user interaction rounds; and set the privacy budget parameter and precision level parameter of the shuffling privacy; (2) Each stage includes a rough exploration stage and a fine exploration stage; (2.1) The rough exploration stage is based on the privacy count of the last round of rough exploration, and obtains a state-action-next state tuple set below the access frequency preset threshold, generates a rough transition probability estimate of the absorbing state Markov decision process and the corresponding state transition trust set, reward function estimate; and an iterative policy search PolicySearch module under a dichotomy framework is adopted, which solves the rough exploration strategy that meets the feasible policy constraint under the model uncertainty of the Markov transition model defined by the state transition trust set, obtains the interaction trajectory data of the user in the current stage, and processes it through the first shuffling privacy process to obtain the privacy count; (2.2) The fine exploration phase is based on the privacy counts of coarse exploration and fine exploration in the previous complete round. It inherits the set of state-action-next state tuples below the access frequency preset threshold in the coarse phase, and generates the fine transition probability estimate, the corresponding state transition trust set, and the reward function estimate of the absorbing state Markov decision process. An experimental design method is adopted to solve the fine exploration strategy that satisfies the feasible strategy constraint and has the property of uniform coverage of state-action pairs under the uncertainty of the transition model defined by the state transition trust set. The interaction trajectory data of the user in all interaction rounds in the current phase is obtained and processed through the second shuffle privacy process to obtain the privacy count. (3) After all stages are completed, output the optimal strategy for trust value in the final state transition trust set.

[0009] Furthermore, the privacy count in the coarse exploration phase is the privacy access count value for the state-action-next state tuple; the privacy count in the fine exploration phase includes the privacy access count value for the state-action-next state tuple, the privacy access count value for the state-action pair, and the privacy cumulative reward value for the state-action pair.

[0010] Furthermore, the number of user interaction rounds at each stage The configuration is set to grow exponentially with stage b, and each stage uses only the data collected in the previous stage for environmental Markov model estimation and policy updates.

[0011] Furthermore, both the first and second shuffle privacy processes are privacy treatments designed to adapt to the reinforcement learning method, including: (1) User-side encoding: Each user encodes the reinforcement learning statistics in the locally generated interaction trajectory data, including whether a specific state-action-next state occurs, whether a specific state-action pair occurs and its reward value. The reinforcement learning statistics are used for subsequent coarse and fine transition probability estimation and reward function estimation. Random noise that satisfies local differential privacy is added during the encoding process to generate encoded messages. (2) Data shuffling: A secure shuffler collects coded messages from all users in the same batch, randomly permutes these messages, and sends the permuted messages to the central analyzer; (3) Central analysis and aggregation: The central analyzer aggregates the received shuffled encoded messages and recovers the noisy aggregated values ​​of each statistic; (4) Post-processing: The noisy aggregated value is optimized and corrected to ensure that the obtained privacy count value and reward value meet the preset accuracy and consistency constraints, as well as the non-negativity and probability constraints required for coarse and fine transition probability estimation and reward function estimation, thereby improving the accuracy of subsequent state transition trust set generation.

[0012] Furthermore, the user-end encoding adopts a mechanism based on binary summation, specifically: if the number of users n in the current batch is less than or equal to a preset parameter... Then, the encoded output of user i is its true statistic plus the sum of m independent and identically distributed Bernoulli random variables with parameter 1 / 2, where m is... Otherwise, the encoded output of user i is its true statistical value plus a parameter. Bernoulli random variable.

[0013] Furthermore, the steps of constructing the coarse transition probability estimate and the fine transition probability estimate of the absorbing state Markov decision process (MDP) include: for all current steps, the state-action-next state tuples identified as low-frequency tuples in the hierarchy h are... The transition probability of it in the original MDP Set it to 0, and transfer this portion of the probability mass, along with other discarded transition probability masses, to a virtual absorption state. In other words: in This is a set of low-frequency tuples whose access frequency is below a preset threshold.

[0014] Furthermore, in step (2), the feasible strategy constraint is: the strategy The upper confidence bound for the value function estimation is required to be no less than that of any other strategy. The lower confidence bound of the value function estimate.

[0015] Furthermore, in the aforementioned rough exploration phase, for each level Each state-action pair The PolicySearch module is invoked to select the policy that currently satisfies the feasible policy constraints and has the highest expected access probability at this level and for this state-action pair, based on the current coarse transition probability estimate. In the fine exploration phase, an experimental design method is used to repeatedly invoke the PolicySearch module to generate intermediate policies, which are then aggregated to obtain the fine exploration policy. To achieve the following property under the condition of satisfying feasible policy constraints: any other policy In all state-action pairs Worst-case coverage The sum is less than ,in This indicates the estimation of the true transition probability. Next, strategy Access State-Action Pair Expected number of times It is an indicator function.

[0016] Furthermore, the PolicySearch module, used in the coarse and fine exploration to solve for policies that satisfy feasible policy constraints, employs an iterative method within a bisection framework, specifically including: (1) Using the state transition trust set, privacy reward estimation, and constraint reward as inputs, initialize the lower bound, upper bound, and value threshold of the value function of the policy search; (2) Iteratively call the extended value iterative EVI algorithm based on the hybrid reward function, and adjust the search interval according to the relationship between the policy value and the lower bound and upper bound of the value function; (3) Repeat the iterative process until convergence to satisfy the optimality condition.

[0017] On the other hand, the present invention also provides a computationally efficient privacy-preserving system based on data shuffling in sequential decision-making, the system comprising: (1) Multiple user devices, each equipped with a local encoder for adding noise to the interactive data; (2) A secure shuffler configured to receive and randomly permutate encoded data from a user equipment; (3) A central analysis server configured to receive shuffled data, perform aggregation and post-processing, and execute policy optimization, updates and decisions of reinforcement learning algorithms, wherein the central analysis server has an embedded high-efficiency computing module to implement a binary search policy search module and an experimental design module.

[0018] The beneficial effects of this invention are specifically manifested in the following ways: 1. Superior Privacy-Utility Trade-off: Compared to the Localized Differential Privacy (LDP) model, this invention introduces a shuffler, allowing users to add less noise. This significantly reduces the damage to learning utility while providing meaningful privacy protection (stronger than the trust assumptions of centralized models), achieving regret performance close to that of the Centralized Differential Privacy (JDP) model. Its regret bound is... ,in The terms are additive, which is superior to the multiplicative dependency of LDP. Here The order of the growth rate of the algorithm or function is hidden in the expression and is related to the total number of rounds. Related log-factors.

[0019] 2. No need for a fully trusted central agent: Unlike centralized DP models, this invention does not require a central learning agent to access the user's raw data. The user's raw data is processed only locally, and the shuffler only processes encoded data that it cannot decrypt itself, greatly reducing the need for centralized trust and improving system security.

[0020] 3. Low strategy switching cost: Because the algorithm adopts a phased update strategy, and the number of phases is low... for The level effectively controls the total number of policy switching attempts, which is beneficial in actual deployments and reduces the system maintenance burden.

[0021] 4. "Forgetting" mechanism to prevent noise accumulation: By using only the data of the current stage for model estimation at each stage, the "forgetting" mechanism effectively avoids the problem of continuous noise accumulation caused by privacy perturbations in multiple stages, which helps to maintain the accuracy of the estimation and ensures the stability of long-term learning.

[0022] 5. Rigorous Theoretical Guarantee: This invention relies on the strategy elimination concept and designs novel coarse-grained and fine-grained exploration strategy search methods, providing rigorous regret bound analysis for the proposed SDP-PE algorithm. -SDP's privacy guarantee proves its effectiveness and reliability.

[0023] 6. Innovative and efficient computing module: Bisection Iterative Policy Search: This invention introduces an innovative bisection iterative policy search method, which can efficiently solve for target policies that satisfy feasible policy constraints. This completely solves the exponential computational problem of traversing and evaluating the active policy set in traditional methods, making the algorithm computationally feasible when dealing with large-scale or even continuous policy spaces, and greatly expanding its application scope.

[0024] High-availability experimental design: In the fine-grained exploration phase, this invention no longer relies on complex external optimization solvers. Instead, it uses an embedded experimental design algorithm to iteratively solve a series of intermediate strategies and ultimately aggregates them into the optimal fine-grained exploration strategy. This method achieves both efficient exploration and reaching the optimal regret bound without the need for an external oracle, significantly improving the algorithm's self-sufficiency and practical availability. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a schematic diagram of the system framework of the method of the present invention.

[0026] Figure 2 A flowchart for reinforcement learning to eliminate shuffle privacy based on efficient computational strategies.

[0027] Figure 3 A schematic diagram of the differential privacy mechanism for shuffling. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, specific embodiments of this invention will be described in detail below. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this invention.

[0029] like Figure 1 As shown, this invention proposes a computationally efficient privacy-preserving method based on data shuffling in sequential decision-making for contextual reinforcement learning environments. This method is implemented in a context that includes... One state, One action, scenario length is Learning within Markov Decision Processes (MDPs), totaling... users (or The learning process involves H rounds / situations of interaction, each with H steps. Each step involves observing a state, performing an action, and receiving a reward. The level of the h-th step in the state transition model refers to the h-th level state transition function (there are a total of H levels of state transition functions). The reinforcement learning process is as follows: Figure 2 As shown. The goal is to minimize accumulated regret while satisfying... -SDP Shuffle Differential Privacy. For privacy budget, To reduce the probability of privacy protection failure. The implementation process of this invention is as follows: (1) Set up a reinforcement learning environment, define the state space, action space, reward function, and set the total number of users for sequential interaction. Interact with each user for one scenario round, and interact with a fixed number of steps in each scenario round. Divide the total learning rounds into several stages, each stage containing multiple user interaction rounds. Set the privacy budget parameter and accuracy level parameter for shuffling privacy. Specifically: Input parameter: Total number of rounds General constants failure probability Privacy Budget And a shuffle privacy facilitator (Privatizer).

[0030] initialization: 1. Initial state transition set It contains all possible state transition functions.

[0031] 2. Calculate auxiliary variables .

[0032] 3. Set the accuracy level for Privatizer (This value and) Related, for example ).

[0033] 4. Total number of stages (Ensure the total number of rounds) or Roughly covered ).

[0034] Main loop (for b=1 to B): Determine the batch size for the current stage b. : 1) Usually .

[0035] 2) If the remaining number of rounds is insufficient to support a complete exponential growth batch, then adjust. Use up all Rounds. For example, if Then adjust accordingly. ,in This indicates the rough exploration batch size in stage b. This represents the batch size for the fine exploration in stage b. In the simplified model, coarse exploration and fine exploration are each represented by […]. Here it is assumed that a stage takes approximately 2 hours in total. One sample (rough + detailed).

[0036] (2) Crude Exploration: Based on the privacy count from the previous round of crude exploration, the crude exploration phase obtains a set of state-action-next state tuples below a preset threshold for access frequency, generates a rough transition probability estimate of the absorbing Markov decision process, and the corresponding state transition trust set and reward function estimate; and uses an iterative policy search module under a dichotomy framework to solve for a crude exploration strategy that satisfies feasible policy constraints under the model uncertainty defined by the state transition trust set, obtains the user's interaction trajectory data in the current phase, and processes it through the first shuffle privacy process to obtain the privacy count; as follows: (2.1) Input: Privacy data roughly explored in the previous stage, and a roughly explored subset of the batch size in the current stage. (For example Used for each level, state, and action. The exploration of the right, in total A rough exploration of privacy budgets (From the total budget) (Distributed in the middle), Privatizer.

[0037] (2.2) Initialization: Low-frequency tuple set Rough transition probability estimation (It can be from the previous stage or initially uniform).

[0038] (2.3) Hierarchical exploration (for h=1 to H): 1) For the current level Prepare an empty dataset. .

[0039] 2) Update the low-frequency tuple set ,in It is a constant (such as 6). For the transition state, The state-action-next state transition count is used for the preliminary exploration in the previous stage.

[0040] 3) Update the coarse absorption state MDP estimate and state transition trust set and its reward function : 3.1) For all Constructing intermediate variables reward function , This is a rough exploration phase – action reward counting. The state-action access count is based on the rough exploration in the previous stage; 3.2) Update the coarse absorption state MDP estimate : For all .

[0041] For all .

[0042] Define absorption state The transfer: .

[0043] 3.3) Define the coarse-grained absorber MDP state transition trust set: For all Constructing an intermediate variable set : here Representative at Spatial probability distribution Let be any possible probability distribution among them.

[0044] For all The state transition function in the intermediate variable set is adjusted for absorption state (and) (Same generation method) , Then, a set of state transition trusts is generated based on the Cartesian product combination: .

[0045] 4) For each state-action pair : 4.1) Selection Strategy ,in Is only A virtual reward function where the reward is 1 at the top and 0 at the bottom. This policy search function aims to search for policies that satisfy feasible policy constraints (policy). The upper confidence bound for the value function estimation is required to be no less than that of any other strategy. (the lower confidence bound of the value function estimate), maximizing access under the current estimate. The probability of.

[0046] 4.2) Execution Strategy Several times (e.g.) (times), and add the collected tracks to middle.

[0047] 5) Privacy protection Batch dataset The result is sent to the Privatizer. The Privatizer, following the shuffle binary summation mechanism described later, outputs a privacy-preserving state-action-next state transition count. And state-action access count State-Action Reward Count .

[0048] (2.4) Output: Preliminary exploration of privacy data , .

[0049] (3) Fine Exploration: The fine exploration phase is based on the privacy counts of coarse exploration and fine exploration in the previous complete round. It inherits the set of state-action-next state tuples below the preset threshold of access frequency from the coarse phase, and generates a fine transition probability estimate, a corresponding state transition trust set, and a reward function estimate for the absorbing Markov decision process. An experimental design method is used to solve for a fine exploration strategy that satisfies feasible policy constraints and has uniform state-action pair coverage under the model uncertainty defined by the state transition trust set. The interaction trajectory data of the user in all interaction rounds in the current phase is obtained and processed through a second shuffle privacy process to obtain the privacy count. The details are as follows: (3.1) Input: Privacy data from the previous coarse and fine exploration phases, and another sub-part of the batch size for the current phase. (For example A meticulous exploration of privacy budgets Privatizer.

[0050] (3.2) Initialization: 1) Empty dataset Inheriting the low-frequency tuple from the rough exploration phase .

[0051] 2) Update the fine-grained absorption state MDP estimation based on privacy data from the previous coarse and fine exploration phases. and state transition trust set , and its reward function : 2.1) For all intermediate variables Reward function ; The state-action-next state transition count is used for the previous stage of detailed exploration. This is a count of state-action visits from the previous stage of detailed exploration. This is a cumulative reward for actions taken during the previous stage of detailed exploration. 2.2) Update the fine absorption state MDP estimate : For all .

[0052] For all .

[0053] Define absorption state The transfer: .

[0054] 2.3) Define the trust set for fine-grained absorbing state MDP state transitions: For all Constructing an intermediate variable set : For all The state transition function in the intermediate variable set is adjusted for absorption state (and) (Same generation method) Combining to generate a set of trust for state transitions: .

[0055] (3.3) Experimental Design and Construction An intermediate strategy Solve and execute a fine-grained exploration strategy : 1) For For all Calculate the state-action access rate of the intermediate strategy. ; For all Intermediate reward for experimental design ; Under the constraint of feasible strategies, an intermediate exploration strategy is designed, and the PolicySearch module described above is used for efficient solution. 2) Aggregation An intermediate strategy generates a fine-grained exploration strategy. .

[0056] 3) Execution strategy common In each round, the collected trajectories are added to the dataset. middle.

[0057] (3.4) Privacy processing Batch dataset Send it to the Privatizer. The Privatizer outputs a privacy-preserving state-action-next state transition count. Status-action access count and state-action cumulative rewards .

[0058] (3.5) Output: Detailed exploration of privacy data , .

[0059] Once steps (2) and (3) are completed, the current stage ends and the next stage begins. After all stages are completed, the optimal trust value strategy in the final state transition trust set is output.

[0060] Detailed explanation of PolicySearch: The PolicySearch module uses the classic Extended Value Iteration (EVI) algorithm, employing a binary search approach to search for policies that satisfy feasible policy constraints and maximize the reward function of another objective during both coarse and fine exploration. Specifically, it includes: (1) Initialize the lower bound of the value function, the upper bound of the value function, and the value threshold of the policy search using the state transition trust set, the privacy reward estimate, and another auxiliary reward as input; (2) Iteratively call the extended value iterative EVI algorithm based on the hybrid reward function, and adjust the search interval according to the relationship between the policy value and the lower bound and upper bound of the value function; (3) Repeat the iterative process until convergence to satisfy the optimality condition.

[0061] The specific algorithm design is as follows: for the constrained reward function Target reward function The real state transitions to State transition trust set Output strategy This strategy satisfies the feasible strategy constraint: in For strategy Value function in state transition trust set The greatest possible value For strategy Value function in state transition trust set The minimum possible value. And the output strategy satisfies the optimality condition: Input parameter: reward function State transition trust set ; Initialization: Threshold Lower bound of the value function Upper bound of the value function , ; For i = 0, 1, 2, ... If ,then Return Elseif then Return Else EndIf EndFor like Figure 3 As shown, the Shuffle Privatizer is described in detail: The Privatizer is used to process datasets collected during coarse and fine exploration to output privacy-preserving counts (such as...). ) and rewards ( ). To protect privacy For example: Parameter settings: Assume the current batch has Individual users. Allocate a privacy budget to each counter. (Total Budget) (All counters assigned to a stage), failure probability Calculation parameters .

[0062] User-side encoder (Local Randomizer): 2.1 Each user For events that occur in its trajectory (e.g., in Step execution Later transferred to Its local count is (Value is 0 or 1).

[0063] 2.2 If :user Encoded output ,in iidBernoulli , .

[0064] 2.3 If :user Encoded output ,in Bernoulli .

[0065] Safe shuffler (Secure Shuffler): 3.1 Collect data from the current batch All coded messages for each user .

[0066] 3.2 Perform uniform random permutations on these messages to obtain .

[0067] 3.3 Send the replaced message to the central analyzer.

[0068] Central Analyzer (Analyzer): 4.1 Receive the message after shuffling .

[0069] 4.2 Recover the noisy sum : like .

[0070] If n>τ: .

[0071] 4.3 Post-processing: [This refers to post-processing of data.] (and similar results) Post-processing is performed to ensure that the obtained privacy-preserving count and reward values ​​meet the preset accuracy and consistency constraints, as well as the non-negativity and probability constraints required for coarse and fine transition probability estimation and reward function estimation, thereby improving the accuracy of subsequent state transition trust set generation.

[0072] For example, in solving an optimization problem, minimizing the... disturbance Under the given conditions, the non-negativity and accuracy requirements are met: .

[0073] make To optimize the solution to the problem.

[0074] Final privacy count: , and ensure The high probability.

[0075] Privacy rewards We can directly use the noisy sum of the analyzer output because it does not need to satisfy probability distribution constraints.

[0076] Through the above steps, the SDP-PE algorithm can meet the requirements. Under the premise of -SDP, effective exploration and learning are carried out, and eventually converge to a near-optimal policy. The key lies in the combination of phased exploration and "forgetting" mechanism, as well as a shuffle privacy sansirifier customized for reinforcement learning, which together achieve a good privacy-utility tradeoff.

[0077] The privacy protection method of this invention can be implemented by a computationally efficient privacy-preserving system based on data shuffling in sequential decision-making, which is typically integrated into reinforcement learning applications. This system may include: User devices / clients: Multiple user devices, each running an agent that interacts with the environment (or the environment represented by the user's own behavior) to generate trajectory data. Each user device has a built-in local encoder module for adding noise to relevant statistics in the trajectory data.

[0078] Secure shuffler: A standalone, trusted server or distributed system (trusted only for the shuffling function, not for accessing the data content) that receives coded data from a batch of user devices and performs random permutation operations.

[0079] Central Analysis Server / Learning Agent: Receives permuted data from the shuffler, executes the functions of the analyzer G (aggregation, denoising, post-processing), and runs the main logic of the Shuffle Differential Privacy Policy Elimination (SDP-PE) algorithm, including policy selection for coarse exploration and fine exploration, policy optimization and decision-making, as well as maintaining the MDP model estimation and state transition trust set. The server ultimately outputs the learned policy.

[0080] High-efficiency computing module (embedded in the central analysis server): This module is a core component of the central analysis server. It is responsible for efficiently solving the policy optimization problem of coarse and fine policies through the embedded binary iterative policy search mechanism, and using experimental design algorithms to iteratively solve and aggregate policies, thereby achieving high-efficiency computing directly in the policy selection and optimization stages, completely avoiding dependence on external computing oracles.

[0081] These components are connected via a network. User devices send coded data to the shuffler, which then sends the processed data to the central analytics server. Based on the learning results, the central analytics server may distribute new strategies to the user devices for the next round of interaction (although this is not discussed in this paper). In each user model, each user interacts once.

[0082] Specific application scenario example: Personalized medical decision-making system To help users better understand the technical solution of this invention, especially its advantages in computational efficiency, the following explanation will use "personalized treatment decision system for chronic diseases (such as type II diabetes)" as an example.

[0083] Technical issues: Doctors need to develop long-term, personalized treatment plans for a large number of diabetic patients to optimize the control of key health indicators (such as glycated hemoglobin A1c levels). This problem faces a double challenge: First, patient medical records are extremely sensitive personal data and must be used under strict confidentiality. Second, due to the vast number of individual patients' physical states (state space) and available treatment methods (action space, such as combinations of different medications, dosages, and dietary recommendations), the number of possible treatment strategies is astronomical. Learning the optimal strategy for the entire patient population presents a huge computational challenge that traditional methods cannot handle. In this scenario, the parameters and advantages of the method described in this invention can be summarized as follows: User and Context Interaction: Each patient participating in the treatment plan learning process completes an "interaction round" or "contextual interaction" over a period of time (e.g., one year). The system learns from data from thousands of patients (K users).

[0084] State space (X): Represents the patient's current clinical signs and biomarkers. For example, state can be composed of a series of indicators such as (glycated hemoglobin level, body mass index BMI, age, history of complications). This is a high-dimensional and complex space.

[0085] Action space (A): Represents the medical interventions a physician can take. For example, actions can be combinations of specific treatment options such as (prescribing 50mg of drug A, prescribing 100mg of drug B, recommending a low-carbohydrate diet, or suggesting 150 minutes of exercise per week). This decision space is enormous.

[0086] Reward function (R): Represents the clinical outcome of a treatment intervention and is core privacy data that needs to be protected. For example, if a patient's glycated hemoglobin level decreases by more than 1% within a quarter, they receive a high positive reward; if a serious adverse drug reaction occurs, they receive a large negative reward.

[0087] Privacy Protection: A patient's complete treatment record (i.e., the sequence of "state-action-reward") is private information strictly protected by law. The shuffling privacy mechanism of this invention ensures that the central server can only obtain anonymous aggregated statistical data (e.g., "N patients in a certain state received an intervention, and the average reward was R"), and cannot trace back to the medical records of any specific patient.

[0088] The efficiency of computation is reflected in: The dilemma of traditional methods: If traditional strategy elimination methods are used, the system needs to enumerate and evaluate a massive number of long-term treatment strategies (e.g., "Strategy 1: Drug A is the first choice for all overweight patients; Strategy 2: Drugs B and C are used in combination for patients with a history of complications..."), which is computationally infeasible.

[0089] The solution of this invention: This invention does not require enumerating these strategies. Through an embedded binary search strategy module, it can efficiently and directly solve for a "currently optimal exploratory treatment strategy" within an uncertain "model trust set" constructed from privacy data. For example, the system can quickly find a clinical trial strategy that best explores the "efficacy of new drug X for patients with a specific genotype." This solution process has polynomial time complexity, rather than exponential time complexity, thus enabling privacy-preserving reinforcement learning in complex medical scenarios.

[0090] In this way, the present invention not only solves the problem of user data privacy protection, but more importantly, it solves the practicality problem of privacy protection schemes in complex decision-making scenarios through innovative computing methods.

[0091] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A computationally efficient privacy-preserving method based on data shuffling in sequential decision-making, characterized in that, The method includes the following steps: (1) Set up a reinforcement learning environment, define the state space, action space, reward function and set the total number of users for sequential interaction, interact with each user for one scenario round, and interact with fixed steps in each scenario round; divide the total learning rounds into several stages, each stage contains multiple user interaction rounds; and set the privacy budget parameters and accuracy level parameters for shuffling privacy. (2) Each stage consists of two parts: a rough exploration stage and a detailed exploration stage; (2.1) In the coarse exploration phase, based on the privacy count of the previous round of coarse exploration, a set of state-action-next state tuples with access frequency below a preset threshold is obtained, and a coarse transition probability estimate of the absorbing Markov decision process and the corresponding state transition trust set and reward function estimate are generated. An iterative policy search module under a dichotomy framework is used to solve the coarse exploration strategy that satisfies the feasible policy constraint under the model uncertainty of the Markov transition model defined by the state transition trust set, obtain the user's interaction trajectory data in the current phase, and process it through the first shuffle privacy process to obtain the privacy count. (2.2) The fine exploration phase is based on the privacy counts of coarse exploration and fine exploration in the previous complete round. It inherits the set of state-action-next state tuples below the access frequency preset threshold in the coarse phase, and generates the fine transition probability estimate, the corresponding state transition trust set, and the reward function estimate of the absorbing state Markov decision process. An experimental design method is adopted to solve the fine exploration strategy that satisfies the feasible strategy constraint and has the property of uniform coverage of state-action pairs under the uncertainty of the transition model defined by the state transition trust set. The interaction trajectory data of the user in all interaction rounds in the current phase is obtained and processed through the second shuffle privacy process to obtain the privacy count. (3) After all stages are completed, output the optimal strategy for trust value in the final state transition trust set.

2. The computationally efficient privacy-preserving method based on data shuffling in sequential decision-making according to claim 1, characterized in that, The privacy count in the coarse exploration phase is the privacy access count for the state-action-next state tuple; the privacy count in the fine exploration phase includes the privacy access count for the state-action-next state tuple, the privacy access count for the state-action pair, and the privacy cumulative reward for the state-action pair.

3. The computationally efficient privacy-preserving method based on data shuffling in sequential decision-making according to claim 1, characterized in that, Number of user interaction rounds in each stage The configuration is set to grow exponentially with stage b, and each stage uses only the data collected in the previous stage for environmental Markov model estimation and policy updates.

4. The computationally efficient privacy-preserving method based on data shuffling in sequential decision-making according to claim 1, characterized in that, Both the first and second shuffle privacy processes are privacy treatments designed to adapt to the reinforcement learning method, including: (1) User-side encoding: Each user encodes the reinforcement learning statistics in the locally generated interaction trajectory data, including whether a specific state-action-next state occurs, whether a specific state-action pair occurs and its reward value. The reinforcement learning statistics are used for subsequent coarse and fine transition probability estimation and reward function estimation. Random noise that satisfies local differential privacy is added during the encoding process to generate encoded messages. (2) Data shuffling: A secure shuffler collects coded messages from all users in the same batch, randomly permutes these messages, and sends the permuted messages to the central analyzer; (3) Central analysis and aggregation: The central analyzer aggregates the received shuffled encoded messages and recovers the noisy aggregated values ​​of each statistic; (4) Post-processing: The noisy aggregated value is optimized and corrected to ensure that the obtained privacy count value and reward value meet the preset accuracy and consistency constraints, as well as the non-negativity and probability constraints required for coarse and fine transition probability estimation and reward function estimation, thereby improving the accuracy of subsequent state transition trust set generation.

5. The computationally efficient privacy-preserving method based on data shuffling in sequential decision-making according to claim 4, characterized in that, The user-side encoding adopts a mechanism based on binary summation, specifically: if the number of users n in the current batch is less than or equal to a preset parameter... Then, the encoded output of user i is its true statistic plus the sum of m independent and identically distributed Bernoulli random variables with parameter 1 / 2, where m is... Otherwise, the encoded output of user i is its true statistical value plus a parameter. Bernoulli random variable.

6. The computationally efficient privacy-preserving method based on data shuffling in sequential decision-making according to claim 1, characterized in that, The steps for constructing the coarse and fine transition probability estimates of the absorber-state Markov decision process (MDP) include: for all current steps, the state-action-next-state tuples identified as low-frequency tuples in the hierarchy h are... The transition probability of it in the original MDP Set it to 0, and transfer this portion of the probability mass, along with other discarded transition probability masses, to a virtual absorption state. In other words: ;in This is a set of low-frequency tuples whose access frequency is below a preset threshold.

7. The computationally efficient privacy-preserving method based on data shuffling in sequential decision-making according to claim 1, characterized in that, In step (2), the feasible strategy constraint is: the strategy The upper confidence bound for the value function estimation is required to be no less than that of any other strategy. The lower confidence bound of the value function estimate.

8. The computationally efficient privacy-preserving method based on data shuffling in sequential decision-making according to claim 1, characterized in that, In the aforementioned rough exploration phase, for each level Each state-action pair The PolicySearch module is invoked to select the policy that currently satisfies the feasible policy constraints and has the highest expected access probability at this level and on this state-action pair based on the current coarse transition probability estimate. In the refined exploration phase, an experimental design method is used to repeatedly call the PolicySearch module to generate intermediate policies, which are then aggregated to obtain the refined exploration policy. To achieve the following property under the condition of satisfying feasible policy constraints: any other policy In all state-action pairs Worst-case coverage The sum is less than ,in This indicates the estimation of the true transition probability. Next, strategy Access State-Action Pair Expected number of times It is an indicator function.

9. The computationally efficient privacy-preserving method based on data shuffling in sequential decision-making according to claim 1, characterized in that, The PolicySearch module, used to solve for policies that satisfy feasible policy constraints in the coarse and fine exploration processes, employs an iterative method within a bisection framework, specifically including: (1) Using the state transition trust set, privacy reward estimation, and constraint reward as inputs, initialize the lower bound, upper bound, and value threshold of the value function of the policy search; (2) Iteratively call the extended value iterative EVI algorithm based on the hybrid reward function, and adjust the search interval according to the relationship between the policy value and the lower bound and upper bound of the value function; (3) Repeat the iterative process until convergence to satisfy the optimality condition.

10. A computationally efficient privacy-preserving system based on data shuffling in sequential decision-making that implements the method of any one of claims 1-9, characterized in that, The system includes: (1) Multiple user devices, each equipped with a local encoder for adding noise to the interactive data; (2) A secure shuffler configured to receive and randomly permutate encoded data from a user equipment; (3) A central analysis server configured to receive shuffled data, perform aggregation and post-processing, and execute policy optimization, updates and decisions of reinforcement learning algorithms, wherein the central analysis server has an embedded high-efficiency computing module to implement a binary search policy search module and an experimental design module.

Images