Data protection method and system based on deep reinforcement learning and differential privacy

Abstract

The invention is suitable for the technical field of data processing, and provides a data protection method and system based on deep reinforcement learning and differential privacy, and the method comprises the steps: firstly obtaining training data set information and to-be-optimized model information, then generating optimized data set information based on a differential privacy mechanism, and finally, generating optimized data set information based on the optimized data set information. And target optimization model information is effectively generated. According to the method, a differential privacy technology and deep reinforcement learning can be combined, and high performance of the model is kept on the premise of protecting user data privacy; the relation between privacy protection and model performance is automatically balanced through fine privacy budget management and a noise adjustment strategy; in addition, the privacy budget distribution of each training step is adaptively adjusted according to the actual condition of model training, so that the privacy protection and the model performance are optimally balanced, and the method is suitable for various deep reinforcement learning application scenes needing to protect the user privacy and has a relatively strong prospect.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and specifically, to a data protection method and system based on deep reinforcement learning and differential privacy. Background Art

[0002] Deep Q-Network (DQN) is a method that combines deep learning and reinforcement learning. DQN is one of the representative algorithms of Deep Reinforcement Learning (DRL). With the rapid development of artificial intelligence technology, DQN has been more widely used because of its powerful capabilities in fields such as games or robot control.

[0003] At present, DRL algorithms usually require a large amount of training data, but this data often contains sensitive personal information, such as user behavior trajectories and preferences, which makes it impossible to effectively train the DRL model while protecting user privacy. The privacy protection effect is poor and needs further improvement. Summary of the invention

[0004] Based on this, the embodiments of the present application provide a data protection method and system based on deep reinforcement learning and differential privacy to solve the problem of poor privacy protection effect in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a data protection method based on deep reinforcement learning and differential privacy, the method comprising:

[0006] Acquire training data set information and model information to be optimized, wherein the model information to be optimized is used to describe the deep reinforcement learning model information to be optimized;

[0007] Based on a preset differential privacy mechanism, the training data set information is desensitized to generate optimized data set information;

[0008] Based on the optimization data set information, the model information to be optimized is trained to generate target optimization model information.

[0009] Compared with the prior art, the beneficial effect is that the data protection method based on deep reinforcement learning and differential privacy provided in the embodiment of the present application, the terminal device can first obtain the training data set information and the model information to be optimized, and then desensitize the training data set information based on the preset differential privacy mechanism, and quickly generate the optimized data set information, and finally, based on the optimized data set information, train the model information to be optimized, and effectively generate the target optimization model information, thereby combining the differential privacy technology with the DQN algorithm in deep reinforcement learning, significantly enhancing the privacy protection effect, and to a certain extent solving the problem of poor current privacy protection effect.

[0010] In a second aspect, an embodiment of the present application provides a data protection system based on deep reinforcement learning and differential privacy, the system comprising:

[0011] Training data set information acquisition module: used to acquire training data set information and model information to be optimized, wherein the model information to be optimized is used to describe the deep reinforcement learning model information to be optimized;

[0012] An optimized data set information generation module is used to perform desensitization processing on the training data set information based on a preset differential privacy mechanism to generate optimized data set information;

[0013] Target optimization model information generation module: used to train the model information to be optimized based on the optimization data set information to generate target optimization model information.

[0014] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect described above when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect described above are implemented.

[0016] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0018] Figure 1 It is a flowchart of a data protection method provided by an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a training process provided by an embodiment of the present application;

[0020] Figure 3 It is a flowchart after step S300 in the data protection method provided in one embodiment of the present application;

[0021] Figure 4 is a module block diagram of a data protection system provided by an embodiment of the present application;

[0022] Figure 5 It is a schematic diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0023] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0024] In the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0026] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0027] See also Figure 1 , Figure 1: is a flow chart of a data protection method based on deep reinforcement learning and differential privacy provided in an embodiment of the present application. In this embodiment, the execution subject of the data protection method is a terminal device. It is understandable that the types of terminal devices include but are not limited to mobile phones, tablet computers, laptops, ultra-mobile personal computers (Ultra-Mobile Personal Computer, UMPC), netbooks, personal digital assistants (Personal Digital Assistant, PDA), etc., and the embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0028] See also Figure 1 The data protection method provided in the embodiment of the present application includes but is not limited to the following steps:

[0029] In S100, training data set information and model information to be optimized are obtained.

[0030] Specifically, the terminal device can obtain the training data set information and the model information to be optimized, wherein the model information to be optimized is used to describe the deep reinforcement learning model information to be optimized; the training data set information is used to train the deep reinforcement learning model information. Without loss of generality, please refer to Figure 2 , the DQN algorithm can use the Experience Replay mechanism to store and sample training data.

[0031] In S200, based on a preset differential privacy mechanism, the training data set information is desensitized to generate optimized data set information.

[0032] Specifically, after the terminal device obtains the training data set information and the model information to be optimized, the terminal device can desensitize the training data set information based on the preset differential privacy mechanism to generate optimized data set information, thereby adding noise to sensitive personal information.

[0033] In some possible implementations, in order to add noise to sensitive personal information, before step S100, the method further includes but is not limited to the following steps:

[0034] In S101, the training data set information is classified and processed to generate target data set information and remaining data set information.

[0035] Specifically, the terminal device can use keyword classification algorithms such as TF-IDF (Term Frequency-Inverse Document Frequency) or TextRank to classify the training data set information to generate target data set information and remaining data set information, where the target data set information includes state information, action information and reward information; the remaining data set information includes data information of the training data set information other than the target data set information.

[0036] Accordingly, the above step S200 includes but is not limited to the following steps:

[0037] In S210 , preset Gaussian noise is added to the target data set information to generate optimized data set information.

[0038] Specifically, after the terminal device generates the target data set information and the remaining data set information, the terminal device can add preset Gaussian noise to the target data set information to generate optimized data set information, thereby achieving the addition of noise that meets the definition of differential privacy to the sampled training data (i.e., state information, action information, and reward information) during the experience replay process, thereby protecting the privacy of user data by adding noise, where Gaussian noise meets the definition of differential privacy.

[0039] In S300, based on the optimization data set information, the model information to be optimized is trained to generate target optimization model information.

[0040] Specifically, after the terminal device generates the optimized data set information, the terminal device can train the optimization model information based on the optimized data set information, and effectively generate the target optimization model information, thereby effectively training the DRL model and improving the privacy protection effect while protecting the user's privacy.

[0041] Without loss of generality, since the Laplace distribution is more sensitive to extreme values, the use of Laplace noise can help build a statistical model that is more resistant to outliers. Therefore, when the processed data is highly sensitive, the terminal device can use Laplace noise to process the data, thereby replacing the processing steps using the differential privacy algorithm with the processing steps using Laplace noise.

[0042] In some possible implementations, the terminal device may also add Gaussian noise to the gradient when calculating the gradient corresponding to the objective function of the target optimization model information. The standard deviation of the Gaussian noise may be adjusted according to the privacy budget and gradient sensitivity, thereby adding noise during the iterative update process of the model and updating the model parameters by minimizing the objective function.

[0043] In some possible implementations, in order to effectively manage the privacy budget, before step S300, the method further includes but is not limited to the following steps:

[0044] In S301, preset global privacy budget information is obtained.

[0045] Specifically, the terminal device can obtain the preset global privacy budget information, so as to pre-set a global privacy budget for the entire training process to control the overall privacy leakage risk.

[0046] Accordingly, the above step S300 includes but is not limited to the following steps:

[0047] In S310, the model information to be optimized is trained according to the optimized data set information and the remaining data set information to generate target optimized model information.

[0048] Exemplarily, the terminal device can divide the optimized data set information and the remaining data set information to generate a training set and a test set, and then use the training set to train the model information to be optimized, and use the test set to test the model information to be optimized to generate target optimization model information, thereby recording the privacy loss of each training step, ensuring that the accumulation of privacy loss does not exceed the global privacy budget during the entire training process, and balancing the relationship between privacy protection and model performance, wherein the cumulative privacy loss value of the training process is less than or equal to the global privacy budget information.

[0049] In some possible implementations, during the training process, the terminal device can dynamically allocate the global privacy budget to each training step according to the actual situation during the training process; illustratively, the terminal device can obtain the loss value of the model. If the loss value is still in a rapid decline stage, it is determined that the model is currently in the early stage of training. Since the current model is highly sensitive to data, the terminal device can allocate a larger privacy budget; if the decline in the loss value tends to be flat, it is determined that the model is currently in the late stage of training. Since the current model is less sensitive to data, the terminal device can allocate a smaller privacy budget, thereby realizing dynamic allocation of the privacy budget and achieving the best balance between privacy protection and model performance.

[0050] In some possible implementations, in order to further optimize the performance of the DRL model, after step S310, the method further includes but is not limited to the following steps:

[0051] In S311, based on a preset regularization term, constraint processing is performed on the objective function of the target optimization model information to generate target improved model information.

[0052] Specifically, the terminal device can constrain the objective function of the target optimization model information based on the preset regularization term, introduce the regularization term into the objective function, and generate the target improved model information, thereby limiting the range of model parameters, improving the generalization ability of the model, and reducing the adverse effects of noise.

[0053] For further information on managing privacy budgets, see Figure 3 After step S300, the method further includes but is not limited to the following steps:

[0054] In S400, training evaluation information of target optimization model information is obtained.

[0055] Specifically, the terminal device can obtain training evaluation information of the target optimization model information, wherein the training evaluation information includes accuracy information and / or loss function value information; the accuracy information is used to describe the accuracy of the target optimization model information; the loss function value information is used to describe the loss function value of the target optimization model information.

[0056] In S410, the training evaluation information is compared with a preset training evaluation threshold.

[0057] Specifically, after the terminal device obtains the training evaluation information, the terminal device may compare the training evaluation information with a preset training evaluation threshold, wherein the training evaluation threshold includes an accuracy threshold and / or a loss function threshold.

[0058] In S420, if the accuracy information is less than the accuracy threshold, or the loss function value information is greater than the loss function threshold, the standard deviation information of the Gaussian noise is obtained.

[0059] Specifically, if the accuracy information is less than the accuracy threshold, or the loss function value information is greater than the loss function threshold, the terminal device can obtain the standard deviation information of the Gaussian noise, where the standard deviation of the noise can be adjusted according to the privacy budget and data sensitivity.

[0060] In S430, the difference between the preset adjustment values ​​is subtracted from the standard deviation information to generate optimized numerical value information.

[0061] Specifically, after the terminal device obtains the standard deviation information, the terminal device can generate optimized numerical information based on the difference between the standard deviation information and the preset adjustment value, so as to automatically and appropriately reduce the standard deviation of the noise when the model performance degrades, wherein the optimized numerical information is used to describe the difference between the standard deviation information and the preset adjustment value.

[0062] In S440, the standard deviation information is replaced with the optimized numerical value information.

[0063] Specifically, after the terminal device generates the optimized numerical information, the terminal device can replace the standard deviation information with the optimized numerical information, thereby adaptively adjusting the standard deviation of the noise according to the actual situation of the model training to improve the learning ability of the model.

[0064] In some possible implementations, the terminal device can also determine the model that achieves the best balance between privacy protection and model performance by evaluating the model performance under different privacy budgets. For example, when the DQN algorithm overestimates the Q value, the terminal device can use the Double DQN algorithm to replace the DQN algorithm, thereby improving the stability and accuracy of the model.

[0065] For example, when a more accurate estimation of state values ​​and action advantages is required, the terminal device can use the Dueling DQN algorithm to replace the DQN algorithm, thereby improving the decision-making ability of the model in complex environments; when the data in the experience replay is unevenly distributed, the terminal device can use the Prioritized Experience Replay (PER) mechanism to replace the DQN algorithm, thereby improving the learning efficiency and convergence speed of the model.

[0066] In some possible implementations, in order to alleviate the gradient vanishing problem and improve the learning ability of the model, when it is necessary to process complex state representation or a deeper network structure is required, the terminal device can introduce a deep residual network into the DRL model; when it is necessary to focus on key information in the state, the terminal device can introduce an attention mechanism into the DRL model, thereby improving the model's representation and decision-making capabilities.

[0067] In some possible implementations, the terminal device can also evaluate the privacy protection effect. For example, based on the theory of differential privacy, the terminal device can systematically analyze the privacy protection effect of the model under different privacy budgets, and then evaluate the risk probability of the attacker leaking the user's privacy information after obtaining the model output results through preset simulated attack experiments. At the same time, in actual applications, actively collect user feedback and their behavior data to comprehensively evaluate the actual protection effect of the model in a real environment.

[0068] The implementation principle of the data protection method based on deep reinforcement learning and differential privacy in the embodiment of the present application is as follows: the terminal device can first obtain the training data set information and the model information to be optimized, and then desensitize the training data set information based on the preset differential privacy mechanism, and quickly generate the optimization data set information; finally, based on the optimization data set information, the model information to be optimized is trained and processed, and the target optimization model information is effectively generated, thereby combining the differential privacy technology with the DQN algorithm in deep reinforcement learning, and effectively achieving the privacy protection effect, which not only maintains the high performance of the model under the premise of protecting the privacy of user data, but also automatically balances the relationship between privacy protection and model performance, and achieves the best balance between privacy protection and model performance.

[0069] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0070] The embodiment of the present application also provides a data protection system based on deep reinforcement learning and differential privacy. For ease of explanation, only the parts related to the present application are shown, such as Figure 4 As shown, the system 40 includes:

[0071] The training data set information acquisition module 41 is used to acquire the training data set information and the model information to be optimized, wherein the model information to be optimized is used to describe the deep reinforcement learning model information to be optimized;

[0072] The optimized data set information generating module 42 is used to perform desensitization processing on the training data set information based on a preset differential privacy mechanism to generate optimized data set information;

[0073] The target optimization model information generation module 43 is used to train the model information to be optimized based on the optimization data set information to generate the target optimization model information.

[0074] Optionally, the system 40 further includes:

[0075] Target data set information generation module: used to classify the training data set information and generate target data set information and remaining data set information, wherein the target data set information includes state information, action information and reward information, and the remaining data set information includes data information of the training data set information other than the target data set information;

[0076] Accordingly, the optimized data set information generating module 42 includes:

[0077] Optimized dataset information generation submodule: used to add preset Gaussian noise to the target dataset information to generate optimized dataset information, where the Gaussian noise meets the definition of differential privacy.

[0078] Optionally, the system 40 further includes:

[0079] Global privacy budget information acquisition module: used to obtain preset global privacy budget information;

[0080] Optionally, the target optimization model information generating module 43 includes:

[0081] Target optimization model information generation submodule: used to train the model information to be optimized according to the optimization data set information and the remaining data set information to generate target optimization model information, wherein the cumulative privacy loss value of the training process is less than or equal to the global privacy budget information.

[0082] Optionally, the system 40 further includes:

[0083] A training evaluation information acquisition module: used to acquire training evaluation information of target optimization model information, wherein the training evaluation information includes accuracy information and / or loss function value information;

[0084] A training evaluation information comparison module: used to compare the training evaluation information with a preset training evaluation threshold, wherein the training evaluation threshold includes an accuracy threshold and / or a loss function threshold;

[0085] Standard deviation information acquisition module: used to obtain the standard deviation information of Gaussian noise if the accuracy information is less than the accuracy threshold, or the loss function value information is greater than the loss function threshold;

[0086] Optimized numerical information generation module: used to generate optimized numerical information according to the difference between the preset adjustment value and the standard deviation information;

[0087] Standard deviation information replacement module: used to replace standard deviation information with optimized numerical information.

[0088] Optionally, the system 40 further includes:

[0089] Target improvement model information generation module: used to constrain the objective function of the target optimization model information based on the preset regularization term to generate the target improvement model information.

[0090] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0091] The present application also provides a terminal device, such as Figure 5 As shown, the terminal device 50 of this embodiment includes: a processor 51, a memory 52, and a computer program 53 stored in the memory 52 and executable on the processor 51. When the processor 51 executes the computer program 53, the steps in the above data protection method embodiment are implemented, such as Figure 1 Steps S100 to S300 shown; or, when the processor 51 executes the computer program 53, the functions of each module in the above device are realized, for example Figure 4 The functions of modules 41 to 43 are shown.

[0092] The terminal device 50 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server, and the terminal device 50 includes but is not limited to a processor 51 and a memory 52. ​​Those skilled in the art will appreciate that Figure 5 It is only an example of the terminal device 50 and does not constitute a limitation of the terminal device 50. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 50 may also include input and output devices, network access devices, buses, etc.

[0093] Among them, the processor 51 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.; the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0094] The memory 52 may be an internal storage unit of the terminal device 50, such as a hard disk or memory of the terminal device 50, or the memory 52 may be an external storage device of the terminal device 50, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device 50; further, the memory 52 may also include both an internal storage unit and an external storage device of the terminal device 50, and the memory 52 may also store a computer program 53 and other programs and data required by the terminal device 50, and the memory 52 may also be used to temporarily store data that has been output or is to be output.

[0095] One embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of each of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.; the computer-readable medium may include: any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0096] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, all equivalent changes made according to the methods, principles, and structures of the present application should be included in the protection scope of the present application.

Claims

1. A data protection method based on deep reinforcement learning and differential privacy, characterized in that: The method comprises: Acquire training data set information and model information to be optimized, wherein the model information to be optimized is used to describe the deep reinforcement learning model information to be optimized; Based on a preset differential privacy mechanism, the training data set information is desensitized to generate optimized data set information; Based on the optimization data set information, the model information to be optimized is trained to generate target optimization model information.

2. The method according to claim 1, characterized in that Before performing desensitization processing on the training data set information based on the preset differential privacy mechanism to generate optimized data set information, the method further includes: Classify the training data set information to generate target data set information and remaining data set information, wherein the target data set information includes state information, action information and reward information, and the remaining data set information includes data information of the training data set information other than the target data set information; Accordingly, the training data set information is desensitized based on the preset differential privacy mechanism to generate optimized data set information, including: Add preset Gaussian noise to the target data set information to generate optimized data set information, wherein the Gaussian noise satisfies the definition of differential privacy.

3. The method according to claim 2, characterized in that Before the training process is performed on the model information to be optimized based on the optimization data set information to generate the target optimization model information, the method further includes: Get the preset global privacy budget information; The step of training the model information to be optimized based on the optimization data set information to generate target optimization model information includes: The model information to be optimized is trained according to the optimized data set information and the remaining data set information to generate target optimized model information, wherein the cumulative privacy loss value of the training process is less than or equal to the global privacy budget information.

4. The method according to claim 3, characterized in that After training the model information to be optimized based on the optimization data set information to generate target optimization model information, the method further includes: Acquire training evaluation information of the target optimization model information, wherein the training evaluation information includes accuracy information and / or loss function value information; Comparing the training evaluation information with a preset training evaluation threshold, wherein the training evaluation threshold includes an accuracy threshold and / or a loss function threshold; If the accuracy information is less than the accuracy threshold, or the loss function value information is greater than the loss function threshold, then obtaining the standard deviation information of the Gaussian noise; Generate optimized numerical value information according to the difference between the standard deviation information and the preset adjustment value; The standard deviation information is replaced by the optimized numerical value information.

5. The method according to claim 3, characterized in that: After training the model information to be optimized according to the optimized data set information and the remaining data set information to generate target optimized model information, the method further includes: Based on the preset regularization term, the objective function of the target optimization model information is constrained to generate target improvement model information.

6. A data protection system based on deep reinforcement learning and differential privacy, characterized in that: The system comprises: Training data set information acquisition module: used to acquire training data set information and model information to be optimized, wherein the model information to be optimized is used to describe the deep reinforcement learning model information to be optimized; An optimized data set information generation module is used to perform desensitization processing on the training data set information based on a preset differential privacy mechanism to generate optimized data set information; Target optimization model information generation module: used to train the model information to be optimized based on the optimization data set information to generate target optimization model information.

7. The system according to claim 6, characterized in that The system further comprises: A target data set information generation module is used to classify the training data set information to generate target data set information and remaining data set information, wherein the target data set information includes state information, action information and reward information, and the remaining data set information includes data information of the training data set information other than the target data set information; Accordingly, the optimized data set information generation module includes: The optimized data set information generation submodule is used to add preset Gaussian noise to the target data set information to generate optimized data set information, wherein the Gaussian noise satisfies the definition of differential privacy.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images