Deep reinforcement learning oriented training data set copyright auditing method and device

Abstract

The application discloses a kind of training data set copyright audit method and device for deep reinforcement learning.The critic model is trained based on time difference learning mode and is used as audit benchmark.Secondly, the application initializes a batch of deep reinforcement learning models using different model parameters, and trains these models on the data set to form a shadow model set. Then, the application inputs the state signal of the data set into the shadow model and the model to be audited respectively, and collects the action signal output by the shadow model and the model to be audited. Then, the application inputs the state-action signal of the shadow model and the model to be audited into the critic model respectively, and records the cumulative reward signal output by the critic model. Finally, the application obtains the final copyright audit result based on the cumulative reward signal of the shadow model and the model to be audited and with the help of hypothesis testing method.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security, and in particular relates to a method and apparatus for copyright auditing of training datasets for deep reinforcement learning. Background Technology

[0002] Deep reinforcement learning has been successfully applied to many complex decision-making tasks, such as autonomous driving, robot control, financial trading, and esports. Deep reinforcement learning models can learn from data records of interactions between experts, human controllers, and even stochastic policies that meet system constraints and the environment. An increasing number of companies and research institutions are releasing high-quality datasets to promote the development of deep reinforcement learning models. However, data thieves may copy these datasets, train proprietary reinforcement learning models, and profit from closed-source model services, violating open-source license terms. Furthermore, non-open-source datasets also face the threat of misuse or intellectual property infringement, such as former employees stealing and distributing company private data.

[0003] Dataset copyright auditing is a means of preventing data misuse or intellectual property infringement. A common copyright auditing method is to inject samples from a specific distribution into the dataset before its release, such as methods based on watermark injection. [1,2] However, many datasets have already been released, and users urgently need a post-release mechanism to meet their practical needs, rather than a pre-release mechanism like watermarking that requires completion before dataset release. Furthermore, users cannot guarantee that an effective data watermark has the same data distribution characteristics as the original dataset, which can negatively impact the performance of the user-trained model on normal tasks.

[0004] [1]Chen Y, et al.Teacher Model Fingerprinting Attacks Against TransferLearning[J].2021.DOI:10.48550 / arXiv.2106.12478.

[0005] [2]Wang B,et al.With Great Training Comes Great Vulnerability:Practical Attacks against Transfer Learning[C].27th USENIX security symposium(USENIX Security 18). Summary of the Invention

[0006] This invention addresses the need for copyright auditing of user-owned private training datasets in the field of artificial intelligence security. It proposes a fine-grained copyright auditing method for training datasets that does not require watermarks, addressing the shortcomings of existing copyright auditing methods that rely on the quality of dataset watermarks and the distribution characteristics of training datasets. This method is used to determine whether the model under audit has stolen a user's private deep reinforcement learning training dataset, thus protecting the user's intellectual property rights.

[0007] The objective of this invention is achieved through the following technical solution: a copyright auditing method for training datasets for deep reinforcement learning, comprising the following steps:

[0008] S1, Users train a critic model on their private deep reinforcement learning training dataset based on temporal difference learning;

[0009] S2, the user initializes m deep reinforcement learning models with the same model structure as the model to be audited, denoted as shadow models (the shadow models are the models trained in the preparation phase, and are named here to distinguish them from the models to be audited in the audit phase). The shadow models have the same model structure as the models to be audited, but have different neural network parameters.

[0010] S3, the user trains the initialized shadow model on their private deep reinforcement learning training dataset to obtain m trained shadow models that fit the features of the dataset.

[0011] S4, the user selects a sequence of data, T in length, from their private deep reinforcement learning training dataset, with each time step containing a state signal s. t Action signal a t Reward Information t and the continuity status signal s t+1 The entire sequence data is represented as H = {(s1, a1, r1, s2), ..., (s... T ,a T ,r T ,s T+1 )};

[0012] S5, The user extracts the state signals from the sequence data H and forms a state signal sequence S = {s1, s2, ..., s...} T The user inputs the state signal sequence S into the m shadow models and the model to be audited, respectively, to obtain the corresponding action signal sequence A, where the action signal sequence of the m shadow models is denoted as... The action signal sequence of the model to be audited is denoted as

[0013] S6, the user aligns the state signal sequence S and the action signal sequence A according to time, resulting in m state-action signal sequences for the shadow models as {(S,A)}. i The state-action signal sequence of the model to be audited is (S,A)|i=1,2,…,m}. * );

[0014] S7, the user inputs the state-action signal sequence into the critic model, and the critic model outputs the cumulative reward for each state-action signal; for the state-action signal sequence (S, A) of the i-th shadow model... i The critic model outputs a cumulative reward sequence. For the state-action signal sequence of the model to be audited, (S,A) * The critic model outputs a cumulative reward sequence.

[0015] S8, The user calculates the cumulative reward sequence for m shadow models. The element-wise average value is used to obtain the average cumulative reward sequence of the shadow model.

[0016] S9, the user calculates the cumulative reward sequence R for each shadow model. i With average cumulative reward sequence The distance between them for moving soil is denoted as

[0017] S10, The user calculates the cumulative reward sequence R of the model to be audited. * With average cumulative reward sequence The distance between them for moving soil is denoted as

[0018] S11, user uses d i and d * And the final audit results are obtained by using hypothesis testing methods;

[0019] Specifically, the following hypothesis testing question needs to be verified: H0: the soil-moving distance d of the model to be audited. * Relative to the set of all shadow model soil-moving distances {d i |i=1,2,…,m} is not an outlier; H1: the soil-moving distance d of the model to be audited. * Relative to the set of all shadow model soil-moving distances {d i |i=1,2,…,m} are outliers;

[0020] Users use the Grubbs test to answer the above hypothesis testing questions and output the final copyright audit results;

[0021] S12, the user determines whether all sequence data in their private deep reinforcement learning training dataset has been copyright audited. If so, the audit process ends and the copyright audit results of all sequence data are output; otherwise, proceed to step S4 to reselect sequence data for copyright auditing.

[0022] Furthermore, the critic model trained in S1 has the ability to predict cumulative rewards. Given input state signals and action signals, the critic model outputs the corresponding cumulative reward information.

[0023] Furthermore, the number m of the shadow models is set according to the user's audit accuracy requirements.

[0024] Furthermore, in S8, the cumulative reward for the t-th time step is calculated using the average cumulative reward... Calculated.

[0025] Furthermore, S9 utilizes the open-source function library SciPy to calculate the soil-moving distance.

[0026] Furthermore, in S11, the specific steps of the Grubbs test are as follows:

[0027] Calculate the set of soil-moving distances for all shadow models {d} i The mean μ of |i=1,2,…,m} d and standard deviation σ d ;

[0028] Calculate the Grubbs test statistic if Then reject H0, meaning the sequence data was not used to train the model to be audited; otherwise, the sequence data was used to train the model to be audited. This indicates that when the degrees of freedom are m-2 and the significance level is... The upper bound of the t-distribution is given by α, where α is the confidence level.

[0029] The present invention also provides a copyright auditing device for training datasets for deep reinforcement learning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the above-described copyright auditing method for training datasets for deep reinforcement learning.

[0030] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the above-described method for copyright auditing of training datasets for deep reinforcement learning.

[0031] The beneficial effects of this invention are as follows: Addressing the shortcomings of existing copyright auditing methods, which rely on the quality of the dataset watermark and the distribution characteristics of the training dataset, this invention proposes a fine-grained training dataset copyright auditing method that does not require a dataset watermark. This method is used to determine whether the model under audit has stolen the user's private deep reinforcement learning training dataset, thereby protecting the user's intellectual property rights. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention.

[0033] Figure 2 This plot shows the cross-audit accuracy between datasets for every two Lunar Lander tasks. The title of each plot describes the type of deep reinforcement learning model, the task, and the distance metric used. The x-axis label is the name of the dataset to be detected, i.e., the target dataset. The y-axis label is the name of the dataset the robot controller was learning from, i.e., the actual dataset. Therefore, the diagonal values ​​show the detection accuracy when the actual dataset is the same as the target dataset, i.e., the true positive rate, while the off-diagonal values ​​are the true negative rate. No value indicates 100% accuracy.

[0034] Figure 3 This section shows the cross-audit accuracy between datasets for every two Bipedal Walker tasks. The title of each graph indicates the type of deep reinforcement learning model, the task, and the distance metric used. The x-axis label is the name of the dataset to be detected, i.e., the target dataset. The y-axis label is the name of the dataset the robot controller was learning from, i.e., the actual dataset. Therefore, the diagonal values ​​show the detection accuracy when the actual dataset is the same as the target dataset, i.e., the true positive rate, while the off-diagonal values ​​are the true negative rate results. No value indicates 100% accuracy.

[0035] Figure 4 This section shows the cross-audit accuracy between datasets for every two Ant tasks. The title of each graph describes the type of deep reinforcement learning model, the task, and the distance metric used. The x-axis label is the name of the dataset to be detected, i.e., the target dataset. The y-axis label is the name of the dataset the robot controller was learning from, i.e., the actual dataset. Therefore, the diagonal values ​​show the detection accuracy when the actual dataset and the target dataset are the same, i.e., the true positive rate, while the off-diagonal values ​​are the true negative rate. No value indicates 100% accuracy. Detailed Implementation

[0036] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0037] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0038] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0039] Figure 1 This is a flowchart illustrating a copyright auditing method for training datasets used in deep reinforcement learning according to one embodiment of the present invention. This embodiment of the present invention demonstrates a copyright auditing method for training datasets used in robot control for deep reinforcement learning, the method comprising the following steps:

[0040] The first step involves the user training a critic model on their private training dataset for robot control, which is designed for deep reinforcement learning, using a temporal difference learning approach. This training dataset must contain reward information, and the trained critic model should be able to predict cumulative rewards. For example, given the robot controller's input signal (a state signal) and output signal (an action signal), the critic model can output the corresponding cumulative reward information.

[0041] The second step involves the user initializing m robot controllers (hereinafter referred to as shadow robot controllers) using the same deep reinforcement learning model structure as the robot controller to be audited. The shadow robot controllers have the same model structure as the robot controller to be audited, such as using the same deep reinforcement learning framework and neural network structure. However, the shadow robot controllers have different neural network parameters. The number of shadow robot controllers, m, is set according to the user's audit accuracy requirements; in this embodiment, m is set to 15.

[0042] The third step involves the user training the initialized shadow robot controller on their private deep reinforcement learning training dataset to obtain a shadow robot controller with m trained features fitted to the dataset.

[0043] Fourth, the user selects a unique sequence of data from their private deep reinforcement learning training dataset. This sequence of data has a length of T, and each time step contains the robot controller input signal s. t Robot controller output signal a t Reward Information t and the robot controller input signal s at the next moment t+1Therefore, the entire sequence data can be represented as H = {(s1, a1, r1, s2), ..., (s... T ,a T ,r T ,s T+1 )}.

[0044] The fifth step involves the user extracting the robot controller input signals from the sequence data H, forming the robot controller input signal sequence S = {s1, s2, ..., s...}. T The user inputs the robot controller input signal sequence S into m shadow robot controllers and the robot controller to be audited, respectively, to obtain the corresponding robot controller output signal sequence A. The output signal sequence of the m shadow robot controllers is denoted as... The output signal sequence of the robot controller to be audited is denoted as

[0045] Step 6: The user aligns the input signal sequence S and the output signal sequence A according to time to obtain m input-output signal sequences {(S,A)} for the shadow robot controller. i The input-output signal sequence (S, A) of the robot controller to be audited is given by the input-output signal sequence (S, A) of the robot controller to be audited. * ).

[0046] Step 7: The user inputs the input-output signal sequence into the critic model, and the critic model outputs the cumulative reward for each input-output signal sequence. For example, for the input-output signal sequence (S, A) of the i-th shadow robot controller... i The critic model outputs a cumulative reward sequence. For the input-output signal sequence (S,A) of the robot controller to be audited * The critic model outputs a cumulative reward sequence.

[0047] Step 8: The user calculates the cumulative reward sequence for the m shadow robot controllers. The element-wise average value is used to obtain the average cumulative reward sequence of the shadow robot controller. For example, for the cumulative reward at time step t, the average cumulative reward can be obtained through... Calculated.

[0048] Step 9: The user calculates the cumulative reward sequence R for each shadow robot controller. i With average cumulative reward sequence The earth-moving distance (EMD) between them is denoted as... Users can easily calculate the soil-moving distance using the open-source function library SciPy.

[0049] Step 10: The user calculates the cumulative reward sequence R of the robot controller to be audited. * With average cumulative reward sequence The distance between them for moving soil is denoted as Users can easily calculate the soil-moving distance using the open-source function library SciPy.

[0050] Step 11, the user uses d i and d * The final audit results are obtained by using hypothesis testing methods.

[0051] Specifically, the following hypothesis testing questions need to be verified:

[0052] H0: Distance d of the robot controller to be audited * Relative to the set of soil-moving distances of all shadow robot controllers {d i |i=1,2,…,m} is not an outlier;

[0053] H1: Soil-moving distance d of the robot controller to be audited * Relative to the set of soil-moving distances of all shadow robot controllers {d i |i=1,2,…,m} are outliers;

[0054] Users use the Grubbs test to answer the above hypothesis testing questions and output the final copyright audit results. The specific steps of the Grubbs test are as follows:

[0055] a) Calculate the set of soil-moving distances for all shadow robot controllers {d} i The mean μ of |i=1,2,…,m} d and standard deviation σ d ;

[0056] b) Calculate the Grubbs test statistic.

[0057] c) If Then reject H0, meaning the sequence data was not used to train the robot controller to be audited; otherwise, the sequence data was used to train the robot controller to be audited. This indicates that when the degrees of freedom are m-2 and the significance level is... The upper bound of the t-distribution is given by α, which is the confidence level. In this embodiment, α is set to 0.01.

[0058] Step 12: The user determines whether all sequence data in their private deep reinforcement learning training dataset has been copyright audited. If so, the audit process ends and the copyright audit results of all sequence data are output; otherwise, proceed to step 4 to reselect sequence data for copyright auditing.

[0059] The above steps are compared with experimental analysis as follows:

[0060] To verify the effectiveness of the method of this invention, experiments were conducted using four deep reinforcement learning models (BC, BCQ, IQL, and TD3PlusBC) and three robot control tasks (Lunar Lander, Bipedal Walker, and Ant) to demonstrate the effectiveness of the proposed method in auditing the copyright of the training dataset.

[0061] Experimental Environment: We used the open-source library d3rlpy to implement all the deep reinforcement learning models. All experimental scripts were implemented in Python 3.8 and run on an NVIDIA GTX 3090 server with 256GB of memory.

[0062] Experimental Setup: For each deep reinforcement learning model and robot control task combination, we trained 30 deep reinforcement learning models using different globally random seeds. The hyperparameters of the deep reinforcement learning models used default hyperparameters from open-source libraries, such as neural network architecture, optimizer settings, and number of training epochs. We constructed the critic model using a fully connected neural network. The critic model has four hidden layers, each with 1024 neurons. We optimized the critic model using the Adam optimizer with a learning rate of 0.001. The entire training process required 150 iterations, with the learning rate decreasing by half every 50 iterations. In the experimental analysis, we incorporated three other classic distance metrics: l1 norm, l2 norm, and cosine distance.

[0063] Audit Performance: We evaluated the effectiveness of our proposed method in 12 combinations of 4 deep reinforcement learning models and 3 robot control tasks. For each experimental combination, we generated 5 different datasets for cross-validation of the proposed method. We trained 30 deep reinforcement learning models on each dataset, resulting in a total of 150 deep reinforcement learning models per experimental combination. We performed detection on the 5 datasets, where the auditor randomly selected 15 models from the target dataset as shadow robot controllers, and the remaining 15 models and 120 models from other datasets constituted positive auditable robot controllers (trained using the auditable dataset) and negative auditable robot controllers (trained without using the auditable dataset). For the auditable dataset, we randomly selected 50 sequence data for detection. Due to the imbalance between the number of positive and negative samples, we present the mean and standard deviation of the true positive rate and true negative rate for each experimental setting in Table 1. Figure 2 (Lunar Lander quest) Figure 3 (Bipedal Walker mission) and Figure 4 The (Ant task) provides cross-audit results between every two datasets.

[0064] Table 1. True positive and true negative rates based on Grubbs' test.

[0065]

[0066] The mean and standard deviation of the true positive rate and true negative rate in each row of Table 1 represent the detection results of each task and model combination under the four distance metrics. Bold values ​​indicate the row with the highest sum of true positive and true negative rates, i.e., the highest accuracy. Each pair of true positive and true negative rates is derived from the values ​​of the diagonal and off-diagonal lines in the corresponding heatmap, respectively. Figure 2 , Figure 3 and Figure 4 The results in.

[0067] In this embodiment, the method execution process involves two types of roles: the user and the robot controller to be audited.

[0068] A user conducts a copyright audit on a robot controller that may have been trained using their data without authorization. First, the user trains a reviewer model on their dataset using temporal difference learning and uses it as the audit baseline. Second, the user initializes a batch of shadow robot controllers with different model parameters and trains these controllers on the dataset to be audited. Next, the user collects the responses of both the shadow and audited robot controllers to input signals from the audited dataset, i.e., their output signals. Then, the user feeds the input-output signals of the shadow and audited robot controllers into the reviewer model and records the cumulative reward signal output by the reviewer model. Finally, the user calculates the element-wise average of the cumulative reward signal of the shadow robot controllers (referred to as the cumulative reward signal average) and performs hypothesis testing based on the distance between the cumulative rewards of the shadow and audited robot controllers and the cumulative reward signal average, thus obtaining the final copyright audit result.

[0069] Experimental analysis results demonstrate that this invention achieves high accuracy in copyright auditing of training datasets.

[0070] This invention also provides a copyright auditing device for training datasets for deep reinforcement learning, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the aforementioned copyright auditing method for training datasets for deep reinforcement learning.

[0071] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-described method for copyright auditing of training datasets for deep reinforcement learning.

[0072] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A method for copyright auditing of training datasets for deep reinforcement learning, characterized in that, Includes the following steps: S1, Users train a critic model on their private deep reinforcement learning training dataset based on temporal difference learning; S2, the user initializes m deep reinforcement learning models with the same model structure as the model to be audited, denoted as shadow models. The shadow models have the same model structure as the model to be audited, but have different neural network parameters. S3, the user trains the initialized shadow model on their private deep reinforcement learning training dataset to obtain m trained shadow models that fit the features of the dataset. S4, the user selects a sequence of data, T in length, from their private deep reinforcement learning training dataset, with each time step containing a state signal s. t Action signal a t Reward Information t and the continuity status signal s t+1 The entire sequence data is represented as H = {(s1, a1, r1, s2), ..., (s...} T a T r T s T+1 )}; S5, The user extracts the state signals from the sequence data H, forming a state signal sequence S = {s1, s2, ..., s...} T The user inputs the state signal sequence S into the m shadow models and the model to be audited, respectively, to obtain the corresponding action signal sequence A, where the action signal sequence of the m shadow models is denoted as... The action signal sequence of the model to be audited is denoted as S6, the user aligns the state signal sequence S and the action signal sequence A according to time, resulting in m state-action signal sequences for the shadow models as {(S, A...} i The state-action signal sequence of the model to be audited is (S, A)|i=1,2,...,m}. * ); S7, the user inputs the state-action signal sequence into the critic model, and the critic model outputs the cumulative reward for each state-action signal; for the state-action signal sequence (S, A) of the i-th shadow model... i The critic model outputs a cumulative reward sequence. For the state-action signal sequence of the model to be audited, (S, A) * The critic model outputs a cumulative reward sequence. S8, The user calculates the cumulative reward sequence for m shadow models. The element-wise average value is used to obtain the average cumulative reward sequence of the shadow model. S9, the user calculates the cumulative reward sequence R for each shadow model. i With average cumulative reward sequence The distance between them for moving soil is denoted as S10, The user calculates the cumulative reward sequence R of the model to be audited. * With average cumulative reward sequence The distance between them for moving soil is denoted as S11, user uses d i and d * And the final audit results are obtained by using hypothesis testing methods; Specifically, the following hypothesis testing question needs to be verified: H0: the soil-moving distance d of the model to be audited. * Relative to the set of all shadow model soil-moving distances {d i |i=1,2,...,m} is not an outlier; H1: the soil-moving distance d of the model to be audited * Relative to the set of all shadow model soil-moving distances {d i |i = 1, 2, ..., m} are outliers; Users use the Grubbs test to answer the above hypothesis testing questions and output the final copyright audit results; S12, the user determines whether all sequence data in their private deep reinforcement learning training dataset has been copyright audited. If so, the audit process ends and the copyright audit results of all sequence data are output; otherwise, proceed to step S4 to reselect sequence data for copyright auditing.

2. The copyright auditing method for training datasets for deep reinforcement learning according to claim 1, characterized in that, The critic model trained in S1 has the ability to predict cumulative rewards. Given state and action signals as input, the critic model outputs the corresponding cumulative reward information.

3. The copyright auditing method for training datasets for deep reinforcement learning according to claim 1, characterized in that, The number m of the shadow models is set according to the user's audit accuracy requirements.

4. The copyright auditing method for training datasets for deep reinforcement learning according to claim 1, characterized in that, In S8, the cumulative reward for the t-th time step, the average cumulative reward is obtained through... Calculated.

5. The copyright auditing method for training datasets for deep reinforcement learning according to claim 1, characterized in that, S9 uses the open-source function library SciPy to calculate the soil-moving distance.

6. The copyright auditing method for training datasets for deep reinforcement learning according to claim 1, characterized in that, In S11, the specific steps of the Grubbs test are as follows: Calculate the set of soil-moving distances for all shadow models {d} i The mean μ of |i=1,2,...,m) d and standard deviation σ d ; Calculate the Grubbs test statistic if Then reject H0, meaning the sequence data was not used to train the model to be audited; otherwise, the sequence data was used to train the model to be audited. This indicates that when the degrees of freedom are m-2 and the significance level is... The upper bound of the t-distribution is given by α, where α is the confidence level.

7. A copyright auditing device for training datasets for deep reinforcement learning, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it is used to implement the copyright auditing method for training datasets for deep reinforcement learning as described in any one of claims 1-6.

8. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the copyright auditing method for training datasets for deep reinforcement learning as described in any one of claims 1-6.

Images