A deep reinforcement learning-based white-box watermark embedding method

Abstract

The application provides a white-box watermark embedding method based on deep reinforcement learning, which comprises a model 1, a model 2 and a model 3. The model 2 is obtained through watermark elimination training of the model 1 by using randomly generated watermark and a key and through training rounds. The model 3 is obtained through watermark embedding training of the model 2 by using the watermark and the key and through reinforcement learning task training. The application can effectively solve the problem that when an attacker obtains a distributed reinforcement learning model, the watermark embedded by the application is eliminated to change the model copyright, can effectively solve the problem that when the attacker does not change the model parameters, a reverse key generation method is used to blur the copyright, and can effectively guarantee the availability of the white-box watermark of the deep reinforcement learning.

Description

Technical Field

[0001] This invention relates to a white-box watermarking embedding method based on deep reinforcement learning, which involves reinforcement learning training algorithms, neural network watermarking embedding algorithms and verification algorithms, and belongs to the field of reinforcement learning model protection. Background Technology

[0002] With the rise of artificial intelligence, deep learning has been studied by more and more people, and neural networks are playing an increasingly important role. Neural network algorithms have achieved excellent results in fields such as image processing and natural language processing. However, in deep learning, training a good model often requires a significant investment of time and money. It necessitates continuous tuning of network hyperparameters and consumes substantial computing resources. Therefore, to lower the barrier to entry, IBM proposed providing pre-trained neural networks to consumers who need high-performance neural networks but cannot afford the high training costs. However, this approach has also raised new issues, such as how to protect the copyright of the provided models. Therefore, researching how to protect the copyright of the provided models is essential.

[0003] Reinforcement learning, as an important method for solving sequential decision-making problems, allows intelligent agents to explore on their own, interact with their environment, and receive rewards based on environmental feedback, thereby optimizing their behavior. Reinforcement learning is suitable for scenarios where problems are difficult to solve using traditional algorithms, such as robot control and game AI. Similarly, the learning process of reinforcement learning does not require manually labeled data; instead, it learns through interaction with the environment. This allows reinforcement learning to be applied to many real-world scenarios, such as autonomous driving and logistics scheduling, without the need for extensive data labeling. Therefore, reinforcement learning has broad application prospects and research value in the field of artificial intelligence.

[0004] Previous research on white-box watermarking in neural networks has primarily focused on image classification tasks, with no white-box watermark embedding methods currently applied to reinforcement learning tasks. Existing methods, such as adding extra passport layers or embedding the watermark after training, require attackers to gain access to the model and prevent further training, significantly limiting their capabilities. Therefore, a method that embeds the watermark during training is chosen. Existing watermark embedding methods can embed watermarks into neural networks without affecting the performance of reinforcement learning tasks. However, this method has been shown to be effective in image classification tasks, where the watermark location can be detected and removed, allowing attackers to add their own watermarks to obtain copyright information.

[0005] Previous research on neural network watermarking has primarily focused on image classification tasks, with very little research specifically on watermarking for deep reinforcement learning. Therefore, this study aims to propose a feasible watermarking method for protecting deep reinforcement learning models. Summary of the Invention

[0006] In view of the above-mentioned deficiencies of the prior art, the purpose of this invention is to provide a white-box watermarking embedding method based on deep reinforcement learning.

[0007] The above-mentioned technical objective of the present invention is achieved through the following solution: A deep reinforcement learning-based white-box watermarking embedding method includes the following steps: S1. Construct the first watermark and the first key; S2. Perform watermark embedding training and reinforcement learning tasks using the first watermark and the first key to generate the first model; S3. Generate a binary bitstream based on the parameters of the watermark layer embedded in the first model; S4. Generate a second watermark based on a binary bitstream; generate a second key based on a binary bitstream. S5. The first model generates a random watermark and a random key, and iteratively performs watermark removal training to generate a second model. S6. Using the first watermark and the second key as input, or using the second watermark and the first key as input, perform watermark embedding training and reinforcement learning tasks to generate a third model; S7, Distribute the third model.

[0008] Preferably, the first watermark is a randomly generated string of length len, where the first watermark ∈ {0,1}len, and the length len is the length of the embedded watermark, with a byte count of 64bit-256bit; the first key is a randomly generated two-dimensional array conforming to a Gaussian distribution, where one dimension is the length of the first watermark, and the other dimension is the parameter that the model needs to embed the watermark layer.

[0009] Preferably, the watermark embedding method utilizes the principles of matrix multiplication and neural network inverse optimization to embed the watermark into the parameters of the neural network. The function is:

[0010] in To enhance the loss function of the learning task, This is a hyperparameter; its default value is 0.001. The loss function for embedding the watermark.

[0011] Preferably, the watermark removal method is consistent with the embedding method.

[0012] Preferably, the watermark is embedded with encrypted network information to become a new watermark.

[0013] The present invention has the following advantages over the prior art: (1) The neural network white-box watermarking method was applied to reinforcement learning tasks. Through experiments, the problems of the watermark removal method were found and used as a defense aspect.

[0014] (2) Combining two defense methods makes reinforcement learning watermarks effective against existing attacks. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the structure of the present invention; Figure 2 This is a detailed embodiment of method 1 of the present invention; Figure 3 This is a detailed embodiment of method 2 of the present invention; In the diagram: B1, first watermark; X1, first key; B2, second watermark; X2, second key; B', random watermark; X', random key. Detailed Implementation

[0016] The embodiments of the present invention will be described in detail below. The embodiments described below are implemented based on the technical solution of the present invention, and detailed implementation methods and specific operation processes are given. However, the protection scope of the present invention is not limited to the embodiments described below.

[0017] A deep reinforcement learning-based white-box watermarking embedding method, as shown in the figure, includes the following steps: In this embodiment, watermark B is randomly generated using random numbers. 1 Key X 1 .

[0018] In this embodiment, a reinforcement learning task is defined, watermark embedding is performed in a selected layer, and the loss function in the neural network is modified. The training rounds required for the watermark embedding task are much shorter than those for the reinforcement learning task. Training continues until the reinforcement learning task meets the expected goal, and then the result is saved as the current model w. 1 Neural networks The function is: (1) in To enhance the loss function of the learning task, This is a hyperparameter, and its default value in this invention is 0.001. The loss function for embedding the watermark.

[0019] Specifically, it is expressed as follows: (2) in To Base is Logarithmic operations; Specifically, it is expressed as follows: (3) in In for The Middle OK.

[0020] Formula for detecting watermark embedding rate for: *100% (4) in Specifically, it is expressed as follows: (5) in Specifically, it is expressed as follows: (6) Specifically, it is expressed as: (7) in, For the first The function value up to the first indivual Sum the function values.

[0021] In this embodiment, the SHA-256 algorithm is used to generate a model w. 1 A binary bitstream embedding watermark layer parameters; generating a watermark B based on the binary bitstream. 2 and key X 2 ; In this embodiment, the already saved B is no longer used after S4. 1 X 1 Training is performed, with the loss calculated using a random watermark B' and a random key X'. Once the watermark detection rate drops to around 50% and the model's watermark loss stabilizes, the model is saved as the current model w. 2 .

[0022] In this embodiment, watermark B is used. 1 and key X 2 Input can be saved directly (B) 2 The algorithm is as follows: (8) in Specifically, it is expressed as follows: (9) Or with watermark B2 and key X 1 Can key X 1 Perform diagonal element replacement and save the replaced key X. 2 The algorithm is as follows: (10) in Specifically, it is expressed as follows: (11) Once the watermark detection rate reaches 100% and the reinforcement learning task meets the expected goals, the model is saved as the current model w. 3 .

[0023] In this embodiment, the model updates model w through the above steps. 3 These parameters ensure that the model provided to consumers guarantees ownership.

[0024] In summary, the main technical advantages of this invention are as follows: First, it applies the neural network white-box watermarking method to reinforcement learning tasks; second, it identifies the problems with existing watermark removal methods through experiments and uses them as a defense mechanism; third, it uses hash encryption to generate watermarks or keys to defend against attacks that do not require training the network to reverse-generate keys, which cannot be solved by the original method; and fourth, it combines two defense methods to ensure that reinforcement learning watermarks can effectively resist existing attacks.

[0025] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A white-box watermarking embedding method based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Construct the first watermark and the first key; S2. Perform watermark embedding training and reinforcement learning tasks using the first watermark and the first key to generate the first model; S3. Generate a binary bitstream based on the parameters of the watermark layer embedded in the first model; S4. Generate a second watermark based on a binary bitstream; Generate a second key based on a binary bitstream; S5. The first model generates a random watermark and a random key, and iteratively performs watermark removal training to generate a second model. S6. Using the first watermark and the second key as input, or using the second watermark and the first key as input, perform watermark embedding training and reinforcement learning tasks to generate a third model; S7, Distribute the third model.

2. The method according to claim 1, characterized in that, The first watermark is a randomly generated string of length len, where the first watermark ∈ {0,1}len. The length len is the length of the embedded watermark, and the number of bytes is 64bit-256bit.

3. The method according to claim 1, characterized in that, The first key is a randomly generated two-dimensional array that conforms to a Gaussian distribution, where one dimension is the length of the first watermark and the other dimension is the length of the parameters that the model needs to embed the watermark layer into.

4. The method according to claim 1, characterized in that, In step S2, the watermark embedding method utilizes the principles of matrix multiplication and neural network inverse optimization to embed the watermark into the parameters of the neural network. The function is: in To enhance the loss function of the learning task, This is a hyperparameter; its default value is 0.

001. The loss function for embedding the watermark.

5. The method according to claim 1, wherein in step S5, the following is true: The watermark removal method is the same as the embedding method.

6. The method according to claim 1, wherein in step S6, the following is true: The watermark is embedded with encrypted network information to become a new watermark.

Images