Controllable text watermark embedding method based on reinforcement learning policy model

Abstract

The application discloses a controllable text watermark embedding method based on a reinforcement learning strategy model, constructs a reinforcement learning strategy model, and controls token output of a large language model through the reinforcement learning strategy model; under strategy control of the reinforcement learning strategy model, watermark embedding is performed on output text based on a watermark embedding strategy, and a candidate token sequence with hidden information is formed. The application introduces a strategy network based on a reward function to dynamically intervene in a token level in a large language model generation process, simultaneously fuses multiple embedding modes such as vocabulary replacement, syntax disturbance and structure addition, and provides a feedback signal through a watermark detector to realize end-to-end closed-loop optimization, so that the naturalness, robustness and detectability of text watermark are significantly improved without modifying a main structure of a language model, and good deployment flexibility and traceability are achieved.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and artificial intelligence generated content (AIGC) security, and specifically a controllable text watermark embedding method based on a reinforcement learning policy model. Background Technology

[0002] In existing technologies, with the widespread application of large language models (LLMs) such as GPT, LLaMA, and PaLM, they have demonstrated powerful generative capabilities in multiple fields such as natural language generation, dialogue systems, code completion, and content creation.

[0003] However, the “uncontrollability” of text generated by such models also brings potential risks of abuse, including but not limited to the spread of false information, academic plagiarism, and automated creation of online troll content. To mitigate these risks, researchers have proposed a technique called “text watermarking”, which aims to embed verifiable and extractable hidden information into the generated text while maintaining its naturalness.

[0004] Traditional text watermarking techniques include methods based on word substitution, syntactic structure reconstruction, and sentence-end appending, such as the "A Method for Embedding and Extracting Text Watermarks" disclosed in authorization announcement number CN 110414194 B. Although some progress has been made in embedding information, there are obvious shortcomings: (1) the embedding position depends on a fixed strategy and lacks flexibility; (2) it may disrupt the fluency of language and the consistency of context; (3) it cannot dynamically adapt to the text generation environment or user needs. In addition, most existing methods cannot form an end-to-end closed-loop optimization system, making it difficult to balance the generation quality and the detectability of the watermark.

[0005] In recent years, reinforcement learning (RL), as a goal-oriented optimization learning mechanism, has been widely used in policy control scenarios in generative tasks, such as text summarization optimization and dialogue policy generation. RL has a natural "goal-driven" characteristic and is suitable for handling optimization problems with multiple conflicting goals. Therefore, reinforcement learning-based methods for introducing control signals and feedback mechanisms into text watermark embedding have good potential.

[0006] However, applying reinforcement learning directly to text watermark embedding still presents technical challenges, such as how to design a suitable reward function to balance language quality and watermark detectability; and how to seamlessly integrate RL policies into the LLM generation process to achieve token-level selection control.

[0007] Therefore, there is an urgent need to propose a new and feasible technical solution that can achieve high-quality, highly detectable watermark embedding targets without interfering with the main LLM model structure, and has good deployability and scalability. Summary of the Invention

[0008] The purpose of this invention is to provide a black-box hinting attack method based on a universal trigger that can efficiently execute cross-task backdoor attacks; the technical solution is as follows: a controllable text watermark embedding method based on a reinforcement learning policy model, comprising the following steps:

[0009] Step 1: Construct a reinforcement learning policy model to control the token output of the large language model.

[0010] Step 2: Under the policy control of the reinforcement learning policy model, watermark embedding is performed on the output text based on the watermark embedding strategy to form a candidate token sequence with hidden information.

[0011] Step 3: Establish a watermark detector to determine whether watermark information has been successfully embedded in the text, and pass the detection result as a signal back to the reinforcement learning policy network of the reinforcement learning policy model to form a closed-loop training system.

[0012] Step 4: Implement hierarchical encoding and mapping of identity information, encode specific identifiable information into watermark payload, and embed it into the text to achieve the functions of "traceability, verifiability, and attributability";

[0013] Step 5: Deploy the watermark embedder and complete model integration. Integrate the watermark embedding mechanism into the existing large language model framework with minimal modifications to support engineering deployment and large-scale application.

[0014] Furthermore, the specific details of constructing the reinforcement learning policy model in step 1 are as follows:

[0015] Step 1.1: A multi-objective fusion reward function design method is proposed to simultaneously consider text generation quality and watermark embedding feasibility. Specifically, the reward function is designed as follows:

[0016] in, (1)

[0017] in, This represents the confidence score of the embedded text being identified as "watermarked" by the watermark detector. It is a probability value with a value of [0, 1] and can be obtained by the watermark detector in step 3 after inputting the text. The text naturalness score, used to measure the stylistic and semantic consistency between the generated text and the original language model output, is defined as the KL divergence with the distribution of the underlying language model.

[0018] (2)

[0019] in, This represents the probability distribution of tokens generated by the current reinforcement learning policy model. The output distribution of the original language model is shown. A smaller KL divergence indicates greater similarity between the two models, meaning better naturalness. Therefore, its negative value is taken as the naturalness score. Simultaneously, corresponding weight parameters are set. and Adjust the balance between naturalness and embedding capability according to the actual application scenario;

[0020] Step 1.2: Construct and optimize a reinforcement learning policy network To learn in a given current generation context Under the given conditions, select the most suitable token. The strategy is to maximize the predefined reward function. The training method employs the standard Policy Gradient optimization algorithm, whose gradient expression is as follows:

[0021] (3)

[0022] in Indicates time step The token selected at the time Indicates the current context state, such as the already generated text sequence; These are the trainable parameters of the policy network; Indicates the parameter The gradient operator, i.e., the partial derivative vector; The objective function for policy optimization is represented by The gradient with respect to the parameter θ;

[0023] Step 1.3: Deploy the reinforcement learning policy network trained in Step 1.2 as a token reranker. This reranker is used to adjust the token selection logic in real time during the actual text generation process, thereby achieving precise control over the embedding behavior at the token level. The optimal token is selected according to the following formula:

[0024] (4)

[0025] Where y represents any token in the candidate set, and Vt This represents the set of candidate tokens (such as the top-k candidate set) output by the large language model at time step t, which contains all possible selected tokens. This indicates the finally selected token. For the policy network to select candidate tokens in the current state - The probability of scoring.

[0026] Furthermore, the specific method for embedding a watermark in the output text based on the watermark embedding strategy in step 2 is as follows:

[0027] Step 2.1: Obtain the language model The candidate token set at each generation step Extract the top-ranked words with the highest probability from the distribution output by the language model. Each token constitutes a candidate set. ,Right now:

[0028] (5)

[0029] This set serves as the input basis for subsequent watermark bit-to-token mapping; where LLM stands for Large Language Model.

[0030] Step 2.2: Assume the watermark information to be embedded is represented as a one-bit sequence. ,in For the i-th bit; the system follows the current bit in each step t. Select a suitable token from the candidate set:

[0031] (6)

[0032] This mapping function can be implemented through positional bias, hash mapping, or table lookup to adapt to the output distribution of different language models.

[0033] Step 2.3: Employ semantically preserving syntactic perturbation techniques to perturb the sentence structure. Methods include subject-verb-object substitution, phrase insertion, and the use of relative clauses, ensuring that the original sentence and the perturbed sentence remain semantically close.

[0034] (7)

[0035] in This represents semantic embedding representation, using sentence vectors extracted by models such as BERT to ensure that the sentence content remains consistent with the original meaning after perturbation; This represents the original sentence, i.e., the text generated by the large language model before any syntactic perturbations. This refers to the perturbated sentence, that is, the sentence generated through syntactic transformation while preserving its semantics;

[0036] Step 2.4: Use the appended structured token sequence as the watermark carrying area. When the context length allows, add the remaining watermark bits to the end of the text using the structured token sequence encoding.

[0037] (8)

[0038] in, It is a function used to map bit strings to token sequences, and can be implemented based on template libraries; This represents the original generated text sequence. This represents the complete text following the appended structured watermark token sequence;

[0039] Step 2.5: To adapt to the embedding requirements of different contexts, a policy controller is used based on the current context state. With embedded state Output the policy score and select the optimal embedding policy. :

[0040] (9)

[0041] in, For the candidate policy set, The strategy scoring function comprehensively evaluates embeddability, semantic preservation, and contextual adaptability, ultimately achieving dynamic embedding path planning.

[0042] Furthermore, the specific method for establishing the watermark detector and participating in joint training feedback in step 3 is as follows:

[0043] Step 3.1: Build a watermark detector based on the Transformer architecture As a discrimination module added to the entire watermark training system, the watermark detector judges any input text x and outputs its probability score of containing the watermark, which is then normalized to a confidence score using the sigmoid function.

[0044] (10)

[0045] in The logit value output by the Transformer encoder. This represents the sigmoid activation function and its output value. This is used to measure the strength of the current text watermark signal;

[0046] Step 3.2: Construct a training set using both watermarked and unwatermarked corpora, and perform supervised training. The loss function is cross-entropy.

[0047] (11)

[0048] in, A label indicating whether a sample contains a watermark. It is the confidence level of the model's prediction; this training process enables the watermark detector to distinguish between watermarked and non-watermarked text.

[0049] Step 3.3: To form a joint optimization mechanism for watermark embedding and detection, the watermark confidence score output by the watermark detector is... It is incorporated into the reward function of the reinforcement learning policy model as an evaluation metric for the watermark embedding effect:

[0050] (12)

[0051] Furthermore, the specific method for implementing hierarchical bit encoding of identity information in step 4 is as follows:

[0052] Step 4.1: First, extract core metadata from the generated task, including the user ID, generation timestamp, and task ID, and then use an encoding function. The UserID, Generation Timestamp, and TaskID are uniformly encoded and converted into a bitstream. ;

[0053] Step 4.2: Candidate set based on each token generation time , bit stream The bitwise mapping is applied to the corresponding token position. The mapping function determines the corresponding candidate token based on the bit value, maps bit 0 to the natural token that is distributed earlier, and maps bit 1 to the perturbed or offset subset of tokens, thereby completing the information embedding without significantly compromising the language quality.

[0054] Step 4.3: A layered embedding mechanism is introduced, and the watermark bitstream is divided into a macro layer and a micro layer: the macro layer is used to embed global information; the micro layer is used for task-level information.

[0055] Furthermore, the specific method for step 5 is as follows:

[0056] Step 5.1: The watermark embedder is encapsulated as an independent "post-processing module". After the large language model LLM completes the output of the token probability distribution, the output sequence is reordered and selected in an interception manner. This "post-processing module" does not modify any network parameters or architecture of the main model, but is attached to the decoding process as a token selection and reordering unit to achieve fine-tuning of the embedded output tokens.

[0057] Step 5.2: The watermark embedder is integrated through a standardized API interface call. Users load the watermark control strategy through HTTP request or local call. The input includes the generated text content, user identity information, and task configuration parameters. The watermark encoding, embedding, and control output are completed automatically.

[0058] Furthermore, the encoding method in step 4.1 is set to a standard hash function, AES encrypted hash, or variable-length binary encoding, and the generated bit stream is used to control the selection of specific tokens in the subsequent watermark embedding process.

[0059] Furthermore, the global information at the macro level in step 4.3 includes the user ID and authorization code; the task-level information at the micro level includes the context identifier or version number of the specific generated content.

[0060] Beneficial Effects: This invention has the following beneficial effects: This invention designs a unified framework that combines reinforcement learning control mechanism with multi-strategy text watermark embedding. By introducing a policy network based on reward function to dynamically intervene in the generation process of large language models at the token level, it integrates multiple embedding methods such as word substitution, grammatical perturbation and structural addition, and uses a watermark detector to provide feedback signals to achieve end-to-end closed-loop optimization. Thus, without modifying the main structure of the language model, it significantly improves the naturalness, robustness and detectability of text watermarks, and has good deployment flexibility and traceability capabilities. Attached Figure Description

[0061] Figure 1 This is an overall flowchart of the present invention;

[0062] Figure 2 This is a schematic diagram of the overall framework of the present invention. Detailed Implementation

[0063] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0064] like Figure 1 and Figure 2 As shown, the controllable text watermark embedding method based on a reinforcement learning policy model has the following specific steps:

[0065] Step 1: The language model generation process is modeled as a sequential decision-making process. The state is the currently generated text fragment, and the actions are all possible tokens in the space. The policy function takes the current token distribution as input and outputs a reordering probability distribution on the candidate token set. To train this policy model, a multi-objective reward function is introduced. This function considers both the quality of language generation (e.g., KL divergence with the basic LLM output) and the watermark embedding effect (e.g., the recognition confidence of the watermark detector). A policy gradient method is used to optimize the policy network. The specific steps are detailed below:

[0066] Step 1.1: A multi-objective fusion reward function design method is proposed to simultaneously consider text generation quality and watermark embedding feasibility. Specifically, the reward function is designed as follows:

[0067] in, (1)

[0068] in, This represents the confidence score of the embedded text being identified as "watermarked" by the watermark detector. It is usually a probability value of [0, 1] and can be obtained by the watermark detector after inputting the text. The text naturalness score, used to measure the stylistic and semantic consistency between the generated text and the original language model output, is defined as the KL divergence with the distribution of the underlying language model.

[0069] (2)

[0070] here This represents the probability distribution of tokens generated by the current strategy model. The output distribution of the original language model (such as GPT or BERT) is used; the smaller the KL divergence, the more similar the two are, i.e., the better the naturalness; therefore, its negative value is taken as the naturalness score; at the same time, appropriate weight parameters are set. and The balance between naturalness and embedding capability is adjusted according to the actual application scenario.

[0071] Step 1.2: Construct and optimize a reinforcement learning policy network To learn in a given current generation context Under the given conditions, select the most suitable token. The strategy is to maximize the predefined reward function. The training method employs the standard Policy Gradient optimization algorithm, whose gradient expression is as follows:

[0072] (3)

[0073] Among them, among them, Indicates time step The token selected at the time Indicates the current context state, such as the already generated text sequence; These are the trainable parameters of the policy network. Indicates the parameter The gradient operator, i.e., the partial derivative vector; The objective function for policy optimization is represented by The gradient with respect to the parameter θ;

[0074] Step 1.3: Deploy the reinforcement learning policy network trained in Step 1.2 as a token reranker. This reranker is used to adjust the token selection logic in real time during the actual text generation process, thereby achieving precise control over the embedding behavior at the token level. The optimal token is selected according to the following formula:

[0075] (4)

[0076] Where y represents any token in the candidate set, V t This represents the set of candidate tokens (such as the top-k candidate set) output by the large language model at time step t, which contains all possible selected tokens. This indicates the finally selected token. For the policy network to select candidate tokens in the current state - The probability of scoring;

[0077] Step 2: Based on the watermark information to be embedded (usually encoded as a binary bit string), select the token corresponding to the target bit from the candidate tokens output by the language model for embedding, thus achieving information injection while preserving semantics; the specific steps are detailed below:

[0078] Step 2.1: Obtain the top-k candidate token set for the language model in each generation step. Extract the top-ranked words with the highest probability from the distribution output by the language model. Each token constitutes a candidate set. ,Right now:

[0079] (5)

[0080] This set serves as the input basis for subsequent watermark bit-to-token mapping;

[0081] Step 2.2: Assume the watermark information to be embedded is represented as a one-bit sequence. ,in For the i-th bit; the system follows the current bit in each step t. Select a suitable token from the candidate set:

[0082] (6)

[0083] This mapping function can be used through positional bias (e.g., even-numbered bits are encoded as 1), hash mapping (e.g.) This can be achieved through lookup tables (such as preset dictionary indexes), flexibly adapting to the output distribution of different language models;

[0084] Step 2.3: Employ semantically preserving syntactic perturbation techniques to perturb the sentence structure. Methods include subject-verb-object substitution, phrase insertion, and the use of relative clauses, ensuring that the original sentence and the perturbed sentence remain semantically close.

[0085] (7)

[0086] in This represents semantic embedding representation, using sentence vectors extracted by models such as BERT to ensure that the sentence content remains consistent with the original meaning after perturbation; This represents the original sentence, i.e., the text generated by the large language model before any syntactic perturbations. This refers to the perturbated sentence, that is, the sentence generated through syntactic transformation while preserving its semantics;

[0087] Step 2.4: Append a structured token sequence as the watermark carrying area. When the context length allows, the remaining watermark bits can be encoded by adding the structured token sequence to the end of the text.

[0088] (8)

[0089] in, It is a function used to map bit strings to token sequences, and can be implemented based on template libraries; This represents the original generated text sequence. This represents the complete text after the structured watermark token sequence has been appended. It can be implemented based on a template library, for example, by embedding it in a natural language style such as: "This output was generated at [ID: 0101101]".

[0090] Step 2.5: To adapt to the embedding requirements of different contexts, a policy controller is used based on the current context state. With embedded state Output the policy score and select the optimal embedding policy. :

[0091] (9)

[0092] in, The set of candidate strategies (such as lexical substitution, syntactic perturbation, and structural addition). The strategy scoring function comprehensively evaluates embeddability, semantic preservation, and contextual adaptability, ultimately achieving dynamic embedding path planning.

[0093] Step 3: Train a watermark detector to identify whether watermark information is embedded in the generated text, and further use the identification result as a feedback signal for the reinforcement learning strategy to train the policy model, thereby constructing a closed-loop optimization system of generation-detection-feedback; the specific steps are as follows:

[0094] Step 3.1: Build a watermark detector based on the Transformer architecture As a discrimination module added to the entire watermark training system, the detector judges any input text x and outputs its probability score of containing the watermark, which is then normalized to a confidence score using the sigmoid function.

[0095] (10)

[0096] in, The logit value output by the Transformer encoder. This represents the sigmoid activation function and its output value. This is used to measure the strength of the current text watermark signal;

[0097] Step 3.2: Construct a training set using both watermarked and unwatermarked corpora, and perform supervised training. The loss function is cross-entropy.

[0098] (11)

[0099] in A label indicating whether a sample contains a watermark. It is the confidence level of the model's prediction. This training process enables the detector to distinguish between watermarked and non-watermarked text.

[0100] Step 3.3: To form a joint optimization mechanism for watermark embedding and detection, the watermark confidence score output by the detector is... It is incorporated into the reward function of the reinforcement learning strategy as an evaluation metric for the watermark embedding effect:

[0101] (12)

[0102] Step 4: The information required to identify the user or platform, such as user ID, platform ID, timestamp, and task number, is uniformly encoded and hierarchically managed to generate a watermark payload with structured meaning. This payload is then embedded as a binary bit sequence into the text generation process; specifically as follows:

[0103] Step 4.1: First, extract core metadata from the generated task, including user ID, generation timestamp, and task ID, and then use an encoding function. The UserID, Generation Timestamp, and TaskID are uniformly encoded and converted into a bitstream. ;

[0104] The encoding method can be a standard hash function, AES encrypted hash, or variable-length binary encoding. The generated bitstream is used to control the selection of specific tokens during the subsequent watermark embedding process.

[0105] Step 4.2: Candidate set based on each token generation time , bit stream The bitwise bitwise mapping is applied to the corresponding token position. The mapping function determines the corresponding candidate token based on the bit value. For example, bit 0 is mapped to the natural token that is distributed earlier in the distribution, and bit 1 is mapped to a perturbed or offset subset of tokens, thereby completing the information embedding without significantly compromising the language quality.

[0106] Step 4.3: A layered embedding mechanism is introduced; the watermark bitstream can be divided into a macro layer and a micro layer; the macro layer is used to embed global information, such as user ID and authorization code; the micro layer is used for task-level information, such as the context identifier or version number of the specific generated content.

[0107] Step 5: Integrate the entire embedding process into the existing large language model system as a modular plugin, as follows:

[0108] Step 5.1: Encapsulate the watermark embedder as an independent "post-processing module". After the LLM completes the output of the token probability distribution, the output sequence is reordered and selected in an interception manner. This "post-processing module" will not modify any network parameters or architecture of the main model. Instead, it is attached to the decoding process (such as sampling or beam search) as a token selection reorderer to achieve fine-tuning of the embedded output token.

[0109] Step 5.2: The watermarking module is integrated through standardized API interface calls. Users load the watermark control strategy through HTTP requests or local calls. The inputs include the generated text content, user identity information, task configuration parameters, etc. The module automatically completes watermark encoding, embedding, and control output.

[0110] Example 1

[0111] To verify the effectiveness of the present invention, the following experiment was conducted in this embodiment: LLaMA-7B was used as the underlying language model, and GPT-2 was selected as the baseline model. Without modifying its network structure, the sampled outputs were reranked through the policy network.

[0112] The policy network is a Transformer structure, consisting of a reward function. Guided learning, reward coefficient set to , Among them, the naturalness score Watermark recognizability It comes from a jointly trained detector. The decoding strategy is Top-k (k=8) and reinforcement learning-based reranking.

[0113] Table 1 shows the performance comparison data of the reinforcement learning-based watermark embedding method of the present invention in terms of text naturalness and detectability with traditional fixed-policy watermarking. The evaluation metrics are: Perplexity, which measures the "naturalness" of the text to the language model (lower values ​​are better); Detect Accuracy, which determines the accuracy of classifying whether the text contains a watermark (e.g., whether it contains embedded identity information); and Bit Recovery Rate (BRR), which measures the proportion of watermark bits that can be successfully recovered from the embedded text. The table below compares the performance of the reinforcement learning-based watermark embedding method in terms of text naturalness and detectability.

[0114] Table 1

[0115]

[0116] Experimental Results Analysis: As can be seen from the experimental results in Table 1, the reinforcement learning watermark embedding strategy proposed in this invention achieves an effective balance between preserving text naturalness and watermark detectability. Compared with traditional rule-based watermarking methods (such as fixed-strategy watermarking in the table), the PPL value is reduced from 20.6 to 20.0 (the lower the PPL value, the better); the DetectAccuracy value is increased from 94.2% to 96.7%; and the BRR value is increased from 85.5% to 89.1%. All data comparisons are superior to the existing fixed-strategy watermarking methods.

[0117] The adaptive embedding control mechanism trained by the strategy of this invention can automatically select a more covert embedding method according to the context, thereby further enhancing the system's covertness and anti-attack capability.

[0118] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the implementation of the present invention or the scope of the claims. All equivalent changes and modifications made in accordance with the scope of patent protection of the present invention should be included within the scope of the present invention patent application.

Claims

1. A controllable text watermark embedding method based on a reinforcement learning policy model, characterized in that, Includes the following steps: Step 1: Construct a reinforcement learning policy model to control the token output of the large language model. Step 2: Under the policy control of the reinforcement learning policy model, watermark embedding is performed on the output text based on the watermark embedding strategy to form a candidate token sequence with hidden information. The specific steps are as follows: Step 2.1: Obtain the large language model Candidate token set: at each generation step Extract the top-probability values ​​from the distribution output by the large language model. Each token constitutes a candidate set. ,Right now: （5）； in, This represents a truncation selection operation based on probability ranking, i.e., from a large language model. From the generated complete vocabulary probability distribution, the top k tokens with the highest probabilities are selected as candidates in descending order, forming the candidate set. It serves as the input basis for subsequent mapping between watermark bits and tokens; where LLM stands for Large Language Model. This refers to the conditional probability distribution generated by the large language model; Represents all possible distributions, Indicates based on the current state All possible distributions; Step 2.2: Assume the watermark information to be embedded is represented as a one-bit sequence. ,in For the i-th bit; the system follows the current bit in each step t. Select a token from the candidate set: （6）； in, This indicates the finally selected token. Indicates the first Watermark bits, Represents the set of candidate tokens. Represents a mapping function; the The mapping function is implemented through positional bias, hash mapping, or table lookup to adapt to the output distribution of different large language models; n is a positive integer; Step 2.3: Employ semantically preserving syntactic perturbation techniques to perturb the sentence structure, specifically as follows: （7）； Among them, Sem The semantic embedding representation is obtained by extracting sentence vectors from the BERT model, ensuring that the sentence content remains consistent with the original meaning after perturbation. This represents the original sentence, generated by a large language model before any syntactic perturbations. This refers to the sentence generated through syntactic transformations after perturbation, while preserving its semantics. Step 2.4: Use the appended structured token sequence as the watermark carrying area. When the context length allows, add the remaining watermark bits to the end of the text using the structured token sequence encoding. （8）； in It is a function used to map bit strings to token sequences, implemented based on a template library; This represents the original generated text sequence. This represents the complete text following the appended structured watermark token sequence; Step 2.5: To adapt to the embedding requirements of different contexts, a reinforcement learning policy model is used based on the current context state. With embedded state Output the policy score and select the optimal embedding policy. : （9）； in For the candidate policy set, The policy scoring function comprehensively evaluates embedding ability, semantic preservation, and context adaptation to achieve dynamic embedding path planning. Step 3: Establish a watermark detector. The watermark confidence of the text is evaluated by the pre-trained watermark detector to determine whether the watermark information has been successfully embedded in the text. The detection result is then used as a signal to be passed back to the reinforcement learning policy network of the reinforcement learning policy model to form a closed-loop training system. Step 4: Implement hierarchical encoding and mapping of identity information, encode specific identifiable information as watermark payload, and embed it into the text; Step 5: Deploy the watermark embedder and complete model integration, integrating the watermark embedding mechanism into the existing large language model framework with minimal modifications.

2. The controllable text watermark embedding method based on a reinforcement learning policy model according to claim 1, characterized in that, The specific steps for constructing the reinforcement learning policy model in step 1 are as follows: Step 1.1: A multi-objective fusion reward function design method is proposed to simultaneously consider text generation quality and watermark embedding feasibility. Specifically, the reward function is designed as follows: in, (1); in, This represents the reward function for multi-objective fusion, used to simultaneously consider text generation quality and watermark embedding feasibility; This represents the confidence score of the embedded text being identified as watermarked text by the watermark detector, with a value of [value missing]. The probability value is obtained by the watermark detector in step 3 after the input text is processed. The text naturalness score measures the stylistic and semantic consistency between the generated text and the output of the original large language model, and is defined as the KL divergence with the distribution of the underlying large language model. （2）； in, This represents the probability distribution of tokens generated by the current reinforcement learning policy model. The output distribution of the original large language model is shown. A smaller KL divergence indicates greater similarity and better naturalness; therefore, its negative value is used as the naturalness score. Simultaneously, corresponding weight parameters are set. and ; Step 1.2: Construct and optimize a reinforcement learning policy network Given the current generation context Under the condition of selecting token- The strategy maximizes the predefined steps. Reward function at time The training method employs the standard Policy Gradient optimization algorithm, whose gradient expression is as follows: （3）； in, Indicate steps The token selected at that time Indicates the current context state; These are the trainable parameters of the policy network. Indicates the parameter The gradient operator, i.e., the partial derivative vector; The objective function for policy optimization is represented by The gradient with respect to the parameter θ; Step 1.3: Deploy the reinforcement learning policy network trained in Step 1.2 as a token reranker. This is used to adjust the token selection logic in real time during the actual text generation process, thereby controlling the embedding behavior at the token level and selecting the optimal token according to the following formula: （4）； Where y represents any token in the candidate set. This represents the set of candidate tokens output by the large language model at step t, containing all the selected tokens. This indicates the finally selected token. For the policy network to select candidate tokens in the current state - The probability of scoring.

3. The controllable text watermark embedding method based on a reinforcement learning policy model according to claim 1, characterized in that, The specific method for establishing the watermark detector and participating in joint training feedback in step 3 is as follows: Step 3.1: Build a watermark detector based on the Transformer architecture As a discrimination module added to the entire watermark training system, the watermark detector can detect any input text. The system makes a judgment, outputs a probability score indicating the presence of a watermark, and normalizes it to a confidence score using the sigmoid function. （10）； in, The logit value output by the Transformer encoder. This represents the sigmoid activation function and its output value. This is used to measure the strength of the current text watermark signal; Step 3.2: Construct a training set using both watermarked and unwatermarked corpora, and perform supervised training. The loss function is cross-entropy. （11）； in, A label indicating whether a sample contains a watermark. It is the confidence level of the model's prediction; the supervised training process enables the watermark detector to distinguish between watermarked and non-watermarked text. Step 3.3: To form a joint optimization mechanism for watermark embedding and detection, the watermark confidence score output by the watermark detector is... It is incorporated into the reward function of the reinforcement learning policy model as an evaluation metric for the watermark embedding effect: （12）。 4. The controllable text watermark embedding method based on a reinforcement learning policy model according to claim 1, characterized in that, The specific method for implementing hierarchical bit encoding of identity information in step 4 is as follows: Step 4.1: First, extract core metadata from the generated task, including the user ID, generation timestamp, and task ID, and then use an encoding function. The UserID, Generation Timestamp, and TaskID are uniformly encoded and converted into a bitstream. ; Step 4.2: Candidate set based on each token generation time , bit stream Bitwise mapping to the token at the corresponding position; The mapping function determines the corresponding candidate token based on the bit value, maps bit 0 to the natural tokens distributed at the beginning, and maps bit 1 to the perturbed or offset subset of tokens, thus completing the information embedding; Step 4.3: A layered embedding mechanism is introduced. The watermark bitstream is divided into a macro layer and a micro layer. The macro layer is used to embed global information, while the micro layer is used for task-level information.

5. The controllable text watermark embedding method based on a reinforcement learning policy model according to claim 1, characterized in that, The specific method for step 5 is as follows: Step 5.1: The watermark embedder is encapsulated as an independent processing module. After the large language model LLM completes the output of the token probability distribution, the output sequence is reordered and selected in an interception manner. The processing module does not modify the network parameters or architecture of the main model. It is attached to the decoding process as a token selection and reordering unit to achieve fine-tuning of the output token embedding. Step 5.2: The watermark embedder is integrated through a standardized API interface call. Users load the watermark control strategy through HTTP request or local call. The input includes the generated text content, user identity information, and task configuration parameters. The watermark encoding, embedding, and control output are completed automatically.

6. The controllable text watermarking embedding method based on a reinforcement learning policy model according to claim 4, characterized in that, The encoding method in step 4.1 is set to a standard hash function, AES encrypted hash, or variable-length binary encoding. The generated bit stream is used to control the selection of specific tokens in the subsequent watermark embedding process.

7. The controllable text watermark embedding method based on a reinforcement learning policy model according to claim 4, characterized in that, The global information at the macro level in step 4.3 includes the user ID and authorization code; the task-level information at the micro level includes the context identifier or version number of the specific generated content.

Images