A watermark embedding and detecting method and system based on a large model

Abstract

The application discloses a watermark embedding and detection method and system based on a large model, wherein the method steps comprise: obtaining a real text sequence to be embedded with a watermark; constructing a watermark embedding and detection model based on a large model; and using the watermark embedding and detection model to complete watermark embedding and detection on the real text sequence. The application realizes efficient embedding and accurate detection of the watermark by directly embedding the watermark in the model training process and optimizing the detection mechanism. The application innovatively introduces a generative adversarial network and a binocular telescope structure, which not only ensures the natural fluency of the generated text, but also significantly enhances the robustness of the watermark under adversarial attacks and natural rewriting. At the same time, the low-rank adapter technology is adopted to reduce the computational cost, so that the method can be widely applied to various pre-trained large models and has strong adaptability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a watermark embedding and detection method and system based on a large model. BACKGROUND

[0002] With the wide application of large language models (LLMs) in text generation, content creation, intelligent customer service, etc., its potential risks have also attracted increasing attention. One of the most prominent problems is how to ensure the traceability and responsibility attribution of AI-generated content. Especially in today's fast and extensive information dissemination, once the AI-generated content has problems such as false information, misleading content or copyright disputes, it is crucial to accurately trace the source of these contents and determine the responsibility. Therefore, it is particularly important to develop a method that can effectively embed watermarks without affecting the naturalness of the text.

[0003] Existing traceability methods mainly fall into two categories: post-processing watermark and generation-time watermark. Post-processing watermark technology achieves this by embedding invisible but identifiable patterns in the generated text. Although this method can meet the traceability needs to some extent, it may damage the natural fluency of the text or be vulnerable to adversarial attacks. On the other hand, generation-time watermark embeds specific markers by adjusting the sampling strategy during the model's text generation process. This method is relatively more robust because it directly acts on the text generation process, but it relies on specific sampling strategies and cannot be directly applied to pre-trained models. If you try to embed watermarks into the entire model's weights during the pre-training phase, it may lead to a decline in model output quality, and it is not a feasible option for pre-trained models. SUMMARY

[0004] To solve the technical problems in the above background, the present application proposes an innovative solution - a watermark embedding and detection method and system for large language models, which directly embeds watermarks in the model training process by fine-tuning a pair of low-rank adapters (LoRA). Specifically, this method involves two key steps: first, directly modify the model's weights during training to embed watermarks, which ensures that the naturalness and fluency of the generated text are not affected; second, use another specially designed detector model to verify whether the text contains the correct watermark information, thus realizing an end-to-end learning process. This method not only avoids dependence on specific sampling strategies, but also enables even pre-trained models to be effectively added with watermarks. In addition, since the low-rank adapter technology is used, it can complete this process with relatively small computational cost while maintaining the overall performance of the model without significant impact.

[0005] To achieve the above object, the application provides a watermark embedding and detection method based on a large model, steps comprising:

[0006] Obtaining a real text sequence to be embedded with a watermark;

[0007] Constructing a watermark embedding and detection model based on a large model;

[0008] Using the watermark embedding and detection model to complete watermark embedding and detection on the real text sequence.

[0009] Preferably, a generative adversarial network is introduced into a binocular telescope structure to construct the watermark embedding and detection model; the watermark embedding and detection model comprises a generator-performer model and a discriminator-observer model;

[0010] The generator-performer model is used to generate text with a watermark, and the watermark embedding is optimized through adversarial training;

[0011] The discriminator-observer model is used to detect the watermark in the text, and the watermark detection is optimized through adversarial training.

[0012] Preferably, the binocular telescope structure uses the performer model and the observer model to evaluate the predictability of the generated text; and by comparing the perplexity difference of the two models, it is detected whether the text is generated by a machine, the binocular telescope structure The expression of the binocular telescope structure comprises:

[0013]

[0014] Wherein, LogPPL(s) represents the perplexity of the observer model for the text sequence s; logXPPL(s) represents the cross perplexity between the observer model and the performer model.

[0015] Preferably, the generator in the generative adversarial network is embedded into the performer model of the binocular telescope structure to generate the generator-performer model; wherein the generator adopts a pre-trained model and is trained on mixed data, the mixed data comprising human text, enhanced text and generated text.

[0016] Preferably, the discriminator in the generative adversarial network is embedded into the observer model of the binocular telescope structure to generate the discriminator-observer model; wherein the discriminator adopts an RNN architecture and is trained on human text and enhanced text.

[0017] Preferably, the loss function of the watermark embedding and detection model comprises:

[0018]

[0019] Wherein, λbinocular , λ natural and λ augment denote the weight of balancing different losses; denotes the adversarial loss; denotes the binocular loss; denotes the naturalness loss; denotes the data augmentation loss.

[0020] Preferably, the data augmentation technique is used to improve the robustness of the watermark under natural rewriting:

[0021]

[0022] where S aug denotes the text sequence after data augmentation; denotes the text distribution after data augmentation; denotes the expected value.

[0023] The application also provides a large model-based watermark embedding and detection system, which is used to implement the above method, comprising: an acquisition module, a construction module and an embedding and detection module;

[0024] The acquisition module is used to acquire a real text sequence to be embedded with a watermark;

[0025] The construction module is used to construct a large model-based watermark embedding and detection model;

[0026] The embedding and detection module is used to complete watermark embedding and detection on the real text sequence by using the watermark embedding and detection model.

[0027] Preferably, the workflow of the construction module includes: introducing a generative adversarial network into a binocular structure to construct the watermark embedding and detection model; the watermark embedding and detection model includes a generator-performer model and a discriminator-observer model;

[0028] The generator-performer model is used to generate text with a watermark and optimize watermark embedding through adversarial training;

[0029] The discriminator-observer model is used to detect the watermark in the text and optimize watermark detection through adversarial training.

[0030] Preferably, the binocular structure uses the performer model and the observer model to evaluate the predictability of the generated text; and by comparing the difference in perplexity of the two models, it detects whether the text is machine-generated, the expression of the binocular structure includes:

[0031]

[0032] wherein, logPPL(s) represents the perplexity of the observer model on the text sequence s; and logXPPL(s) represents the cross perplexity between the observer model and the performer model.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] The present application realizes efficient embedding and accurate detection of watermarks by directly embedding watermarks in the model training process and optimizing the detection mechanism. It innovatively introduces a generative adversarial network and a binocular telescope structure, which not only ensures the natural fluency of the generated text, but also significantly enhances the robustness of the watermark under adversarial attacks and natural rewriting. At the same time, the use of low-rank adapter technology reduces the computational cost, making the method widely applicable to various pre-trained large models, with strong adaptability. In addition, through constraint optimization and regularization strategies, the detectability of the watermark and the naturalness of the text are further balanced, effectively avoiding the problem of text quality degradation caused by over-optimization, providing reliable technical support for the traceability and responsibility attribution of AI-generated content, and having important application value and broad application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the present application, the following briefly introduces the drawings needed to be used in the embodiments, obviously, the drawings described in the following embodiments are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0036] Figure 1 The figure is a schematic diagram of the watermark embedding and detection model structure of the embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0038] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0039] Embodiment one

[0040] The present embodiment provides a watermark embedding and detection method based on a large model, the steps comprising:

[0041] S1. Obtain a real text sequence to be embedded with a watermark.

[0042] S2. Construct a watermark embedding and detection model based on a large model.

[0043] Generative adversarial networks (GANs) are introduced into the binoculars structure to construct the watermark embedding and detection model. The binoculars structure can improve the robustness of the watermark to some extent, but because watermark embedding and detection depend on the output distribution of the model, which can be disrupted by adversarial perturbations or natural rewriting, the binoculars method is still susceptible to adversarial attacks or text rewriting.

[0044] The core idea of ​​the binoculars method is to utilize two language models (called the performer model). ) and Observer Model This is used to evaluate the predictability of the generated text. By comparing the difference in perplexity between the two models, it is possible to detect whether the text was generated by a machine.

[0045] Binoculars fractional definition:

[0046]

[0047] Where logPPL(s) represents the perplexity of the observer model with respect to the text sequence s; logXPPL(s) represents the cross-perplexity between the observer model and the performer model. The high-resolution binoculars score represents the observer model's perplexity. The predictability of text sequences s was found to be low, but compared with the performer model. The highly consistent predicted distributions generally indicate that the text was machine-generated. Low binoculars scores represent the observer model. The text sequence s was found to be highly predictable, or related to the performer model. The inconsistent distribution of predicted values ​​usually indicates that the text was written by a human.

[0048] (1) Perplexity PPL:

[0049]

[0050] Where s = {x1, x2, ..., x} L} represents the text sequence; L represents the length of the text sequence (i.e., the number of tokens); x i s[:i] represents the i-th token in the sequence; s[:i] represents the first i-1 tokens in the sequence s; Representing the observer model Given s[:i], predict x. i The probability of.

[0051] (2) Cross perplexity XPPL

[0052]

[0053] where, represents the observer model represents the probability distribution over the vocabulary V given s[:i], V represents the vocabulary, containing all possible Tokens; represents the performer model represents the log probability distribution over the vocabulary V given s[:i]; <·,·> represents the inner product between two distributions, used to measure their consistency.

[0054] To solve the above problems, the embodiment proposes a strategy of embedding and detecting watermark by further optimizing the binocular telescope method. The model structure of the embodiment is as shown in Figure 1 The construction process is as follows:

[0055] (1) Introduce the generative adversarial network (GAN) into the binocular telescope method, and let the generator and discriminator optimize watermark embedding and detection in adversarial training, so that the model can resist adversarial attacks and improve the detection ability of the watermark under adversarial attacks and natural rewriting.

[0056] (2) Use data enhancement techniques (such as text rewriting, synonym replacement) to add adversarial samples and rewritten texts, i.e. enhanced texts, in the training process, to improve the robustness of the watermark under natural rewriting.

[0057] (3) If the architecture and training data of the performer model and the observer model are too similar, the perplexity difference between the two may not be able to effectively distinguish between human and machine texts. Therefore, the method proposes an architecture and data heterogenization scheme:

[0058] 1) Architecture heterogenization: different architectures are adopted for and In this embodiment, the LoRA fine-tuning architecture based on Transformer is adopted, i.e. the low-rank adapter in the figure; the RNN architecture is adopted), to capture the deep features of the generated text through structural differences.

[0059] 2) Data heterogenization: for , training on mixed data of human text + enhanced text + generated text simulates the "generator" behavior. For , only high-quality human text + enhanced text is trained to enhance the sensitivity to non-natural patterns.

[0060] Specifically, the generator and the discriminator are respectively embedded into the performer model and the observer model of the binocular telescope structure.

[0061] The generator is responsible for generating text with watermarks, which is based on the performer model and optimizes watermark embedding through adversarial training. The generator adopts a "pre-trained model" (this embodiment adopts a LoRA fine-tuning architecture based on Transformer). It should be noted that the "pre-trained model" in the LoRA fine-tuning architecture based on Transformer in this embodiment is replaceable and has wide adaptability, such as the DeepSeek series of open source large models DeepSeek, the Qwen series of Alibaba, the GLM series of ZhiChu, the Baichuan series of Baichuan Intelligence, the Llama series and Code Llama of Meta, the Phi series of Microsoft, the Gemma series and Gemini series of Google, etc.

[0062] The discriminator is responsible for detecting watermarks in text, which is based on the observer model and optimizes watermark detection through adversarial training. In this embodiment an RNN architecture is adopted.

[0063] The loss function of the model in this embodiment is as follows:

[0064] Through adversarial training of the two models, the text generated by the "generator-performer model" can be accurately detected by the "discriminator-observer model" while maintaining naturalness. At the same time, the "discriminator-observer model" needs to be able to distinguish the watermark in the real text and the generated text, while ensuring that the generated text remains fluent and natural. The optimization goal can be expressed as:

[0065]

[0066] where λ binocular , λ natural and λ augment represent the weights of balancing different losses; represents the adversarial loss; represents the binocular telescope loss; represents the naturalness loss; represents the data enhancement loss.

[0067] The above loss function will be explained as follows:

[0068] (1) Adversarial loss The goal of the generator is to generate text that can deceive the discriminator,

[0069] The goal of the discriminator is to distinguish between real text and generated text.

[0070]

[0071] where s real represents a sequence of text written by a real human; s gen represents a sequence of text generated by the generator with a watermark; represents the output of the discriminator on real text s real , representing s real represents the probability of real text; represents the output of the discriminator on generated text s gen , representing s gen represents the probability of real text; represents the log probability of real text by the discriminator, which measures the ability of the discriminator to correctly identify real text; represents the log probability of generated text by the discriminator, which measures the ability of the discriminator to correctly identify generated text. Optimization goal: Discriminator maximize i.e., correctly distinguish between real text and generated text. Generator minimize i.e., generate text that can deceive the discriminator.

[0072] (2) Binoculars loss: optimize the binoculars score to ensure that the generated text has detectable watermark features.

[0073]

[0074] (3) Naturalness loss: ensure that the generated text remains natural.

[0075]

[0076] where, represents the probability of the i-th Token predicted by the performer model given the first i-1 Tokens. x i represents the i-th Token in the sequence.

[0077] (4) Data augmentation loss: use data augmentation techniques (such as text rewriting, synonym replacement) to improve the robustness of the watermark under natural rewriting:

[0078]

[0079] where s augrepresents a text sequence after data augmentation; represents a text distribution after data augmentation; represents an expected value.

[0080] (5) Total loss function:

[0081]

[0082] wherein λ binocular , λ natural and λ augment represent the weights of balancing different losses; represents an adversarial loss; represents a binocular loss; represents a naturalness loss; represents a data augmentation loss.

[0083] The optimization strategy of the above model is as follows:

[0084] First, initialize the pre-training model weight to the performer model and the observer model, and load the generator and the discriminator; then, prepare the real text dataset and the adversarial augmented data, and apply the low rank adapter (LoRA) to reduce the training cost. In the training process, the adversarial training framework of alternating optimization is adopted: when updating the discriminator, sampling from the real data and the watermark text generated by the generator, calculating the adversarial loss and the binocular loss to optimize the discriminator parameters; when updating the generator, integrating the adversarial loss, the binocular loss, the naturalness loss (constraining the fluency of generated text based on cross entropy) and the data augmentation loss (enhancing robustness by rewriting and synonym replacement), realizing the balance between watermark embedding and naturalness. At the same time, the regularization optimization strategy (such as exponential barrier function and quadratic barrier function) is introduced to constrain the cross entropy loss, prevent the model from over-optimizing and cause the decline of text quality, and ensure the coordinated improvement of watermark detection performance and naturalness through constraint optimization. Finally, through end-to-end adversarial training, heterogeneous model architecture design and diversified data augmentation, this method realizes the efficient embedding and high robustness detection of watermark.

[0085] S3. Utilize the watermark embedding and detection model to complete the watermark embedding and detection of the real text sequence.

[0086] In the optimization process, watermark embedding and text naturalness are two competing objectives, requiring a balance between watermark robustness and text naturalness. Min-max adversarial optimization suffers from training instability and difficulty in achieving balance. On one hand, min-max optimization may lead to oscillating behavior during training, with the model alternating between overly strong and weak watermark embedding. On the other hand, hard constraints make the optimization process difficult, easily leading to poor convergence, and making it hard to find a balance between watermark detectability and text naturalness. To address these issues, this method transforms the optimization problem into a constrained optimization problem and introduces a regularization term to stabilize the training process. During optimization, the watermark strength is controlled to prevent over-optimization, while simultaneously optimizing both watermark robustness and text naturalness.

[0087] (1) Constrained optimization: The optimization problem is transformed into a constrained optimization problem, ensuring that the cross-entropy loss does not exceed a certain threshold ∈ ：

[0088] Limited by in, The difference in binoculars scores is used to maximize the detectability of the watermark; The constraint is defined as follows: the goal is to minimize the difference in binoculars scores under this constraint, ensuring that the cross-entropy loss does not exceed the threshold ∈, thereby maintaining the naturalness of the text.

[0089] (2) Regularization optimization: introduce regularization terms (such as exponential barrier function or quadratic barrier function) to prevent over-optimization.

[0090] 1) Exponential barrier function: This method imposes a strong penalty on constraint violations, ensuring the stability of the training process.

[0091]

[0092] Among them, when When the value exceeds ∈, the optimization process is strongly suppressed, thus preventing unstable updates.

[0093] 2) Quadratic barrier function: This method imposes a smaller penalty for small constraint violations and a double penalty for large violations. It allows for a certain degree of constraint violation and provides greater flexibility.

[0094]

[0095] Example 2

[0096] The embodiment also provides a watermark embedding and detection system based on a large model, comprising an acquisition module, a construction module and an embedding and detection module; the acquisition module is used to acquire a real text sequence to be embedded with a watermark; the construction module is used to construct a watermark embedding and detection model based on a large model; and the embedding and detection module is used to complete watermark embedding and detection on the real text sequence by using the watermark embedding and detection model.

[0097] How the present application solves the technical problems in actual work will be described in detail below in combination with the embodiment.

[0098] First, the acquisition module is used to acquire a real text sequence to be embedded with a watermark.

[0099] Then, the construction module constructs a watermark embedding and detection model based on a large model, and the process comprises:

[0100] The generative adversarial network is introduced into the binoculars structure to construct the watermark embedding and detection model. The binoculars structure can improve the robustness of the watermark to a certain extent, but because the watermark embedding and detection depend on the output distribution of the model, and these distributions may be destroyed by adversarial perturbations or natural rewriting, the binoculars method may still be affected by adversarial attacks or text rewriting.

[0101] The core idea of the binoculars method is to use two language models (called performer model (PerformerModel, ) and observer model (ObserverModel, ) to evaluate the predictability of the generated text. By comparing the difference in perplexity of these two models, it can be detected whether the text is machine-generated.

[0102] Binoculars score definition:

[0103]

[0104] Wherein, log PPL(s) represents the perplexity of the observer model for the text sequence s; and log XPPL(s) represents the cross perplexity between the observer model and the performer model. A high binoculars score indicates that the observer model finds that the predictability of the text sequence s is low, but is highly consistent with the prediction distribution of the performer model , which usually means that the text is machine-generated. A low binoculars score indicates that the observer model finds that the predictability of the text sequence s is high, or is inconsistent with the prediction distribution of the performer model , which usually means that the text is written by a human.

[0105] (1) Perplexity PPL:

[0106]

[0107] where s = {x1, x2,..., xL} represents a text sequence; L represents the length of the text sequence (i.e., the number of Tokens); xi represents the i-th Token in the sequence; s[:i] represents the first i-1 Tokens of the sequence s; L i represents the observer model Given s[:i], the probability of predicting xi is calculated. i

[0108] (2) Cross perplexity XPPL

[0109]

[0110] where represents the observer model Given s[:i], the probability distribution over the vocabulary V, where V represents the vocabulary containing all possible Tokens; represents the performer model Given s[:i], the log probability distribution over the vocabulary V; <·, ·> represents the inner product between two distributions, which is used to measure their consistency.

[0111] To solve the above problems, the embodiment proposes a strategy of further optimizing the binocular telescope method to embed and detect watermarks. The model structure of the embodiment is shown in Figure 1 , and its construction process is as follows:

[0112] (1) Introduce the generative adversarial network (GAN) into the binocular telescope method, and let the generator and the discriminator optimize watermark embedding and detection in the adversarial training, so that the model can resist adversarial attacks and improve the detection ability of the watermark under adversarial attacks and natural rewriting.

[0113] (2) Use data augmentation techniques (such as text rewriting and synonym replacement) to add adversarial samples and rewritten texts, i.e., augmented texts, in the training process, to improve the robustness of the watermark under natural rewriting.

[0114] (3) If the architectures and training data of the performer model and the observer model are too similar, the difference in perplexity between the two may not be able to effectively distinguish between human and machine texts. Therefore, the method proposes an architecture and data heterogenization scheme:

[0115] 1) Architecture heterogenization: to and ​​​Adopt different architectures (in this embodiment, Adopt the LoRA fine-tuning architecture based on Transformer, i.e., the low-rank adapter in the figure. Adopt the RNN architecture, capture the deep features of the generated text through structural differences.

[0116] 2) Data isomerization: for Training on mixed data of human text + augmented text + generated text, simulating the "generator" behavior. For Training only on high-quality human text + augmented text, strengthening the sensitivity to unnatural patterns.

[0117] Specifically, the generator and the discriminator are embedded into the performer model and the observer model of the binocular telescope structure.

[0118] Among them, the generator is responsible for generating text with watermarks, which is based on the performer model Optimize watermark embedding through adversarial training. The generator adopts a "pre-trained model" (this embodiment adopts a LoRA fine-tuning architecture based on Transformer). It should be noted that the "pre-trained model" in the LoRA fine-tuning architecture based on Transformer in this embodiment is replaceable and has wide adaptability, such as the DeepSeek series of open source large models, the Qwen series of Alibaba, the GLM series of Zhi Chi, the Baichuan series of Bai Chuan Intelligence, the Llama series and Code Llama of Meta, the Phi series of Microsoft, the Gemma series and Gemini series of Google, etc.

[0119] The discriminator is responsible for detecting watermarks in text, which is based on the observer model Optimize watermark detection through adversarial training, and in this embodiment adopts an RNN architecture.

[0120] The loss function of the model in this embodiment is as follows:

[0121] Through adversarial training of the two models, the "generator-performer model" generates text that is natural while being accurately detected by the "discriminator-observer model". At the same time, the "discriminator-observer model" needs to be able to distinguish the watermark in the real text and the generated text, while ensuring that the generated text remains fluent and natural. The optimization goal can be expressed as:

[0122]

[0123] where λ binocular , λ natural and λ augment represent the weights balancing different losses; represents the adversarial loss; represents the binocular loss; represents the naturalness loss; represents the data augmentation loss.

[0124] The above loss functions will be described below:

[0125] (1) Adversarial loss The goal of the generator is to generate text that can deceive the discriminator,

[0126] The goal of the discriminator is to distinguish between real text and generated text.

[0127]

[0128] where s real represents the text sequence written by real human beings; s gen represents the text sequence with watermark generated by the generator; represents the output of the discriminator for the real text s real , and s real represents the probability of real text; represents the output of the discriminator for the generated text s gen , and s gen represents the probability of real text; represents the logarithmic probability of the discriminator for real text, which measures the ability of the discriminator to correctly identify real text; represents the logarithmic probability of the discriminator for generated text, which measures the ability of the discriminator to correctly identify generated text. Optimization goal: discriminator maximize i.e. correctly distinguish between real text and generated text. Generator minimize i.e. generate text that can deceive the discriminator.

[0129] (2) Binocular loss: optimize the binocular score to ensure that the generated text has detectable watermark features.

[0130]

[0131] (3) Naturalness loss: ensure that the generated text remains natural.

[0132]

[0133] where, P (x | x ＜i denotes the first i-1 Tokens in the sequence, x i denotes the i-th Token in the sequence.

[0134] (4) Data augmentation loss: using data augmentation techniques (such as text rewriting, synonym replacement) to improve the robustness of the watermark under natural rewriting:

[0135]

[0136] where s aug denotes the text sequence after data augmentation; denotes the text distribution after data augmentation; denotes the expected value.

[0137] (5) Total loss function:

[0138]

[0139] where λ binocular , λ natural and λ augment denote the weights of balancing different losses; denotes the adversarial loss; denotes the binocular loss; denotes the naturalness loss; denotes the data augmentation loss.

[0140] The optimization strategy of the above model is as follows:

[0141] First, initialize the pre-trained model weights to the performer model and the observer model, and load the generator and discriminator; then prepare the real text dataset and the adversarial augmented data, and apply the low rank adapter (LoRA) to reduce the training cost. In the training process, an alternating optimization adversarial training framework is adopted: when updating the discriminator, samples are taken from the real data and the watermark text generated by the generator, and the adversarial loss and the binocular loss are calculated to optimize the discriminator parameters; when updating the generator, the adversarial loss, the binocular loss, the naturalness loss (based on cross-entropy constraint to generate smooth text) and the data augmentation loss (to enhance robustness through rewriting and synonym replacement) are integrated to achieve the balance between watermark embedding and naturalness. At the same time, regularization optimization strategies (such as exponential barrier function and quadratic barrier function) are introduced to constrain the cross-entropy loss, prevent the model from over-optimizing and cause the text quality to decline, and ensure the collaborative improvement of watermark detection performance and naturalness through constraint optimization. Finally, this method realizes efficient embedding and high robustness detection of watermark through end-to-end adversarial training, heterogeneous model architecture design and diversified data augmentation.

[0142] Finally, the watermark embedding and detection model is used to complete the watermark embedding and detection of the real text sequence.

[0143] In the optimization process, watermark embedding and text naturalness are two competing objectives, and a balance needs to be found between watermark robustness and text naturalness. The min-max adversarial optimization has the problems of training instability and difficulty in balancing. On the one hand, min-max optimization can lead to oscillatory behavior in the training process, with the model alternating between embedding too strong and too weak watermarks. On the other hand, hard constraints make the optimization process difficult, leading to poor convergence and difficulty in balancing between watermark detectability and text naturalness. To solve the above problems, the embodiment converts the optimization problem into a constrained optimization problem, and introduces a regularization term to stabilize the training process, control the watermark intensity in the optimization process, prevent overfitting, and optimize the robustness of the watermark and the naturalness of the text.

[0144] (1) Constrained optimization: convert the optimization problem into a constrained optimization problem to ensure that the cross-entropy loss does not exceed a certain threshold ∈ ：

[0145] Limited by where, represents the difference in binocular scores, used to maximize the detectability of the watermark; represents the constraint condition, the goal is to minimize the difference in binocular scores under this constraint condition, ensure that the cross-entropy loss does not exceed the threshold ∈, so as to maintain the naturalness of the text.

[0146] (2) Regularization optimization: introduce a regularization term (such as an exponential barrier function or a quadratic barrier function) to prevent overfitting.

[0147] 1) Exponential barrier function: this method imposes a strong penalty on constraint violations, ensuring the stability of the training process.

[0148]

[0149] where, When exceeds ∈, the optimization process is strongly inhibited, preventing unstable updates.

[0150] 2) Quadratic barrier function: small constraint violations are penalized less, while large violations are penalized quadratically. This method allows for some degree of constraint violation, providing greater flexibility:

[0151]

[0152] The above described embodiments are only to illustrate the preferred modes of the present application, and are not intended to limit the scope of the present application. Any modification and improvement made by those skilled in the art to the technical solutions of the present application without departing from the design spirit of the present application shall fall within the protection scope of the present application as defined by the claims.

Claims

1. A method for watermark embedding and detection based on large models, characterized by the steps of The method comprises the following steps: obtaining a real text sequence to be embedded with a watermark; constructing a watermark embedding and detection model based on a large model; introducing a generative adversarial network into a binocular telescope structure to construct the watermark embedding and detection model; the watermark embedding and detection model comprises a generator-performer model and a discriminator-observer model; the generator-performer model is used to generate text with a watermark and optimize watermark embedding through adversarial training; the discriminator-observer model is used to detect the watermark in the text and optimize watermark detection through adversarial training; the generator in the generative adversarial network is embedded into the performer model of the binocular telescope structure to generate the generator-performer model; wherein the generator adopts a pre-trained model and is trained on mixed data, the mixed data comprising human text, enhanced text and generated text; the discriminator in the generative adversarial network is embedded into the observer model of the binocular telescope structure to generate the discriminator-observer model; wherein the discriminator adopts an RNN architecture and is trained on human text and enhanced text; the pre-trained model adopts a LoRA fine-tuning architecture based on Transformer; the watermark embedding and detection model is used to complete watermark embedding and detection on the real text sequence.

2. The large model-based watermark embedding and detection method of claim 1, wherein, The binocular structure evaluates the predictability of the generated text using a performer model and an observer model; and detects whether the text is generated by a machine by comparing the perplexity difference of the two models, the binocular structure includes an expression of: ` wherein, represents the perplexity of the observer model on the text sequence s; represents the cross-perplexity between the observer model and the performer model. 3.The large model-based watermark embedding and detection method of claim 1, wherein, the loss function of the watermark embedding and detection model comprises: wherein, , and denote the weights balancing different losses; denotes the adversarial loss; denotes the binocular loss; denotes the naturalness loss; denotes the data augmentation loss.

4. The large model-based watermark embedding and detection method of claim 3, wherein, using data enhancement technology to improve the robustness of the watermark under natural rewriting; wherein, represents the text sequence after data augmentation; represents the text distribution after data augmentation; represents the expected value.

5. A large model based watermark embedding and detection system for implementing the method of any one of claims 1-4, characterized in that, The method comprises the following steps: an acquisition module, a construction module and an embedding and detection module.

Images