Text watermark generation method, detection method and system for black-box large language model

Abstract

The present application relates to the technical field of information security, and particularly relates to a text watermark generation method, a detection method and a system for a black-box large language model, a first text is generated based on a user input instruction and by using a black-box large language model; a watermark text is generated and output by using a pre-trained watermark embedding model, the watermark embedding model comprises an encoder for embedding representation of the first text and watermark information and a decoder for fusion output of an embedding vector, and the watermark information is data obtained by encoding user-specific metadata by using a parity check code. The present application embeds multi-bit user metadata in the model output text, fuses the watermark information and the text information in the embedding space, can realize user-level traceability, and can be applied to responsibility tracing scene applications of AI-generated false information and malicious content on the Internet in terms of capacity, accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a text watermark generation method, detection method and system for black-box large language models. Background Technology

[0002] In recent years, various institutions and enterprises have successively launched a new generation of high-performance large language models (LLMs), such as DeepSeekR1, Qwen3, and GPT-4.5. The ability of large language models to understand and generate text content has reached a new level, and people are increasingly relying on them to complete tasks such as knowledge retrieval, document summarization, and copywriting generation. However, in practical applications, there are instances where large models are used to generate academic papers, leading to academic misconduct, or to generate malicious emails, false statements, or news and spread them on the internet. The abuse of large models can damage the internet ecosystem. If copyright and user-related metadata could be embedded as watermarks in the text before the content generated by the large model is submitted to the user, and extracted when necessary, this could serve as evidence for tracing the source. Text watermarking technology evolved from text steganography. Compared to traditional text steganography, which primarily focuses on the successful transmission of confidential information between communicating parties, current text watermarking technology embeds latent patterns and features into the text. On the one hand, it must ensure the watermark's concealment; on the other hand, it must ensure that the text's semantics, layout, or form remain almost unchanged. Watermarking techniques for image and audio modalities have been extensively studied. However, due to the inherent discreteness of text modal information, changes to characters can easily lead to semantic and layout changes. This makes watermarks difficult to implant and easy to tamper with, requiring researchers to design more complex and robust watermarking schemes.

[0003] Existing text watermarking technologies can be divided into two categories: zero-bit watermarking and multi-bit watermarking. Zero-bit watermarking forms watermark signals by changing document format, replacing synonyms or sentence structures, or interfering with the text generation process of large models. This can be achieved by adding triggers during training, adding biases after dividing the red-green word list during logits generation, or designing sampling strategies during token sampling. However, these zero-bit watermarks can only prove the existence of the watermark after extraction; because they do not encode or embed more information, they cannot provide user-level copyright statements or source tracing. For multi-bit watermarking, simple synonym replacement is currently not robust to text rewriting. The ABI scheme achieves watermarking based on a multi-color list during the logits generation of large models, while the Waterfall scheme adds orthogonal perturbations. Their schemes are robust but require white-box acquisition of large models. Generative multi-bit watermarking schemes are based on end-to-end training using transformers, which can embed watermark information into the text while maintaining semantic and expressive coherence, and use neural networks rather than statistical methods for multi-bit watermark message extraction. However, issues such as the training cost of watermark implantation models, the length of watermark message sequences, and extraction accuracy make it difficult for generative watermarking schemes to be implemented. Summary of the Invention

[0004] To address the issues of limited capacity and low extraction accuracy of AI-generated content (AIGC) watermarks on the Internet, this invention provides a text watermark generation method, detection method, and system for black-box large language models. By embedding multi-bit user metadata into the model's output text after the fact, the watermark information and text information are fused in the embedding space, enabling user-level traceability capabilities. This improves the usability of practical deployment in terms of capacity, accuracy, and robustness.

[0005] According to the design scheme provided by this invention, on the one hand, a text watermark generation method for black-box large language models is provided, comprising:

[0006] Receive user input instructions, and generate the first text based on the user input instructions and a black-box large language model;

[0007] A pre-trained watermark embedding model is used to generate and output watermarked text. The watermark embedding model includes an encoder for embedding representation of the first text and watermark information and a decoder for fusing the embedding vectors. The watermark information is data encoded using parity check codes to encode user-specific metadata.

[0008] As a text watermark generation method for black-box large language models in this invention, the first text and watermark information are further encoded, including:

[0009] The first text is segmented into discrete word units, forming a word unit sequence;

[0010] The encoder maps the word sequence to the text embedding vector in the corresponding embedding space;

[0011] A linear layer is used to map the watermark information to a watermark embedding vector in the same embedding space.

[0012] As a text watermark generation method for black-box large language models in this invention, the embedded vectors are further fused and output, including:

[0013] Add the text embedding vector and the watermark embedding vector together;

[0014] The summed embedding vectors are input into a pre-trained decoder, which then generates and outputs the watermarked text.

[0015] As a text watermark generation method for black-box large language models in this invention, the watermark embedding model is further trained using the cross-entropy loss function, and a low-rank decomposition adaptation method is used for model fine-tuning during the training process.

[0016] Furthermore, the present invention also provides a text watermark detection method, which further includes:

[0017] A pre-trained watermark discriminator is used to determine whether a text to be processed on the Internet has been watermarked.

[0018] If there is no added watermark in the text to be processed, it is determined that the text to be processed on the Internet is not related to the users of the large language model service provider.

[0019] If the text to be processed contains an added watermark, the watermark information in the text to be processed is extracted using a pre-trained watermark extractor, and user metadata is extracted from the watermark information through parity check code decoding operation. Based on the user metadata, the target user is matched from the users of the large language model service provider. The watermark extractor and the watermark embedding model of the above text watermark generation method are trained end-to-end to simulate the watermark embedding and extraction process.

[0020] As a text watermark detection method of the present invention, the watermark discriminator training process further includes:

[0021] Construct sample data with / without watermark labels using the trained watermark embedding model;

[0022] The watermark discriminator is trained in a supervised manner using sample data.

[0023] As a further step in the text watermark detection method of the present invention, user metadata is extracted from the watermark information through a parity check code decoding operation, including:

[0024] The user metadata in the watermark is extracted step by step using the bit-flipping method to decode and the LDPC decoding algorithm.

[0025] Furthermore, the present invention also provides a text watermark detection system, comprising: a watermark discrimination module and a watermark recognition module, wherein,

[0026] The watermark discrimination module is used to determine whether the text to be processed on the Internet has been watermarked using a pre-trained watermark discriminator. If there is no watermark in the text to be processed, it is determined that the text to be processed on the Internet is not related to the user of the large language model service provider; if there is a watermark in the text to be processed, watermark recognition is performed.

[0027] The watermark recognition module is used to identify and extract watermark information in the text to be processed using a pre-trained watermark extractor. It extracts user metadata from the watermark information through parity check code decoding operation, and matches the target user from the users of the large language model service provider based on the user metadata. The watermark extractor and the watermark embedding model in the above text watermark generation method adopt an end-to-end training process to simulate the training of watermark embedding and extraction.

[0028] The beneficial effects of this invention are:

[0029] 1. This invention models the watermarking process as a sequence-to-sequence task, fuses watermarking information with text information in the embedding space, and uses LoRA to efficiently fine-tune the pre-trained model of the encoder-decoder architecture. It supports the post-processing embedding of multi-bit user metadata into the model output text, realizing user-level traceability capabilities and significantly improving the usability of actual deployment in terms of capacity, accuracy, and robustness.

[0030] 2. This invention uses LoRA low-rank matrix adaptation to lightweight fine-tune the pre-trained model, improving training efficiency while maintaining semantic consistency; it models the watermark embedding-extraction process as a person in a communication channel, uses error correction redundancy coding for text watermarking, and uses low-density parity-check codes (LDPC) to improve watermark accuracy; by constructing supervised training data of "watermarked / unwatermarked" text pairs, a dedicated discriminator is trained, providing a reliable false detection control mechanism for large-scale practical deployment.

[0031] 3. This invention is based on an end-to-end training architecture of encoding-reparameterization-decoding, which can embed multi-bit user metadata after the model generates text, enabling user-level content traceability. By introducing low-density parity-check codes (LDPC) for watermark encoding, the anti-interference and error correction capabilities are significantly enhanced; combined with low-rank adaptive (LoRA) for lightweight fine-tuning of the pre-trained language model, deployment efficiency is improved; in addition, an independent watermark discriminator module is designed to effectively suppress false detections. Experiments were conducted on multiple datasets. Experimental results show that: (a) for short texts of 80 tokens, this scheme expands the watermark capacity to 4 times that of the state-of-the-art scheme, and can embed up to 32 bits of user-related metadata information; (b) at the same time, the user traceability accuracy (UTA) reaches more than 91% under 8~32 bit load, and the bit extraction accuracy (BEA) still remains above 82% under attacks such as token addition, deletion, modification, paraphrasing and re-watermarking, showing strong robustness; (c) the training time is shortened by 5.5 times, and it is better than or close to the optimal baseline in terms of inference speed and memory usage; (d) the constructed watermark discriminator can effectively prevent false detections with an AUROC of more than 0.998. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the text watermark generation process for a black-box large language model in the embodiment.

[0033] Figure 2 This is a schematic diagram illustrating the principle of responsibility tracing for AIGC content on the Internet in this embodiment;

[0034] Figure 3 This is a schematic diagram of the model training process in the embodiment;

[0035] Figure 4 This is a schematic diagram of UTA comparison under different datasets and different bits in the embodiment;

[0036] Figure 5 This is a schematic diagram illustrating the comparison of semantic preservation and BER on different CCs and datasets in the embodiments;

[0037] Figure 6 This is a schematic diagram of the ROC curve of the watermark recognizer in the embodiment;

[0038] Figure 7 This is a schematic diagram illustrating the robustness comparison under A1 attacks at different ML stages in the embodiments;

[0039] Figure 8 This is a schematic diagram showing the robustness comparison under different attack ratios at 8 bits in the example;

[0040] Figure 9 This example compares the robustness of different attack ratios under 16-bit conditions in the embodiment.

[0041] Figure 10 This is a schematic diagram illustrating the robustness comparison under A2 attack in the embodiments;

[0042] Figure 11 This is a schematic diagram illustrating the robustness comparison under A3 attack in the embodiments;

[0043] Figure 12 This illustration shows the loss reduction during training under different LoRA settings and watermark extractor frameworks in the example. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the invention will be further described in detail below with reference to the accompanying drawings and technical solutions.

[0045] As large language models become more widespread and the quality of their generated text continues to improve, the risk of their misuse for generating false information, malicious content, and academic misconduct is becoming increasingly prominent, necessitating the development of effective source tracing and copyright protection technologies. Text watermarking technology, as one solution, aims to embed metadata such as user identity into the generated text in an invisible manner. However, existing methods often face challenges such as limited watermark capacity, low extraction accuracy, and high training overhead, making it difficult to achieve a good balance between practicality and robustness. Therefore, this invention provides embodiments, see [link to embodiments]. Figure 1 As shown, a text watermark generation method for black-box large language models is provided, including:

[0046] S101. Receive user input instructions, and generate the first text based on the user input instructions and using a black-box large language model;

[0047] S102. Generate and output watermarked text using a pre-trained watermark embedding model. The watermark embedding model includes an encoder for embedding representation of the first text and watermark information and a decoder for fusing and outputting the embedding vector. The watermark information is data encoded using parity check codes to encode user-specific metadata.

[0048] In this embodiment, the source tracing and copyright declaration of text generated by the large language model are mainly achieved through text watermarking technology. Text watermarking technology enables the output of the large model to have hidden patterns and features that are difficult to detect with the naked eye, which can both promote AIGC text detection and facilitate copyright declaration and source tracing. Unlike other modalities such as images or audio, text is discrete. Arbitrary changes to words, symbols, or order can easily cause changes in semantics and format, making it difficult to implant and easy to detect and tamper with. This is the root cause of the difficulty and inherent vulnerability of current text watermarking technology.

[0049] Specifically, the encoding of the first text and watermark information can be designed to include:

[0050] The first text is segmented into discrete word units, forming a word unit sequence;

[0051] The encoder maps the word sequence to the text embedding vector in the corresponding embedding space;

[0052] A linear layer is used to map the watermark information to a watermark embedding vector in the same embedding space.

[0053] Then, the text embedding vector and the watermark embedding vector are added together; the added embedding vector is input into the pre-trained decoder, which generates the watermarked text and outputs it.

[0054] Text generated by LLM Metadata watermark information related to copyright and users As input, the text is processed by a watermark embedding algorithm to obtain the watermarked text. This is the output.

[0055] During training, watermark embedding is treated as a sequence-to-sequence rewriting (paraphrasing) problem, using a pre-trained Encoder-Decoder architecture model. (Text) After tokenization, the vectors are mapped by the Encoder to the corresponding embedding space. Simultaneously, use a linear layer (Linear) to store the watermark information. Mapped to an embedding vector in the same embedding space The two embedding vectors are added together and then processed by the decoder to form the watermark text. , here This represents the vector addition operation in the embedding space.

[0056] To accelerate the convergence of the watermark embedding model during training, Low-Rank Adaptation (LoRA) can be used to efficiently fine-tune the parameters of the pre-trained model. Let the weight matrix of a given layer of the pre-trained model PLM be denoted as . The parameters that need to be updated for fine-tuning are: The process for fine-tuning using the LoRA method involves freezing the PLM parameters. Only the reduced-dimensional matrix A and the increased-dimensional matrix B need to be trained, i.e. ,in As a scaling factor, during forward propagation, only the parameters of BA and PLM need to be changed. Superposition. During inference, the input and output dimensions of the model remain unchanged; for a given input... The output after LoRA fine-tuning simply requires adding the output of the frozen layer and the output of low-rank matrix inference, i.e. .because and The rank r of the matrix is ​​very small, and the number of parameters involved in training is... It is far less than what was originally needed to be updated. Parameter quantity Therefore, it can accelerate model convergence while maintaining almost the same performance, greatly saving the overhead required for fine-tuning.

[0057] The goal of training the watermark embedding process is to obtain watermarked text that is almost semantically unchanged from the original text, thus not affecting user experience. A standard cross-entropy loss function can be used, expressed as:

[0058] .

[0059] To improve the robustness of watermark information extraction, the watermark implantation-extraction process is handled from a communication perspective. Before implanting the watermark, the watermark information can be first processed. Encode using Low-Density Parity-Check (LDPC) codes.

[0060] Let p be a binary watermark message. During encoding, it is necessary to A redundant code of length q bits was then added. , forming a total The codeword of the position is At this time, it is called for -LDPC encoding algorithm, where for A generating matrix. And It can be determined by the check matrix. Completely certain, yes .

[0061] According to coding theory, if the total length of the codeword is given During watermark extraction, at most t bits of error may occur, meaning the maximum error correction capability must be t. The Hamming Bound (Ball Packing Bound) provides a lower limit for the length of redundant bits in linear block codes at this point in coding theory.

[0062] Given And t, we only need to design the calculation of the smallest integer. This makes the above inequality true.

[0063] In other words, to correct more errors, more bits of valid information (p) must be sacrificed when channel capacity is limited. During inference, the maximum number of error bits when the watermark extraction most frequently results in an error is defined as t. A redundant code of q = 2t + 1 bits is designed, with p bits of message... Random samples are then used as watermark information for embedding and extraction.

[0064] Furthermore, based on the above-described watermark generation method, this embodiment of the invention also provides a text watermark detection method, including:

[0065] A pre-trained watermark discriminator is used to determine whether a text to be processed on the Internet has been watermarked.

[0066] If there is no added watermark in the text to be processed, it is determined that the text to be processed on the Internet is not related to the users of the large language model service provider.

[0067] If the text to be processed contains an added watermark, the watermark information in the text to be processed is extracted using a pre-trained watermark extractor, and user metadata is extracted from the watermark information through parity check code decoding operation. Based on the user metadata, the target user is matched from the users of the large language model service provider. The watermark extractor and the watermark embedding model of the above text watermark generation method are trained end-to-end to simulate the watermark embedding and extraction process.

[0068] like Figure 2 As shown, in the process of tracing responsibility for AIGC content on the internet, the main participants in the interaction are user A and the large model service provider B. User A is the user of the text content, that is, the party who needs to be held accountable for disseminating harmful content. B possesses a black-box large model, providing content generation services for A, and can be designed with a watermark embedding model. Watermark recognition model Watermark recognition model The interaction process between A and B can be summarized as follows:

[0069] 1. User A sends a request containing prompt X to Large Language Model Service Provider B.

[0070] 2. Service provider B submits X to a large model that it can only access in a black-box manner.

[0071] 3. B receives the generated content T and then uses the watermark embedding algorithm. The content T' formed by embedding multi-bit information M to create a watermark, i.e. M contains user-specific metadata, thus providing traceability.

[0072] 4.B will Send to A.

[0073] 5.A published , circulating May be modified and thus consistent with There is a discrepancy.

[0074] 6. If If deemed harmful or infringing, the relevant party will The decision is left to B. First, B needs to perform watermark identification. ,like Then directly determine The spread of this information is unrelated to B's users; if ,So:

[0075] 7.B. Extract the watermark. ,like If user A is found responsible, the relevant parties will take further action.

[0076] Watermarks should be successfully implanted without affecting the semantics of the original text. Watermarks should be successfully extracted, and the user-specific metadata they carry should be usable for verification and accountability. The cost of implanting and extracting watermarks should be low. For trainable watermarks, the cost of training for deployment should also be low. Modifications to the watermarked text should not affect the successful extraction of the watermark.

[0077] Assume the adversary is an ordinary internet user who can access and upload resources and obtain services from B, but cannot obtain the unwatermarked raw output of B's ​​black-box model, nor can they obtain B's model group { , , The adversary's aim is to spread false or malicious content on the internet, or to use content generated by large models as academic papers or publications.

[0078] The threatening methods employed by an adversary can be summarized as follows:

[0079] A1. Text Modification Attack: The adversary suspects the existence of a watermark, but wants to exploit the text generated by B. The semantics of the text provided by B Adding, deleting, or replacing synonyms to alter the wording is an attempt to undermine the traceability originally provided by the watermark.

[0080] A2. Paraphrasing Attack: The adversary uses a large language model to manipulate the text generated by B. They paraphrased and rewrote the text in an attempt to destroy the watermark it contained.

[0081] A3. Heavy Watermark Attack: The adversary uses similar methods available to them to launch a heavy watermark attack. The watermark was re-added, causing confusion with the original watermark.

[0082] like Figure 3 As shown in this embodiment, the training process for watermark embedding and watermark extraction uses the text of the training dataset. Watermark information obtained by simulating metadata related to user attributes through random sampling. As input, the watermark information extracted by the extractor As an output, this entire process is an end-to-end training process. For the watermark discriminator designed in this scheme, it undergoes additional supervised training on a dataset containing "watermarked / unwatermarked" labels constructed from the input and output of the trained watermark embedding model. During the inference phase, compared to the training phase, the watermark embedding model, after deployment, requires additional sampling operations to output the watermarked text. For text from which watermarks need to be extracted, the watermark must first be identified by a watermark detector before it is processed by the watermark extractor to output the watermark. .

[0083] In the watermark discriminator training process, sample data containing watermark labels / without watermark labels are constructed using the trained watermark embedding model; the watermark discriminator is then trained in a supervised manner using the sample data.

[0084] Given that watermark extractors will extract watermarks from any text (regardless of whether the watermark is present), when users face a large amount of text requiring watermark verification, it is necessary to first determine whether the text has a watermark to prevent misjudgment and accountability errors. Therefore, an independently trained discriminator is used to determine whether the text has a watermark, and watermark extraction will only be performed after the determination is successful.

[0085] During training, the original text and watermarked text To construct the training dataset, annotations are performed; the original text annotations are used. The value is 0 for text containing the watermark and 1 for text containing the watermark. Then, a pre-trained Encoder model is used for supervised fine-tuning, transforming the watermark recognition problem into a binary classification problem. The loss function used is the cross-entropy loss function described in the equation, where... This represents the label output by the watermark recognizer. A prediction of 1 indicates that the watermark is present, and 0 indicates that the watermark is absent.

[0086]

[0087] After the watermark is embedded, Instead of using activation functions and sampling, we use the Gumbel-Softmax method for reparameterization, which makes watermark embedding-watermark extraction an end-to-end training process during training.

[0088]

[0089] Specifically, Gumbel-Softmax employs argmax one-hot encoding in the forward propagation and a softmax differentiable operation that approximates argmax in the backpropagation. During training, it takes the logits generated after watermark embedding as input, maps them to the dimension of the embedding space through a linear layer, and then passes them to the watermark extractor for processing.

[0090] The watermark extractor is implemented using an Encoder-Only architecture, constructed from a pre-trained Encoder model and linear layers. During training, the input is... The latent spatial features are used as the output, which is the extracted watermark. To ensure that the extracted watermark is identical to the implanted watermark, the loss function used is the mean absolute error loss function, i.e. .

[0091] Corresponding to the encoding process of watermark information, LDPC decoding of the watermark information needs to be performed after watermark extraction. Let the information extracted by the watermark extractor before LDPC decoding be denoted as... Watermark extraction may randomly cause tbit errors, and the i-th bit error is denoted as . ,Right now At this time, there must be .

[0092] Decoding can be performed using the bit-flipping method. First, calculate each... The relevant verification equation is not satisfied. Number of Then it flipped over the threshold. of corresponding Iterate repeatedly until Finally, output .

[0093]

[0094] Furthermore, based on the above-described watermark generation and detection methods, this embodiment of the invention also provides a text watermark detection system, comprising: a watermark discrimination module and a watermark recognition module, wherein,

[0095] The watermark discrimination module is used to determine whether the text to be processed on the Internet has been watermarked using a pre-trained watermark discriminator. If there is no watermark in the text to be processed, it is determined that the text to be processed on the Internet is not related to the user of the large language model service provider; if there is a watermark in the text to be processed, watermark recognition is performed.

[0096] The watermark recognition module is used to identify and extract watermark information in the text to be processed using a pre-trained watermark extractor. It extracts user metadata from the watermark information through parity check code decoding operation, and matches the target user from the users of the large language model service provider based on the user metadata. The watermark extractor and the watermark embedding model in the above text watermark generation method adopt an end-to-end training process to simulate the training of watermark embedding and extraction.

[0097] To verify the effectiveness of this solution, the following explanation is based on experimental data:

[0098] Four datasets were used to test the performance of the watermarking and extraction frameworks on datasets containing human-written content and AI-generated content, respectively. WikiText-2 is a dataset of human-written encyclopedic materials obtained from Wikipedia. HC3 is a dataset consisting of answers generated by Chat-GPT on a question-and-answer platform. ChatGPT Abstract and Human Abstract are a pair of datasets containing paper abstracts and GPT-3.5 Turbo-rewritten abstracts, respectively. Subsequently, 80% of each dataset was randomly used as training data, and the remaining 20% ​​was used as test data.

[0099] To evaluate the practicality of this scheme, experiments were conducted on watermark extraction, text quality, and scheme overhead. For watermark extraction capability, watermark extraction accuracy and watermark bit accuracy were used as metrics; for text quality, BERT-score and BLEU-4 metrics were used; and for scheme overhead, the storage space and time overhead required during training and inference were measured.

[0100] Metadata Length (ML): This scheme embeds user information-related metadata as watermarks into the text. The metadata space size is defined as the length of the binary metadata information that the scheme supports embedding.

[0101] Channel Capacity (CC): Since this scheme uses LDPC encoding, the actual number of embedded binary information bits is the binary information codeword after LDPC encoding, which is more than the metadata length (the information bits used as codewords). The binary metadata used by other baseline schemes is the actual embedded information, i.e., ML == CC.

[0102] User traceability accuracy (UTA): When the number of entries is... On the test dataset, the extracted metadata watermark information can correctly trace the user's data entries. Proportion, that is .

[0103] Bit Extraction Accuracy: When the number of entries is... On the test dataset, the extracted n-bit watermark bit string of length CC is compared bit by bit with the embedded n-bit watermark bit string of length CC. The number of matching bits for entry i is determined. Mean and .

[0104] BERT-score: The cosine similarity between the word vectors of the text output by the black-box model (reference text) and the watermarked text (candidate text). The Roberta-large model is used for word embedding, and the F1 score of BERT-score is used to measure the semantic consistency of the text before and after watermarking.

[0105] BLEU-4: Measures the degree of n-gram overlap between the watermarked text and the text output by the black-box model, where "4" indicates that n=4, i.e., considering the average matching scores of 1-gram, 2-gram, 3-gram, and 4-gram. Previous work mentioned that a BLEU-4 score greater than 0.15 can be considered a coherent text transformation.

[0106] REMARK-LLM, Waterfall, and Where am ifrom are used as the baselines for multi-bit watermarking.

[0107] REMARK-LLM: An encoder-decoder framework is trained in an end-to-end manner. During watermark embedding, user metadata is added in the same embedding space as the text. The decoder generates watermarked text, which is then submitted to a transformer-based watermark decoder for watermark extraction.

[0108] Waterfall: The original text is rewritten using Llama-2 to generate semantically similar text. During the rewriting process, based on user metadata, the vocabulary is permuted using vocabulary space permutation, an orthogonal function is added as a watermark signal, and the perturbed logits are mapped back to the original space and sampled to generate tokens. During watermark extraction, an average cumulative distribution is constructed for each token in the text, and the inner product is calculated with the orthogonal function corresponding to the user metadata. If a threshold is exceeded, watermark verification is completed.

[0109] Where am I from: When embedding the watermark, the user's metadata is used to divide the preference list, and a bias is added to the token logits belonging to this list. This makes it more likely to be selected after softmax. During watermark extraction, the hit rate of preferences is statistically analyzed. The preference list for each user during watermark generation is simulated, and the frequency of the token appearing in their preference list within the text is counted. Finally, a "confidence score" is calculated, and the user with the highest score is selected as the source.

[0110] I. Validity Analysis

[0111] The user traceability accuracy (UTA) of our proposed scheme was compared with that of three other baseline schemes on four datasets. First, the data entry length for each scheme was limited to 80 tokens, and the user space lengths tested were 8 bits, 16 bits, and 32 bits. For our proposed scheme and the REMARK-LLM scheme, training was performed on each dataset first, followed by testing on each dataset separately. For the other two training-free schemes, Waterfall and where am i, inference was performed directly on the four datasets. However, the user space length of the Waterfall scheme is limited by the vocabulary size, supporting a maximum watermark length of approximately 14.98 bits. This length was used for comparison with the 16-bit length of other schemes. When embedding 32 bits of information, it was marked as unsupported. For the where am i scheme, its computational complexity is high, and after embedding 32 bits, effective extraction is not possible for a considerable period; therefore, it was also marked as unsupported.

[0112] Figure 4 The user traceability accuracy (UTA) of this solution is demonstrated. This solution exhibits high effectiveness across four datasets and three user space lengths, expanding the user space size. (Length increased by 4 times). With a user space length of 8 bits, both this scheme and the waterfall scheme can accurately locate over 90% of users, while REMARK-LLM performs inconsistently across different schemes, similar to the where am i scheme, averaging less than 70% accurate user location. With a user space length of 16 bits, this scheme still maintains over 90% accuracy, while the where am i scheme's accuracy drops by an average of 17% across four datasets, and REMARK-LLM and the waterfall scheme completely lose their effectiveness at this point, with accuracy almost reaching 0%. With a user space length of 32 bits, only this scheme maintains over 96% accuracy, while the other baseline schemes are ineffective.

[0113] II. Channel Quality (Actual Number of Bits Implanted) Analysis

[0114] Three metrics—BEA, bertscore, and BLEU-4—were used to examine the proposed scheme and three baseline models' capabilities in embedding and retrieving information (8-64 bits) and the impact of watermarks on text content. The token count was set to 80. The `waterfall` and `where am i` schemes do not support embedding information exceeding 32 bits. Figure 5 As shown in the figure, the top two indicators in each category are highlighted in red for 8-bit and 16-bit solutions, while only the best indicator is highlighted in red for solutions of 32 bits and above.

[0115] Regarding the ability to implant and extract information, when implanting 8 bits of information, this scheme and Waterfall maintain a bit extraction accuracy of over 95%, REMARK-LLM is slightly worse but still maintains over 90%, and Where Am I is the worst. When the implanted information is doubled to 16 bits, the accuracy of this scheme still remains above 90%, the accuracy of REMARK-LLM decreases by about 24%, Waterfall almost fails in message implantation and extraction, and the message extraction result of the Where Am I scheme is almost random guessing (50%). For 32-bit, 48-bit, and 64-bit schemes, the message implantation and extraction capabilities of this scheme maintain an advantage of about 20% over the REMARK-LLM scheme. In summary, this scheme has strong message implantation and extraction capabilities and high stability.

[0116] Regarding the impact of watermarking on the original text, we focus on the BERT score and BLEU-4 metrics. For the 8-bit and 16-bit schemes, our scheme and REMARK-LLM treat the watermarking task as a sequence-to-sequence task, and their BERT scores are better than the other two baseline schemes. Because the `waterfall` and `where am i` schemes transform the logits of the large model, their vocabulary sampling is similar to the word selection and results of the original large model sampling. Therefore, the overlap rate between words in the watermarked text and words in the original text is relatively high, resulting in higher BLEU-4 scores. The BLEU-4 values ​​of our scheme and the REMARK-LLM scheme are also greater than 0.15, which can be considered as semantically consistent with the original text. For data with 32 bits or more, the BERT score difference between this scheme and REMARK-LLM consistently remained within 0.04. The distance between the sentences before and after watermarking in the embedding space was similar for both schemes. While REMARK-LLM generally performed better on the BLEU-4 metric, it exhibited unstable performance with scores below 0.1 on the wiki dataset. Except for the case where this scheme achieved a BLEU-4 score of 0.1483 with 64 bits embedded in the Human Abstract data, the scores remained consistently above 0.15, maintaining the original text semantics and demonstrating stable performance.

[0117] III. Efficiency Analysis (Inference Time, Memory)

[0118] The average inference time (seconds) and maximum memory usage (GB) of various schemes on the wikitext dataset with 80 tokens and 200 samples were examined. The results are shown in Tables 1 and 2. The waterfall and where am i schemes do not support watermarking and extraction of 32 bits or more, and are marked in the table.

[0119] The inference time results show that when embedding and extracting 8 bits of information, this scheme has a 5.5-fold advantage over the REMARK-LLM scheme and a 20-fold advantage over the other two baseline schemes. When embedding and extracting 16 bits of information, the overhead of this scheme is the same as the REMARK-LLM scheme, and it has a 23-fold and 37-fold improvement over the other two schemes, respectively. When the embedding and extraction length is 32 bits, this scheme is about 1 second slower than the REMARK-LLM scheme, with comparable overhead. Waterfall uses a large model size for watermark embedding, resulting in high time overhead. The where am i scheme requires traversing the user space for one-by-one detection when detecting watermarks, so its time consumption increases with the increase of the user space.

[0120] Table 1. Comparison of inference time overhead between this scheme and the baseline scheme under different machine learning scenarios.

[0121]

[0122] Regarding the memory overhead of inference time, the Waterfall scheme has a memory overhead of about 10 times that of other schemes, and it uses a large number of model parameters, which is a significant disadvantage. In contrast, the memory overhead of our scheme is about 750M less than that of REMARK-LLM and where ami, which is not much different.

[0123] Table 2 Comparison of inference memory overhead between this scheme and the baseline scheme under different ML scenarios.

[0124]

[0125] IV. Performance Analysis of Watermark Recognition Device

[0126] Positive samples of the watermark recognizer are watermarked text, and negative samples are unwatermarked text. For the watermark recognizer's decision threshold t, if its output probability value is greater than the threshold t, it is determined to be a positive sample; if the output probability value is less than t, it is determined to be a negative sample.

[0127] A watermark recognizer was trained on four datasets containing 8-bit, 16-bit, and 32-bit watermarks, respectively. The ROC curves and AUROC values ​​are shown below. Figure 6 As shown, the AUROC of the watermark recognizer can reach a range of over 0.9964 under any parameter settings, and the average value of 0.9987 fully demonstrates the effectiveness of the watermark recognizer.

[0128] V. Threat (Robustness) Analysis

[0129] 1. Robustness analysis under A1 attack

[0130] The attack ratio is the percentage of adversary-modified tokens in the watermarked text relative to the total number of tokens in the watermarked text. In the experiment, the ratio of A1 attacks (addition, deletion, and modification attacks) was initially fixed at 10%. The BEA and UTA of this scheme and the baseline scheme were tested at 8, 16, and 32 bits. Waterfall had low accuracy at its highest watermark capacity (approximately 14.98 bits), so only the 8-bit case was examined. The 32-bit case, which is not supported by the Where Am I scheme, was also not examined.

[0131] like Figure 7As shown in the figure, for the case of 8-bit insertion and extraction, this scheme exhibits extremely high accuracy and robustness against add, delete, and modify attacks, with BEA and UTA remaining almost at 100%, while the accuracy of other baseline schemes decreases to varying degrees. For the 16-bit case, the BEA of this scheme remains stable, while the UTA decreases by about 10%, but still remains above 80%, indicating acceptable effectiveness. Although the other two schemes are relatively robust, their effectiveness is not strong. The REMARK-LLM scheme has less than 3% UTA, and the where am i scheme has BEA and UTA at around 50%, similar to random guessing. For the 32-bit case, this scheme can extract information with over 82% BEA while resisting A1 attacks, exceeding the REMARK-LLM scheme by about 12%. However, the UTA of this scheme shows a significant decrease of 85% after being subjected to an A1 attack.

[0132] Subsequently, to further examine the robustness of each scheme, the word attack ratios for three attack modes—addition, deletion, and modification—were varied under the conditions of inserting and extracting 8-bit and 16-bit information, and the average extraction accuracy of each scheme was tested. For example... Figure 8 As shown, for the 8-bit case, this scheme can ensure almost 100% extraction accuracy while remaining robust to attacks of various types and intensities. The REMARK-LLM and waterfall schemes also remain robust to attacks of various intensities, but their overall accuracy differs from this scheme by approximately 5%. The Where am i scheme exhibits the worst robustness and accuracy.

[0133] like Figure 9 As shown, for the 16-bit case, this scheme still outperforms the other two schemes with a BEA accuracy of approximately 94% and high robustness under attacks of various A1 ratios. The robustness of REMARK-LLM is comparable to this scheme, but its BEA lags behind this scheme by about 10%. The Where am i scheme has the lowest BEA value and is more vulnerable to deletion and replacement attacks.

[0134] 2. Robustness Analysis under A2 Attack

[0135] The same T5 model and the command "paraphrase" were used for each scheme to paraphrase the watermarked text, and then the text was submitted for watermark extraction.

[0136] like Figure 10As shown, at 8 bits, the waterfall scheme is the most robust because its watermark signal is distributed across multiple n-gram units, independent of specific word order or position. The present scheme is second, maintaining approximately 35% UTA while outperforming the REMARK-LLM and where am i schemes with BEA advantages of 30% and 35%, respectively. At 16 bits, the present scheme and the REMARK-LLM generative schemes almost lose usability. where am i, based on synonym substitution and syntactic reconstruction, still maintains 25% usability, but its BEA is only half that of the two generative schemes. At 32 bits, the present scheme and the REMARK-LLM scheme are equally robust, both losing usability after an attack.

[0137] 3. Robustness analysis under A3 attack

[0138] After embedding different watermarks into the watermark results of each scheme using their respective methods, the results are then submitted to a watermark extractor for watermark extraction. The accuracy is then analyzed using the original watermark and the attacked watermark as labels.

[0139] A successful defense against a watermarking attack is defined as the extracted content of the watermarked text being identical to the original watermark. For example... Figure 11 As shown, for the 8-bit case, the Waterfall scheme, due to its use of a family of orthogonal functions as the watermark signal, exhibits the best accuracy for both the original and re-watermarked watermarks because the perturbation functions of different users are mutually orthogonal and their interference is minimal. However, after a re-watermarking attack, the latent space of the watermarked text changes significantly. Both the proposed scheme and the REMARK-LLM scheme show poor robustness, but the proposed scheme achieves high accuracy against attacked watermarks, indirectly demonstrating its effective watermark embedding and extraction. For the 16-bit case, both the proposed scheme and the REMARK-LLM scheme show poor robustness. The "where am I from" scheme exhibits the highest UTA, while the BEA of the proposed scheme and REMARK-LLM exceeds the UTA by approximately 15%. For the 32-bit case, both the proposed scheme and the REMARK-LLM scheme lose usability after the attack, but the BEA of the proposed scheme is approximately 20% higher than that of REMARK-LLM.

[0140] VI. Ablation Experiment Analysis

[0141] 1. Research the efficiency of LoRA components and pre-trained model architecture watermark extractors.

[0142] During training, the token length was limited to 80 tokens, and the Wikitext training dataset was used to implant and extract 32-bit watermark information. The watermark extractor was constructed using full fine-tuning and LoRA fine-tuning, stacked three-layer transformer encoders, and a pre-trained encoder model. The training time and number of epochs to a usable state were examined. The LoRA configuration was r=2, alpha=16, drop_out=0.05, and the pre-trained encoder model used was T5Encoder. Subsequently, inference was performed using the test dataset, and the average accuracy, BERT score, and BLEU-4 metric for each scheme were evaluated.

[0143] Experiments showed that the loss value for watermark information extraction reached [amount missing]. At that point, its average accuracy is already high enough to be considered as having achieved the training objective. Although the loss values ​​between the watermarked text and the original text have not converged... However, the BERT score and BLEU-4 score have reached a usability level.

[0144] Table 3. Training Costs and Inference Performance under Different LoRA Settings and Watermark Extractor Frameworks

[0145]

[0146] from Figure 12 As shown in Table 3, the LoRA approach saves training time and accelerates model convergence compared to full fine-tuning, improving average accuracy at a cost of approximately 0.02 BERT scores. Using a pre-trained encoder architecture in conjunction with LoRA achieves effectiveness faster than full fine-tuning with a trasformer encoder stacking architecture, reducing training time by 5.5 times. While this results in a significant disadvantage for BLEU-4, it still maintains a usability of 0.15.

[0147] 2. Investigate the impact of LoRA's rank parameter setting on the accuracy and semantics of implanting different numbers of bits into the system.

[0148] Since this scheme uses different linear layer dimensions when embedding different numbers of bits, it is necessary to select the rank parameter of LoRA to prevent overfitting and underfitting during LoRA fine-tuning. As shown in Table 4, as the number of bits increases, the dimension of the linear layer increases, requiring a larger LoRA rank parameter. For each number of bits implanted, the rank parameter corresponding to the one with the higher BERT score and BLEU-4 score is preferentially selected as the main experimental scheme.

[0149] Table 4. The impact of LoRA rank setting on BEA and semantic score.

[0150]

[0151] 3. Research and analyze the impact of the redundant bit length set during LDPC encoding and decoding on the accuracy of watermark information.

[0152] Inference is still performed on the Wikitext dataset, testing the correct extraction rate of 1000 watermark entries for each set. With a fixed channel capacity for each set, watermark information of random bit length is sampled, then LDPC encoded accordingly to form redundant bits of the corresponding length in the table. The encoded information is then used for watermark embedding and extraction, followed by LDPC decoding. The decoded information bits are compared with the original information bits; a perfect match is considered a positive example, otherwise a negative example.

[0153] Table 5. Impact of different redundancy code settings on ML and UTA

[0154]

[0155] As the table shows, using longer redundancy codes can improve the accuracy of information bits, but at the cost of compressing the information bits. With increased channel capacity, the number of error bits in the model increases, and the bit extraction rate decreases, thus requiring more redundancy bits for error correction. Larger channel capacity results in longer inference time; therefore, when information bit accuracy is comparable, schemes with smaller CC and fewer redundancy code bits can be prioritized to save costs.

[0156] The experimental data above demonstrates that the proposed solution, through LDPC channel coding and LoRA efficient fine-tuning, achieves, for the first time, an extension of metadata length from 8 bits to 32 bits for 80-token short texts with low training cost, while maintaining an extraction accuracy of over 91%. Furthermore, the use of a watermark discriminator prevents false detections during large-scale deployments, enabling the solution to achieve robustness, accuracy, and high availability with large capacity. This makes it suitable for application scenarios involving accountability for AI-generated false information and malicious content on the internet.

[0157] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0158] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0159] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0160] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0161] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A text watermark generation method for black-box large language models, characterized in that, Include: Receive user input instructions, and generate the first text based on the user input instructions and a black-box large language model; A pre-trained watermark embedding model is used to generate and output watermarked text. The watermark embedding model includes an encoder for embedding representation of the first text and watermark information and a decoder for fusing the embedding vectors. The watermark information is data encoded using parity check codes to encode user-specific metadata.

2. The text watermark generation method for black-box large language models according to claim 1, characterized in that, The first text and watermark information are encoded, including: The first text is segmented into discrete word units, forming a word unit sequence; The encoder maps the word sequence to the text embedding vector in the corresponding embedding space; A linear layer is used to map the watermark information to a watermark embedding vector in the same embedding space.

3. The text watermark generation method for black-box large language models according to claim 1, characterized in that, The fused output of the embedded vectors includes: Add the text embedding vector and the watermark embedding vector together; The summed embedding vectors are input into a pre-trained decoder, which then generates and outputs the watermarked text.

4. The text watermark generation method for black-box large language models according to claim 1, characterized in that, The watermark embedding model is trained using the cross-entropy loss function, and the model is fine-tuned during the training process using the low-rank decomposition adaptation method.

5. A text watermark detection method, characterized in that, include: A pre-trained watermark discriminator is used to determine whether a text to be processed on the Internet has been watermarked. If there is no added watermark in the text to be processed, it is determined that the text to be processed on the Internet is not related to the users of the large language model service provider. If the text to be processed contains an added watermark, the watermark information in the text to be processed is extracted using a pre-trained watermark extractor, and user metadata is extracted from the watermark information through parity check code decoding operation, so as to match the target user from the users of the large language model service provider based on the user metadata. The watermark extractor and the watermark embedding model described in claim 1 adopt an end-to-end training process to simulate the training of watermark embedding and extraction.

6. The text watermark detection method according to claim 5, characterized in that, The watermark discriminator training process includes: Construct sample data with / without watermark labels using the trained watermark embedding model; The watermark discriminator is trained in a supervised manner using sample data.

7. The text watermark detection method according to claim 5, characterized in that, User metadata is extracted from the watermark information through parity check decoding, including: The user metadata in the watermark is extracted step by step using the bit-flipping method to decode and the LDPC decoding algorithm.

8. A text watermark detection system, characterized in that, It includes: a watermark discrimination module and a watermark recognition module, wherein, The watermark discrimination module is used to determine whether the text to be processed on the Internet has been watermarked using a pre-trained watermark discriminator. If there is no watermark in the text to be processed, it is determined that the text to be processed on the Internet is not related to the user of the large language model service provider; if there is a watermark in the text to be processed, watermark recognition is performed. The watermark recognition module is used to identify and extract watermark information in the text to be processed using a pre-trained watermark extractor. It extracts user metadata from the watermark information through parity check code decoding operation, so as to match the target user from the users of the large language model service provider based on the user metadata. The watermark extractor and the watermark embedding model described in claim 1 adopt an end-to-end training process to simulate the training of watermark embedding and extraction.

9. An electronic device, characterized in that, include: At least one processor, and a memory coupled to said at least one processor; The memory stores a computer program that can be executed by the at least one processor to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, enables the implementation of the method as described in any one of claims 1 to 7.

Images