Multi-bit text watermarking method and apparatus based on multi-head semantic tokenization model
By constructing a multi-head semantic vocabulary tagging model and training the model using a composite loss function, the limitations of robustness and embedding capacity in black-box text watermarking methods are solved, achieving efficient embedding and robust detection of multi-bit information, and ensuring the semantic quality and anti-attack capability of the text.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ARTIFICIAL INTELLIGENCE RES INST OF HEFEI COMPREHENSIVE NAT SCI CENT (ANHUI ARTIFICIAL INTELLIGENCE LAB)
- Filing Date
- 2026-04-24
- Publication Date
- 2026-05-29
AI Technical Summary
Existing black-box text watermarking methods have significant limitations in terms of robustness and embedding capacity. Traditional word substitution-based methods are fragile, and semantic watermarking methods only support single-bit embedding, making it difficult to meet the needs of multi-bit information embedding.
A multi-head semantic vocabulary tagging model is constructed. By utilizing a shared semantic encoder and multiple projection heads, the model is trained through a composite loss function to achieve multi-bit watermark embedding, including semantic consistency loss, channel independence loss, and tag unbiased loss, to ensure the robustness and independence of the tags.
It achieves efficient and robust embedding of multi-bit information in text generated by large language models, maintaining the fluency and semantic accuracy of the text, resisting rewrite attacks, and reducing the error rate.
Smart Images
Figure CN122113068A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of information security and artificial intelligence technology, and in particular to a multi-bit text watermarking method and apparatus based on a multi-head semantic lexical tagging model. Background Technology
[0002] With the rapid development of deep learning technology, large language models, represented by ChatGPT, GPT-4, and LLaMA, have demonstrated astonishing capabilities in natural language processing tasks. These models can generate fluent, logically rigorous, and content-rich text, and are widely used in various fields such as news writing, code generation, creative writing, and intelligent customer service. However, the widespread application of large language models has also brought a series of security and ethical challenges that cannot be ignored. Text watermarking technology, as a proactive defense measure, provides a feasible technical path to solve these problems by implicitly embedding specific identifying information in the generated text. Existing text watermarking methods are mainly divided into two categories: white-box watermarking and black-box watermarking. White-box watermarking methods usually require access to the model's internal parameters or intervention in the logits distribution during the generation process. However, in practical applications, many large models only provide services through APIs, and users cannot obtain the model's internal state, which limits the application scope of white-box watermarking.
[0003] In contrast, black-box watermarking directly modifies the generated complete text to embed information, offering greater versatility and flexibility. Despite some progress in black-box text watermarking technology, existing methods still have significant limitations in robustness and embedding capacity. Traditional word substitution-based watermarking methods often rely on fixed thesaurus or hash rules based on preceding words to select replacement words. This embedding mechanism, based on surface-level lexical features, is extremely vulnerable. Once an attacker interprets, rewrites, or replaces synonyms in the text, the original word sequence is disrupted, leading to the loss of watermark information. Existing semantic-based watermarking methods utilize the semantic vectors of sentences to generate markers, exhibiting strong attack resistance. However, most only support single-bit embedding, unable to embed multi-bit specific information. Furthermore, simply simulating multi-channel by segmenting the vocabulary leads to a drastic reduction in the number of candidate words, severely impairing the readability and semantic quality of the text.
[0004] Therefore, designing a text watermarking method that can maintain the high robustness of semantic watermarking while overcoming the capacity limitations of single-head models and achieving efficient and high-quality multi-bit embedding has become an urgent technical challenge. Summary of the Invention
[0005] The main objective of this invention is to provide a multi-bit text watermarking method and apparatus based on a multi-head semantic lexical tagging model. The aim is to design a text watermarking method that can maintain the high robustness of semantic watermarking while breaking through the capacity limitation of single-head models, and achieve efficient and high-quality multi-bit embedding.
[0006] To achieve the above objectives, this invention proposes a multi-bit text watermarking method based on a multi-head semantic lexical tagging model, comprising: A multi-head semantic lexical tagging model is constructed, which includes a shared semantic encoder and N projection heads. The N projection heads each provide N independent binary tagging subspaces, where N is a positive integer greater than 1. The multi-head semantic vocabulary tagging model is trained using a composite loss function, which includes semantic consistency loss. Independent channel loss and labeled unbiased loss The semantic consistency loss The tags generated from semantically similar texts possess invariance; the channel-independent loss The N binary label subspaces are constrained to be mutually decoupled in terms of statistical distribution or parameter distribution; the label unbiased loss This is used to constrain the tags generated by each projection head to be evenly distributed across the entire vocabulary; The text is input into the trained multi-head semantic vocabulary tagging model to obtain N sets of tag sequences for the vocabulary; Obtain the multi-bit watermark sequence to be embedded, and determine the mapping relationship between each position in the text and the binarized marker subspace according to the protocol; Based on the mapping relationship, words that satisfy the target tag value in the corresponding tag subspace are found from the vocabulary and embedded to generate watermarked text.
[0007] Preferably, the number of projection heads is N, including a first projection head, a second projection head, and an Nth projection head; each projection head is implemented through a multilayer perceptron, which maps the input semantic vector to the hidden space through a first fully connected layer, maps it to the output space through a second fully connected layer to obtain the original score, and uses a sign function to convert the original score into discrete labels, the calculation formula of which is:
[0008]
[0009]
[0010] in, This represents the hidden layer output of the k-th projection head. and These are the first-layer weight matrix and bias, respectively. For contextual semantic embedding of the input, This indicates the output score. and These are the weights and biases of the second layer, respectively. For the generated list of discrete tags.
[0011] Preferably, the channel-independent loss Using channel orthogonal loss The channel orthogonal loss The formula used to penalize the correlation between the output vectors of different projection heads is as follows:
[0012] in, and Let represent the output vectors of the a-th and b-th projection heads, respectively. and The integer indices are between 1 and N and are not equal.
[0013] Preferably, the semantic consistency loss This is achieved by calculating the difference between the edit distance between the tag lists and the similarity between the semantic embeddings, using the following formula:
[0014] in, This represents the weights that are dynamically adjusted based on semantic similarity. This represents the Levenshtein distance between the label lists generated by the k-th projection head for the i-th and j-th samples. Let represent the cosine similarity between the i-th sample and the j-th sample, where and This is the index of the training samples.
[0015] Preferably, the label has unbiased loss. The calculation formula is:
[0016] in, Let represent the output score of the k-th projection head for the i-th sample, where For the index of the training samples, This is the function for calculating the mean.
[0017] Preferably, before generating the watermarked text, the method further includes obtaining a selected candidate set, specifically including the following steps: A masked language model is used to predict candidate words at the positions of words to be replaced in the text, generating an initial candidate set. Calculate the semantic similarity between each word in the initial candidate set and the original word. And the semantic similarity between the replaced sentence and the original sentence. ; Retain satisfaction and The vocabulary constitutes the selected candidate set, wherein and These are the preset word meaning similarity threshold and sentence meaning similarity threshold, respectively.
[0018] Preferably, when selecting words that meet the corresponding tagging requirements from the selected candidate set, the target word with the highest contextual fit is selected. The calculation formula is as follows:
[0019] in, For the selected candidate set, Let c be the label value of candidate word c under the k-th projection head. For the target bit value, This represents the contextual fitness score of word c in the current text environment.
[0020] Preferably, the method further includes blind detection extraction of the watermark text, wherein the blind detection extraction includes: Segment the suspicious text and extract semantic embedding vectors; The semantic embedding vector is input into the trained multi-head semantic vocabulary tagging model to reconstruct N sets of tag lists; Extract the tag values corresponding to words at each position according to the preset embedding protocol, and reconstruct the binary bit stream; By calculating the Z-score Hypothesis testing was performed to verify the existence of the watermark.
[0021] Preferably, the Z-score The calculation is performed separately for each channel k, and the formula is as follows:
[0022] in, This represents the proportion of words in the k-th channel that are marked with the first preset tag value. The total number of samples in the k-th channel. This represents the theoretical probability in the watermark-free state.
[0023] Preferably, the shared semantic encoder uses a pre-trained BERT model to extract the contextual semantic embeddings of the input text. .
[0024] This application also discloses a multi-bit text watermarking device based on a multi-head semantic lexical tagging model, comprising: The model building module is used to build a multi-head semantic vocabulary tagging model, which includes a shared semantic encoder and N projection heads, each of which provides N independent binary tagging subspaces. The model training module is used to train the multi-head semantic vocabulary tagging model using a composite loss function, which includes semantic consistency loss. Independent channel loss and labeled unbiased loss ; The sequence acquisition module is used to input text into the multi-head semantic vocabulary tagging model after training, and obtain N sets of tag sequences for the vocabulary; The protocol mapping module is used to obtain the multi-bit watermark sequence to be embedded and determine the mapping relationship between each position in the text and the binarized marker subspace according to the protocol. The watermark embedding module is used to find words that satisfy the target tag value under the corresponding tag subspace from the vocabulary according to the mapping relationship, and embed them to generate watermarked text.
[0025] The above technical solution has the following advantages: The method provided in this invention achieves semantic space decoupling through a multi-head mechanism, enabling the model to embed multiple bits of information in parallel or serially within the same text. The watermark capacity increases linearly with the number of projection heads. Since each projection head operates on the complete vocabulary, it avoids the candidate word shortage problem caused by physical vocabulary segmentation in traditional methods. This ensures that high-quality synonyms are always found for replacement during the embedding process, maintaining the fluency and semantic accuracy of the text. By introducing semantic consistency loss to train the model, the generation and detection of the watermark rely on stable deep semantics rather than surface vocabulary. Even if the text undergoes deep rewriting, translation, or summarization by a large model, as long as the core semantics are preserved, the watermark information can be effectively detected. Furthermore, the channel orthogonality constraint achieved through channel independence loss effectively reduces interference between different bit channels, ensuring the clarity of multiple signals and maintaining an extremely low bit error rate. Attached Figure Description
[0026] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a schematic diagram of the watermark embedding process provided in an embodiment of the present invention.
[0027] Figure 2 This is a schematic diagram of the watermark extraction process provided in an embodiment of the present invention.
[0028] Figure 3 This is a schematic diagram of the multi-head semantic lexical tagging model architecture provided in an embodiment of the present invention. Detailed Implementation
[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.
[0030] Example 1 like Figures 1 to 3 As shown, this embodiment provides a multi-bit text watermarking method based on a multi-head semantic lexical tagging model, aiming to solve the problems of copyright protection and source tracing of content generated by large language models. With the popularization of artificial intelligence technology, the text generated by large language models has approached human-level fluency and logic, leading to an increased risk of false information dissemination and copyright infringement. Traditional black-box watermarking methods often rely on fragile lexical hashing rules; once the text is rewritten or replaced with synonyms, the watermark information is easily lost. While existing semantic watermarking methods are robust, due to the singularity of the tagging space, they often only achieve single-bit embedding, making it difficult to meet the needs of embedding complex multi-bit information such as user identification codes.
[0031] This embodiment implements watermark embedding by constructing and utilizing a multi-head semantic lexical tagging model. The method first requires constructing a multi-head semantic lexical tagging model. This model includes a shared semantic encoder 1.1 and at least one projection head 1.2. In specific hardware or logic implementations, the semantic encoder 1.1 and projection head 1.2 are composed of deep neural networks. The projection head 1.2 is configured to provide N independent binary tag subspaces, where N is a positive integer greater than 1, and the value of N is determined according to the actual length of the watermark bits to be embedded. In this embodiment, the shared semantic encoder 1.1 uses a pre-trained BERT model for an input sentence. First, the tokens are converted into a token sequence using BERT's token segmenter. This sequence is then input into the BERT model, and the extracted output vector serves as the context semantic embedding vector for the entire sentence. This vector is represented as... This semantic extraction method based on pre-trained models can capture the deep semantic features of text, thus making the subsequently generated tags highly invariant to changes on the surface of the text.
[0032] Furthermore, the number of projection heads 1.2 is typically set to N, i.e., the first projection head, the second projection head, and so on up to the Nth projection head. Each projection head 1.2 is implemented through a multilayer perceptron network structure. During the computation process, the input semantic vector E is mapped to the hidden space through the first fully connected layer, and its calculation formula is as follows:
[0033] In the above formula, This represents the hidden layer output of the k-th projection head. and These are the weight matrix and bias of the first layer, respectively. Then, the hidden layer output is mapped to the output space through a second fully connected layer to obtain the original score, calculated as follows:
[0034] in, This indicates the output score. and These represent the weights and biases for the second layer, respectively. Finally, the original scores are converted into discrete labels using a sign function, resulting in a list of generated discrete labels. The calculation formula is as follows:
[0035] in In this way, for each word in the vocabulary, there will be a corresponding binary tag value of 1 or negative 1 under each projection head, thus forming N independent binary tag subspaces across the entire vocabulary dimension.
[0036] To ensure that the tags generated by the model are both robust and remain independent across multiple channels, this embodiment uses a composite loss function to train the multi-head semantic vocabulary tagging model. The composite loss function includes semantic consistency loss. Unbiased loss of labels and independent channel loss .
[0037] Specifically, semantic consistency loss Its purpose is to constrain the invariance of tags generated from semantically similar text, thereby resisting rewriting attacks. It achieves this by calculating the difference between the edit distance between tag lists and the similarity between semantic embeddings, using the following formula:
[0038] in, This represents the weights that are dynamically adjusted based on semantic similarity. This represents the Levenshtein distance between the label lists generated by the k-th projection head for the i-th and j-th samples. Let represent the cosine similarity between the i-th and j-th samples, where i and j are the indices of the training samples. During training, when the semantic similarity of sample pairs... Greater than the preset high threshold When the semantic similarity is less than a preset low threshold, the model is forced to ensure that the distance between the labeled lists approaches 0; when the semantic similarity is less than a preset low threshold, the model is forced to ensure that the distance between the labeled lists approaches 0. At that time, the model is encouraged to generate larger distances.
[0039] Labeled unbiased loss This is used to constrain the tokens generated by each projector to be evenly distributed across the entire vocabulary, meaning the ratio of generated 1s to negative 1s is close to 1:1, thereby maximizing the watermark's concealment and preventing statistical analysis attacks. The calculation formula is as follows:
[0040] in, This represents the output score of the k-th projection head for the i-th sample. This is the function for calculating the mean.
[0041] Channel Independent Loss In this embodiment, channel orthogonal loss is specifically used. This is used to penalize the correlation between the output vectors of different projectors, ensuring that each projector focuses on different dimensions of semantic features and guaranteeing that the generated N sets of labeled sequences are statistically decoupled. Its calculation formula is:
[0042] in, and These represent the output vectors of the a-th and b-th projection heads, respectively. Furthermore, this embodiment also constrains the first-layer weight matrix of different projection heads. Orthogonality is achieved by using the cosine similarity between them.
[0043] The model training process uses the total loss function :
[0044] Using a large-scale text corpus, the Adam optimizer is employed to minimize the total loss.
[0045] After model training is complete, the watermark embedding stage begins. First, the original text to be processed is input into the trained multi-head semantic vocabulary tagging model to obtain N sets of tag sequences for the vocabulary. Then, the multi-bit watermark sequence to be embedded is obtained. According to a preset protocol, the mapping relationship between each word position in the text and the binary tag subspace is determined. This embodiment adopts a serial polling strategy; for the t-th replaceable word position in the text, the first polling term is used to determine the position of the t-th word. Each projection head is responsible for this.
[0046] To ensure the quality of the embedded text, this embodiment also includes a step of obtaining a selected candidate set before generating the watermarked text. Specifically, this involves using a masked language model to predict candidate words at the positions of the words to be replaced in the text, generating an initial candidate set. Specifically, the top-K words with the highest predicted probabilities are selected. Next, the semantic similarity between each word in the initial candidate set and the original word is calculated. Specifically, the cosine similarity of GloVe or Word2Vec vectors is used for calculation; simultaneously, the semantic similarity between the replaced sentence and the original sentence is calculated. Specifically, the cosine similarity of BERT sentence vectors is used for calculation. The system only retains those that satisfy... and The vocabulary used to construct a selected candidate set. .
[0047] Finally, based on the aforementioned mapping relationship, words that satisfy the target tag value in the corresponding tag subspace are selected from the preferred candidate set of the vocabulary for embedding. In selecting specific words, this embodiment selects the target words with the highest contextual fit. To ensure the natural flow of the text, the calculation formula is as follows:
[0048] in, Let c be the label value of candidate word c under the k-th projection head. For the target bit value, This represents the contextual fitness score of word c in the current text environment. By replacing the original word in this way, a semantically robust watermarked text containing multi-bit information is ultimately generated.
[0049] Example 2 This embodiment provides a multi-bit text watermarking device based on a multi-head semantic vocabulary tagging model, used to implement the method described in Embodiment 1 above. The device is logically divided into multiple functional modules, including a model building module, a model training module, a sequence acquisition module, a protocol mapping module, and a watermark embedding module.
[0050] The model building module is configured to construct a multi-head semantic vocabulary tagging model. In this model, the shared semantic encoder 1.1 consists of a pre-trained Transformer architecture, specifically using the BERT model to extract the contextual semantic embedding vector E of the text, with a dimension of 768. The number of projection heads 1.2 is set to N, providing N independent binary tag subspaces, enabling the device to embed multiple watermark information in parallel within the same text unit. Each projection head 1.2 is implemented using a multilayer perceptron. The input semantic vector is mapped to the hidden space through a first fully connected layer, and then mapped to the output space through a second fully connected layer to obtain the original score. A sign function is then used to convert the original score into discrete tags. The specific discrete tag generation process follows the following formula:
[0051]
[0052]
[0053] In the above calculation process, This represents the hidden layer output of the k-th projection head. and These are the first-layer weight matrix and bias, respectively. For contextual semantic embedding of the input, This indicates the output score. and These are the weights and biases of the second layer, respectively. For the generated list of discrete tags.
[0054] The model training module is used to train the multi-head semantic vocabulary tagging model using a composite loss function. During training, the device minimizes the total loss function using the Adam optimizer. To ensure the robustness of the watermark, a semantic consistency loss is introduced in the model training module. It implements constraints by calculating the difference between the Levenshtein distance between the tag lists and the cosine similarity between the semantic embeddings. The specific formula is as follows:
[0055] In addition, to ensure the independence of each channel, the model training module also introduces channel orthogonality loss. This is used to penalize the correlation between the output vectors of different projection heads. Labeled unbiased loss. This is used to constrain the tokens generated by each projector to have a uniform distribution across the entire vocabulary, meaning that the probability of a token being 1 or negative 1 is close to 50%. The calculation formula is as follows:
[0056] The sequence acquisition module inputs the text into the trained model to obtain N sets of labeled sequences for the entire vocabulary. The protocol mapping module is responsible for obtaining the multi-bit watermark sequence to be embedded and determining the correspondence between the position of each word in the text and the binary labeled subspace according to the preset embedding protocol, i.e., mapping rules.
[0057] The watermark embedding module performs the specific watermark injection task. This module first uses a masked language model to predict the positions of the words to be replaced in the text to generate an initial candidate set. Then, the semantic similarity between each word in the initial candidate set and the original word is calculated. And the semantic similarity between the replaced sentence and the original sentence. Only when the preset threshold is met. and Only the best words will be retained in the selected candidate set. In the final step, the watermark embedding module selects words from the curated candidate set that meet the corresponding tag requirements and have the highest contextual fit to replace the original words, generating the final watermark text. Its target words... The selection criteria are:
[0058] Example 3 This embodiment further illustrates the blind detection and extraction process of the watermarked text, which does not require the participation of the original carrier text.
[0059] During the detection phase, the device first segments the suspicious text into words and extracts its contextual semantic embedding vector. Then, this semantic embedding vector is input into a pre-trained multi-head semantic vocabulary tagging model to reconstruct N sets of tag lists. Because the model incorporates semantic consistency loss during training, the reconstructed tag lists exhibit strong stability against text rewriting attacks.
[0060] Next, based on a preset embedding protocol—the mapping between text positions and projection head indices—the device extracts the tag values corresponding to words at each position and reassembles them into a binary bitstream. Finally, the device calculates the Z-score. Hypothesis testing was performed to verify the existence of the watermark. The Z-score was calculated for each channel k, and the formula is as follows:
[0061] In the above formula, This represents the proportion of words in the k-th channel that are marked with the first preset tag value. The total number of samples in the k-th channel. This represents the theoretical probability in the watermark-free state, typically set to 0.5. Through this statistical test, the system can reliably determine whether the text carries the expected multi-bit watermark information.
[0062] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a multi-bit text watermarking method based on a multi-head semantic lexical tagging model as described in Embodiment 1.
[0063] This storage medium can be implemented using any type of non-volatile storage device or a combination thereof. Specifically, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM). Furthermore, the storage medium can also be magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or read-only optical disc (CD-ROM). Magnetic surface memory can specifically be disk storage or magnetic tape storage.
[0064] This embodiment also provides a control device, which includes a processor and a storage medium for storing a computer program. The processor executes the computer program stored in the storage medium to implement the various technical steps described in the foregoing embodiments. In specific industrial applications, this control device can be integrated into a server for real-time watermark injection into text content generated by a large language model, or it can serve as an independent detection terminal for copyright identification and source tracing of suspicious text on the Internet. When executing the program, the processor loads a pre-trained BERT model as a semantic encoder 1.1 and calls a trained multi-head projection head 1.2 to achieve efficient embedding and robust extraction of multi-bit information while ensuring semantic consistency of the text.
[0065] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.
Claims
1. A multi-bit text watermarking method based on a multi-head semantic lexical tagging model, characterized in that, include: A multi-head semantic lexical tagging model is constructed, which includes a shared semantic encoder and N projection heads. The N projection heads each provide N independent binary tagging subspaces, where N is a positive integer greater than 1. The multi-head semantic vocabulary tagging model is trained using a composite loss function, which includes semantic consistency loss. Independent channel loss and labeled unbiased loss The semantic consistency loss The tags generated from semantically similar texts possess invariance; the channel-independent loss The N binary label subspaces are constrained to be mutually decoupled in terms of statistical distribution or parameter distribution; the label unbiased loss This is used to constrain the tags generated by each projection head to be evenly distributed across the entire vocabulary; The text is input into the trained multi-head semantic vocabulary tagging model to obtain N sets of tag sequences for the vocabulary; Obtain the multi-bit watermark sequence to be embedded, and determine the mapping relationship between each position in the text and the binarized marker subspace according to the protocol; Based on the mapping relationship, words that satisfy the target tag value in the corresponding tag subspace are found from the vocabulary and embedded to generate watermarked text.
2. The multi-bit text watermarking method based on a multi-head semantic lexical tagging model as described in claim 1, characterized in that, The number of projection heads is N, including a first projection head, a second projection head, and so on up to the Nth projection head. Each projection head is implemented using a multilayer perceptron. The input semantic vector is mapped to the hidden space through a first fully connected layer, and then mapped to the output space through a second fully connected layer to obtain the original score. The original score is then converted into discrete labels using a sign function, calculated as follows: in, This represents the hidden layer output of the k-th projection head. and These are the first-layer weight matrix and bias, respectively. For contextual semantic embedding of the input, This indicates the output score. and These are the weights and biases of the second layer, respectively. For the generated list of discrete tags.
3. The multi-bit text watermarking method based on a multi-head semantic lexical tagging model as described in claim 1, characterized in that, The channel independent loss Using channel orthogonal loss The channel orthogonal loss The formula used to penalize the correlation between the output vectors of different projection heads is as follows: in, and Let represent the output vectors of the a-th and b-th projection heads, respectively. and The integer indices are between 1 and N and are not equal.
4. The multi-bit text watermarking method based on a multi-head semantic lexical tagging model as described in claim 1, characterized in that, The semantic consistency loss This is achieved by calculating the difference between the edit distance between the tag lists and the similarity between the semantic embeddings, using the following formula: in, This represents the weights that are dynamically adjusted based on semantic similarity. This represents the Levenshtein distance between the label lists generated by the k-th projection head for the i-th and j-th samples. Let represent the cosine similarity between the i-th sample and the j-th sample, where and This is the index of the training samples.
5. The multi-bit text watermarking method based on a multi-head semantic lexical tagging model as described in claim 1, characterized in that, The unbiased loss of the label The calculation formula is: in, Let represent the output score of the k-th projection head for the i-th sample, where For the index of the training samples, This is the function for calculating the mean.
6. The multi-bit text watermarking method based on a multi-head semantic lexical tagging model as described in claim 1, characterized in that, Before generating the watermarked text, the method further includes obtaining a selected candidate set, specifically including the following steps: A masked language model is used to predict candidate words at the positions of words to be replaced in the text, generating an initial candidate set. Calculate the semantic similarity between each word in the initial candidate set and the original word. And the semantic similarity between the replaced sentence and the original sentence. ; Retain satisfaction and The vocabulary constitutes the selected candidate set, wherein and These are the preset word meaning similarity threshold and sentence meaning similarity threshold, respectively.
7. The multi-bit text watermarking method based on a multi-head semantic lexical tagging model as described in claim 6, characterized in that, When selecting words that meet the corresponding tagging requirements from the selected candidate set, the target word with the highest contextual fit is selected. The calculation formula is as follows: in, For the selected candidate set, Let c be the label value of candidate word c under the k-th projection head. For the target bit value, This represents the contextual fitness score of word c in the current text environment.
8. The multi-bit text watermarking method based on a multi-head semantic lexical tagging model as described in claim 1, characterized in that, The method further includes blind detection extraction of the watermark text, wherein the blind detection extraction includes: Segment the suspicious text and extract semantic embedding vectors; The semantic embedding vector is input into the trained multi-head semantic vocabulary tagging model to reconstruct N sets of tag lists; Extract the tag values corresponding to words at each position according to the preset embedding protocol, and reconstruct the binary bit stream; By calculating the Z-score Hypothesis testing was performed to verify the existence of the watermark.
9. A multi-bit text watermarking method based on a multi-head semantic lexical tagging model as described in claim 8, characterized in that, The Z-score The calculation is performed separately for each channel k, and the formula is as follows: in, This represents the proportion of words in the k-th channel that are marked with the first preset tag value. The total number of samples in the k-th channel. This represents the theoretical probability in the watermark-free state.
10. A multi-bit text watermarking method based on a multi-head semantic lexical tagging model as described in any one of claims 1 to 9, characterized in that, The shared semantic encoder uses a pre-trained BERT model to extract the contextual semantic embeddings of the input text. .
11. A multi-bit text watermarking device based on a multi-head semantic lexical tagging model, characterized in that, include: The model building module is used to build a multi-head semantic vocabulary tagging model, which includes a shared semantic encoder and N projection heads, each of which provides N independent binary tagging subspaces. The model training module is used to train the multi-head semantic vocabulary tagging model using a composite loss function, which includes semantic consistency loss. Independent channel loss and labeled unbiased loss ; The sequence acquisition module is used to input text into the multi-head semantic vocabulary tagging model after training, and obtain N sets of tag sequences for the vocabulary; The protocol mapping module is used to obtain the multi-bit watermark sequence to be embedded and determine the mapping relationship between each position in the text and the binarized marker subspace according to the protocol. The watermark embedding module is used to find words that satisfy the target tag value under the corresponding tag subspace from the vocabulary according to the mapping relationship, and embed them to generate watermarked text.