A multi-bit watermark embedding and extraction method fusing context awareness

Abstract

The present application relates to a kind of fusion context-aware multi-bit watermark embedding and extraction method.Watermark embedding method includes: candidate word set generation and embedding timing determination based on Top-p strategy;Model confidence filtering based on Top-1 Gap;Pseudo-random word group based on watermark message and seed;Group refinement;Semantic consistency filtering and determinate sampleable set;Probability renormalization and sampling output.Watermark extraction method includes: input preprocessing and sequence initialization;Word element conditional judgment and bit decoding;Periodic sampling based on fixed interval.The embodiment systematically solves the triangular dilemma of "capacity, quality, robustness" difficult to consider in the prior art, realizes embedding traceable multi-bit information under the premise of almost not reducing text generation quality, and can effectively resist semantic maintenance type modification attack.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security and digital rights management technology, specifically involving a method for embedding and extracting multi-bit watermarks with context awareness. Background Technology

[0002] With the rapid development of Large Language Models (LLMs) technologies, such as ChatGPT and GPT-4, the quality and creativity of their generated text have approached or even surpassed human levels, and they are widely used in content creation, intelligent dialogue, code generation, and other scenarios. However, the "double-edged sword" effect of this technology has also become apparent: on the one hand, text generated by LLMs may be misused to create false information, commit academic misconduct, or generate malicious content, posing a serious challenge to social trust and security; on the other hand, the copyright ownership and traceability protection of the models themselves, as core intellectual property rights trained with enormous computing power and data, has become an urgent industry issue. Against this backdrop, embedding traceable and verifiable "digital fingerprints"—i.e., watermarking technology—into AI-generated content has become a research hotspot in both academia and industry.

[0003] Traditional digital watermarking technology primarily targets multimedia data such as images, audio, and video, embedding information by modifying specific frequency or spatial domain coefficients. However, text, as a discrete sequence of symbols, has low redundancy and is extremely sensitive to modification; a change in a single character can lead to semantic alterations or grammatical errors, making traditional methods difficult to apply directly. Early text watermarking methods were mostly based on format fine-tuning (such as replacing spaces and punctuation) or synonym replacement, but these methods have poor concealment, low capacity, and are easily compromised by simple format normalization processes.

[0004] For neural network models, researchers have proposed the concept of model watermarking. This involves inserting a backdoor by modifying model parameters or training data during the training phase, causing the model to produce a preset output under specific trigger inputs, thereby proving ownership. However, these methods primarily protect the model itself, rather than the specific content it generates. In recent years, watermarking techniques for LLM-generated text have begun to emerge.

[0005] Existing technologies mainly focus on watermarking methods based on the generation stage of large language models. The core idea is to perturb the probability distribution (Logits) of the next word through specific rules and before final sampling, thereby embedding detectable statistical signals. A typical example of this type of technology is the KGW (Green-Red List) method proposed by Kirchenbauer et al. (Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023). A Watermark for Large Language Models). Proceedings of the 40th International Conference on Machine Learning (ICML 2023). This method, when generating each word, pseudo-randomly divides the entire vocabulary into a "green list" and a "red list" based on the preceding text using a hash function. By adding a fixed probability offset (delta) to words in the green list, the model tends to select words from the green list during sampling, thus leaving a statistically detectable bias in the generated text—that is, abnormally high frequency of green-listed words. This type of method provides an important approach to achieving seamless tagging of LLM-generated text, but it is essentially still a single-bit watermark, only able to identify whether the text originates from a specific model. It has limitations in information capacity and cannot carry more refined source information, such as user IDs or session identifiers.

[0006] Therefore, existing technologies face significant limitations: First, most mainstream methods are single-bit watermarks, which can only serve as a binary label indicating whether a particular model generated the watermark. The capacity of a single bit is insufficient to carry multi-bit traceability information such as model version, user ID, and generation timestamp. In actual copyright disputes or malicious content tracking, single-bit information falls far short of meeting the needs of refined management, limiting its application in complex auditing and refined management. Second, most existing watermarks are static and semantically unrelated. For example, they predefine a fixed "green vocabulary" or uniformly adjust the probability of words at all generation locations, independent of context. This approach lacks context awareness and may lead to the forced implantation of traces in key locations (such as factual statements and emotional core words), damaging the logical coherence and quality of the text, or forcing a reduction in watermark strength to maintain fidelity, sacrificing robustness. Finally, there is a dilemma between stealth and capacity: high stealth requires weak signals, but they are easily erased by simple post-processing operations such as rewriting, translation, and summarization (which are common attack methods); while increasing information capacity further exacerbates this contradiction.

[0007] Therefore, developing a new watermarking technology that can perceive contextual semantics, adaptively embed multi-bit information, and achieve a better balance between imperceptibility, robustness, and information capacity has become a key technical requirement to ensure the healthy and secure development of the large language model ecosystem. Summary of the Invention

[0008] To address the aforementioned problems in the existing technology, this invention provides a method for embedding and extracting multi-bit watermarks with context awareness. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a method for embedding multi-bit watermarks with context awareness, comprising the following steps: S11. Obtain the original probability distribution of all candidate word elements in the vocabulary of the current context calculated by the large language model; use a kernel sampling strategy to filter out a set of candidate words from the original probability distribution; when the number of word elements in the set of candidate words meets the candidate set size threshold, execute step S12. S12. Calculate the probability difference between the word with the highest probability and the word with the second highest probability in the candidate word set. When the probability difference meets the confidence threshold, proceed to step S13. S13. Using a cryptographic hash function and a fixed seed, the tokens in the candidate word set are mapped to group numbers. Based on the static mapping relationship and the watermark bits to be embedded, the tokens corresponding to the target group number are selected to form the target candidate group. S14. The target candidate group is truncated according to the probability of the word in the original probability distribution, and the word with the highest probability is retained to obtain the refined candidate group. S15. Perform semantic consistency filtering on the lexical units in the refined candidate group to obtain a sampleable set; when the sampleable set is not empty, proceed to step S16. S16. Perform probability renormalization on each lexical in the sampleable set to obtain a new sampling distribution, and sample from the new sampling distribution based on a preset fixed-interval embedding strategy to generate watermarked lexical.

[0009] In one embodiment of the present invention, step S11 includes: S111, Obtain the current context of the large language model calculation. In the vocabulary The original probability distribution of all candidate lexical units and the original probability distribution Sort in descending order to obtain an ordered list. and their corresponding ordered probabilities ; S112. Employ a kernel sampling strategy to convert the ordered probability... The probabilities in each case are summed from front to back to obtain the cumulative probability. The cumulative probability is greater than or equal to the cumulative probability threshold. The lexical units constitute the candidate word set. ; S113, when judging the candidate word set Number of Chinese lexical elements Greater than or equal to the candidate set size threshold When the candidate word set is determined, step S12 is executed. Number of Chinese lexical elements Smaller than the candidate set size threshold At that time, standard sampling is performed according to the original probability distribution.

[0010] In one embodiment of the present invention, step S12 includes: S121, from the candidate word set The word with the highest recognition probability and its probability and the second most probable word. and its probability ; S122, Calculate the probability With probability The probability difference between them: ; S123, when the probability difference Less than or equal to the confidence threshold When the probability difference is... Greater than the confidence threshold At that time, standard sampling is performed according to the original probability distribution.

[0011] In one embodiment of the present invention, step S13 includes: S131, Set fixed seed And determine the number of groups. ,in, , The number of watermark bits that can be carried by each lexical unit; S132, Each word element in the candidate word set Text representation or index with fixed seed By concatenating the vectors, we obtain the concatenated vector: ,in, Indicates word elements The text representation or index; S133. Calculate the concatenated vector using a cryptographic hash function. The hash value yields a binary string. ; the binary string Treat as an integer and relative to the number of groups The modulo operation is used to obtain each lexical unit in the candidate word set. Group number: ,in, ; S134, Based on the preset number of watermark bits With group number Based on the static mapping relationship, the target candidate group is selected from the candidate word set according to the number of watermark bits of the word to be embedded. .

[0012] In one embodiment of the present invention, step S14 includes: S141, the target candidate group All morphemes in According to the original probability distribution Sort the probabilities in descending order to obtain the sorted list. ; S142. Based on the group cutoff parameter For the sorted list Truncate the section and retain the section with the highest probability. each word element Refined candidate group obtained .

[0013] In one embodiment of the present invention, step S15 includes: S151, Obtain the current preceding text The end The context semantic vector is obtained by averaging the hidden state vectors of each word in the last layer or an intermediate layer of the large language model. ; S152, Obtain the refined candidate group Each word element Word embedding vectors ; S153, Calculate the refining candidate group Each word element Word embedding vectors With context semantic vector Cosine similarity: ; S154. Preserve the cosine similarity. Greater than or equal to the semantic similarity threshold lexical elements ; S155, Retained Lexical Elements According to cosine similarity Sort in descending order and select the top few with the highest similarity. each word element To obtain the sampleable set When the sampleable set If the set is not empty, proceed to step S16; when the sampleable set is... When the set is empty, standard sampling is performed according to the original probability distribution.

[0014] In one embodiment of the present invention, step S16 includes: S161, the sampleable set Each word element The original probability distribution By summing the results, we can obtain the summation probability: Based on the summing probability For the sampleable set Each word element Perform probability renormalization and calculate each word. The watermarked sampling probability distribution: This yields a new sampling distribution; S162. Based on a preset fixed-interval embedding strategy, sample from the new sampling distribution to generate watermarked lexical units. The preset fixed-interval embedding strategy is as follows: if embedding fails at the predetermined embedding site, the embedding process is started at the next adjacent position; if embedding succeeds at the predetermined embedding site, the embedding process is started at each interval. The embedding process is initiated at each location.

[0015] Another embodiment of the present invention provides a multi-bit watermark extraction method incorporating context awareness, comprising the following steps: S21. Use the word segmenter of the large language model to segment the text sequence to be detected into a word sequence, and create a list of valid positions and a list of decoded bits; S22. Input the prefix of the current word into the large language model to obtain the simulated original probability distribution, and use the kernel sampling strategy to reproduce the simulated candidate word set; when the words in the simulated candidate word set pass the candidate set size threshold and confidence threshold, and the current word and its prefix pass the semantic consistency judgment, use the cryptographic hash function and fixed seed to calculate the group number of the current word, map the group number to the watermark bit fragment, store the watermark bit fragment in the decoded bit list, and store the position index of the watermark bit fragment in the text sequence to be detected in the valid position list; S23. Starting from the first position in the list of valid positions, periodically sample at fixed intervals during embedding, and extract the corresponding bit segments from the list of decoded bits according to the sampling results and concatenate them in order to form a sampled bit sequence. S24. Divide the sampled bit sequence into several complete segments according to the length of the watermark message, calculate the proportion of corresponding bits that are the same between any two pairs of segments in the several complete segments and calculate the average value to obtain the segment consistency rate; when it is determined that there is a watermark message in the several complete segments according to the segment consistency rate, the watermark message is recovered from the several complete segments by majority voting.

[0016] In one embodiment of the present invention, step S22 includes: S221, Set the current word element prefix Obtain the original probability distribution of the simulation from the input large language model. A kernel sampling strategy is used to reproduce the simulated candidate word set from the original simulated probability distribution. When the simulated candidate word set When the number of lexical units is greater than or equal to the candidate set size threshold, proceed to step S222; when the simulated candidate word set... If the number of Chinese words is less than the candidate set size threshold, the watermark is not extracted. S222, Calculate the simulated candidate word set. The probability difference between the word with the highest probability and the word with the second highest probability, when the probability difference Less than or equal to the confidence threshold When the probability difference is reached, step S223 is executed; Greater than the confidence threshold At that time, the watermark is not extracted; S223, Calculate the current word element Its prefix semantic similarity between When the semantic similarity Greater than or equal to the semantic similarity threshold Execute step S224; when the semantic similarity Less than the semantic similarity threshold Do not extract watermark; S224. Use the same fixed seed as in the embedding phase. and cryptographic hash functions Calculate the current word element Group number : The grouping numbers are determined based on the same static mapping relationship during the embedding phase. Decoded into watermark bit fragments ,in, Number of groups , The number of watermark bits; the watermark bit segment Store the decoded bit list The watermark bit fragment Position index in the text sequence to be detected Store in valid location list .

[0017] In one embodiment of the present invention, step S24 includes: S241. Based on the length of the watermark message M bits will sample bit sequence Divide into several lengths Fragments of bits, and discard those with insufficient fragments. From bits, several complete fragments are obtained. ; Calculate several complete segments The segment consistency rate is obtained by calculating the proportion of corresponding bits that are the same between any two pairs of segments and averaging all consistency proportions. When the segment consistency rate If the value is greater than or equal to 0.7, it is determined that a watermark message exists in the aforementioned complete segments; S242. Check the bit value of each bit in the several complete segments, and use the majority voting principle to select the bit value that appears most frequently as the recovery message for the corresponding bit, thus obtaining the watermark message. .

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. In the watermark embedding method of the present invention, the candidate word set is judged by the candidate set size threshold and the confidence threshold, and the semantic consistency of the words in the refined candidate group is filtered. Through the adaptive three-level dynamic scheduling with quality priority, multiple reasonable options and semantically compatible "safe" embedding sites that exist in the model itself are dynamically and intelligently selected. This fundamentally aligns the watermark embedding behavior with the uncertainty of model generation, avoids introducing disturbances in semantically critical or highly deterministic positions, and ensures that watermark embedding hardly damages the fluency and semantic quality of the text at the source, thus maintaining the original quality and fluency of the text to the greatest extent. 2. In the watermark embedding method of the present invention, cryptographic hash functions and fixed seeds are used to map the lexical units in the candidate word set to group numbers. The candidate word set is divided into multiple groups in a pseudo-random and deterministic manner in advance or in real time. The randomness of hashing ensures the concealment of the watermark pattern, and a direct static mapping between group identifiers and watermark bit segments is established, so that the watermark information is dispersed and randomly woven into the text, making it difficult to detect. It does not depend on model parameters or training, and further ensures that the final word selection is highly related to the context through semantic consistency filtering, thus achieving efficient and deterministic multi-bit information carrying. 3. In the watermark extraction method of the present invention, the candidate set size threshold and confidence threshold are judged for the lexical units in the simulated candidate word set, and the semantic consistency of the current lexical unit and its prefix is ​​judged. By completely reproducing the three-level condition judgment during embedding, the suspected signal is accurately located, which can greatly filter noise, accurately locate tokens that may contain watermark signals, screen out a large number of invalid positions, and significantly reduce computational complexity. Furthermore, the effective position list is periodically sampled at a fixed interval during embedding, and the sampled bit sequence is divided into complete segments and the segment consistency is calculated. Utilizing the prior of periodic embedding of watermark information, the extracted bit sequence is judged based on repetition, which can effectively combat the noise caused by local text modification. The signal is gathered only from the positions that conform to the embedding rules, which significantly improves the accuracy and robustness of extraction and effectively suppresses false alarms. It achieves watermark extraction with high robustness and low false alarm rate without the participation of the original model. 4. The watermark embedding method and watermark extraction method of the present invention systematically solve the triangular dilemma of "capacity, quality and robustness" in the prior art, and realize the embedding of traceable multi-bit information with almost no reduction in text generation quality, and can effectively resist attacks such as semantically preserving modification. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a context-aware multi-bit watermark embedding method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating another context-aware multi-bit watermark embedding method provided in this embodiment of the invention; Figure 3 This is a schematic diagram of watermark information-token group mapping provided in an embodiment of the present invention; Figure 4 A flowchart illustrating a context-aware multi-bit watermark extraction method provided in an embodiment of the present invention; Figure 5 A flowchart illustrating another context-aware multi-bit watermark extraction method provided in this embodiment of the invention; Figure 6This is an example diagram illustrating word-by-word meta-conditional judgment and bit-by-bit decoding provided in an embodiment of the present invention. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0021] The purpose of this invention is to overcome the limitations of existing single-bit, static watermarking methods and provide a multi-bit watermark embedding and extraction method for large language models that integrates adaptive scheduling mechanisms and context-awareness. This method aims to mimic the way humans naturally convey additional information in communication—skillfully selecting words based on the context and importance of the dialogue—thereby achieving high-capacity, highly concealed copyright information embedding with almost no loss in text generation quality. This provides a powerful and practical technical foundation for the credible traceability and controllable governance of AI-generated content. The core of the watermark embedding method lies in deeply binding the watermark signal embedding timing with the model's generation confidence and lexical semantic coherence, achieving a natural and concealed embedding. The watermark extraction process is a reverse security verification of the embedding rules.

[0022] Example 1 Please see Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating a context-aware multi-bit watermark embedding method provided in an embodiment of the present invention. Figure 2 This is a flowchart illustrating another context-aware multi-bit watermark embedding method provided in this embodiment of the invention. The goal of this watermark embedding method is to identify safe locations suitable for watermark embedding without affecting text quality. It is performed synchronously with the autoregressive text generation of large language models (such as GPT, LLaMA, ChatGLM, etc.) using forward inference, and specifically includes the following steps: S11. Obtain the original probability distribution of all candidate word elements in the vocabulary of the current context calculated by the large language model; use a kernel sampling strategy to filter out the candidate word set from the original probability distribution; when the number of word elements in the candidate word set meets the candidate set size threshold, execute step S12.

[0023] Step S11 is the key step in the watermark embedding process. It aims to identify naturally occurring locations with multiple reasonable options during model generation, preventing forced embedding when the model has no other choice, which could lead to a decrease in text quality. Specifically, it includes the following steps: S111. At the moment of generating each token, the large language model, based on the generated context, Calculate the vocabulary The original probability distribution of all candidate lexical units Then output the original probability distribution. Sort in descending order to obtain an ordered list. and their corresponding ordered probabilities ,in, This represents the highest probability.

[0024] S112. First, set a probability accumulation threshold. (Typical value range 0.85~0.95). Then, a kernel sampling (Top-p) strategy is used to adjust the ordered probability. The probabilities in each case are summed from front to back to obtain the cumulative probability. When the cumulative probability is greater than or equal to the cumulative probability threshold ,Right now At that time, corresponding lexical units are selected from the original probability distribution to form a candidate word set. This set It is a dynamic kernel that contains most of the candidate words that the model considers reasonable in the current context, while truncating long-tail low-probability words, which is the basis for ensuring the quality of the generated text.

[0025] S113. Set a minimum candidate set size threshold. And compare the candidate word sets Number of Chinese lexical elements and .when When, it means that the model has at least at this position A reasonable word selection presents an embedding opportunity, and the process proceeds to step S12; when the candidate word set... Number of Chinese lexical elements Less than the candidate set size threshold If the current generation position is deemed unsuitable for embedding, the original probability distribution is used directly. Perform standard sampling and output word units. This process immediately terminates all subsequent watermark embedding operations at the current moment. The determination of embedding opportunity in this step is the primary guarantee of the quality-first principle.

[0026] S12. Calculate the probability difference between the word with the highest probability and the word with the second highest probability in the candidate word set. When the probability difference meets the confidence threshold, proceed to step S13.

[0027] In large language models, the output logits of the last layer is a score for each token in the vocabulary. This is then converted into probabilities using Softmax. ,in, It is the first The logit value of each token. It refers to its predicted probability. To further avoid semantic abruptness caused by forced embedding at "high confidence" positions in the model, step S12 performs a second quality protection based on the positions with multiple reasonable candidate words selected in step S11. Its core lies in further evaluating the model's certainty regarding the optimal choice for candidate positions already determined to have embedding opportunities, thereby avoiding forcibly introducing perturbations at positions where the model already has high confidence. Specifically, it includes the following steps: S121, From the candidate word set The word with the highest recognition probability and its probability and the second most probable word. and its probability .

[0028] S122, Calculate the probability With probability The probability difference between them: Probability difference Intuitively reflects the model's performance and The strength of preference between them; the larger the gap value, the more the model believes. It is far superior to other options, and its choice has a higher degree of certainty.

[0029] S123. First, preset a maximum allowable confidence threshold. When the probability difference Less than or equal to the confidence threshold ,Right now When the model is considered to be in a low-confidence or hesitant state, this position is considered a safe area for embedding the watermark, and step S13 is continued. When the probability difference... Greater than the confidence threshold ,Right now If the model is in a high-confidence state at this moment, forcibly embedding a watermark in this state is highly likely to result in awkward or semantically inaccurate text. Therefore, the system chooses to skip watermark embedding and terminate the subsequent embedding process, continuing to embed according to the original probability distribution. Standard sampling is performed to ensure the fluency and logical consistency of the text.

[0030] S13. Using cryptographic hash functions and fixed seeds, the tokens in the candidate word set are mapped to group numbers. Based on the static mapping relationship and the watermark bits to be embedded, the tokens corresponding to the target group number are selected to form the target candidate group.

[0031] Please see Figure 3 , Figure 3This is a schematic diagram of watermark information-token grouping mapping provided in an embodiment of the present invention. For secure embedding sites that pass the first two rounds of screening, step S13 begins to construct the mapping relationship between the watermark signal and specific words. This process does not rely on a secret key, but controls the grouping through the watermark message itself and a private fixed seed, realizing a self-verifying grouping mechanism for the watermark content. Specifically, the steps include: S131. When deploying the system, set a fixed seed. This is used to ensure consistency in grouping and to determine the number of groups. , It should be a power of 2, for example, the number of groups is 4, 8, or 16. The number of groups determines the number of watermark bits that each token can carry. .For example, When =4, =2, meaning that each embedding position can encode 2 bits of information (00,01,10,11).

[0032] S132. Each word element in the candidate word set Text representation or index (ID) with fixed seed By concatenating the vectors, we obtain the concatenated vector: ,in, Indicates word elements The text representation or index, Indicates splicing.

[0033] S133. Calculate the concatenation vector using a cryptographically secure hash function (e.g., SHA-256). The hash value yields a binary string. ; Translate binary strings Treat it as a very large integer and the number of groups The modulo operation yields each lexical unit in the candidate word set. Group number: This operation will Uniformly mapped to [0, The candidate word set is within the integer range of [-1]. Each word in Each group was assigned a deterministic group number. .

[0034] S134, Based on the preset number of watermark bits With group number Based on the static mapping relationship, the target candidate group is selected from the candidate word set according to the number of watermark bits of the word to be embedded. .

[0035] Specifically, the static mapping relationship between the number of watermark bits and the group number is a predefined mapping. In this embodiment, watermark bit "00" corresponds to group 0; watermark bit "01" corresponds to group 1; watermark bit "10" corresponds to group 2; and watermark bit "11" corresponds to group 3. When it is necessary to embed a specific watermark bit (e.g., "01") at the current position, it is selected from the candidate word set. Select Words with a value of 1 constitute the current target candidate group. Target candidate group It may include multiple lexical units.

[0036] This step achieves a pseudo-random, deterministic mapping of watermark information bits to a subset of candidate tokens.

[0037] S14. The target candidate group is truncated according to the probability of the word in the original probability distribution, and the word with the highest probability is retained to obtain the refined candidate group.

[0038] To ensure the efficiency of subsequent steps and the quality of the final output lexical units, step S14 requires processing the target candidate groups. Refining is then carried out. This includes the following steps: S141, Target candidate group All morphemes in According to their original probability distribution Sort the probabilities in descending order to obtain the sorted list. .

[0039] S142. First, set an in-group cutoff parameter. Based on the group cutoff parameters For the sorted list Truncate, retain only The highest probability of the front each word element Refined candidate group obtained .

[0040] This step preserves a high-probability subset through truncation. Even if a word is hashed into the target candidate group, if its original probability is low, it means the model considers it inappropriate in this context, and forcibly selecting it would severely damage text quality. Therefore, the truncation operation ensures that subsequent selections are only made from high-quality words that the model deems reasonable and belong to the target group, providing a high-quality carrier for watermark embedding.

[0041] S15. Perform semantic consistency filtering on the lexical units in the refined candidate group to obtain a sampleable set; when the sampleable set is not empty, proceed to step S16.

[0042] Step S15, based on the refined candidate groups that have been sorted by probability and truncated, performs a final, deep semantic-based selection. This is a crucial step in ensuring the semantic coherence of the text after watermark embedding. Its purpose is to prevent words that are grammatically or probabilistically reasonable but inconsistent with the deep semantics of the current context from being selected. This ensures that the selected words not only carry the watermark information but also perfectly integrate into the existing textual context, fundamentally guaranteeing the naturalness and fluency of the generated text. Specific steps include: S151. Get the current text above. The end The context semantic vector is obtained by averaging the hidden state vectors of each word in the last layer or an intermediate layer of the large language model. . It encapsulates the recent semantics and intent of the current text.

[0043] S152, For the refined candidate group Each word element Obtain the pre-trained word embedding vectors generated in the large language model. .

[0044] S153, For each word element Calculate its word embedding vector With context semantic vector Cosine similarity: .

[0045] S154. First, set a semantic similarity threshold. Then according to Preserving cosine similarity Greater than or equal to the semantic similarity threshold lexical elements Filter out all < This step removes lexical units that have a weak semantic relevance to the current context.

[0046] S155. For the words retained after threshold filtering, the retained words... According to cosine similarity Sort in descending order and select the top few with the highest similarity. each word element This yields the final sampleable set. .

[0047] Furthermore, after the above filtering and selection, a sample set can be obtained. If the set is empty, it means that among the words that satisfy all the preceding conditions (belonging to Top-p, low gap, belonging to the target group, high probability within the group), none of them sufficiently match the semantics of the current context. In this case, to absolutely prioritize the semantic quality of the text, the system abandons embedding a watermark at the current location and reverts to the standard sampling strategy. When the sampleable set... If the set is not empty, proceed to step S16.

[0048] S16. Perform probability renormalization on each lexical in the sampleable set to obtain a new sampling distribution, and sample from the new sampling distribution based on a preset fixed-interval embedding strategy to generate watermarked lexical.

[0049] Step S16 is based on the final sampleable set determined by strict semantic consistency filtering, which was output in step S15. This is the final operation in the watermark embedding process.

[0050] Before performing this step, the system follows a preset fixed-interval embedding strategy: if embedding fails at the predetermined embedding site, the embedding process is started at the next adjacent position; if embedding succeeds at the predetermined embedding site, the embedding process is started at each interval. The embedding process is initiated at each location. Specifically, the watermark is not attempted to be embedded at every generation location, but rather at every interval. Locations (e.g.) =5) Only then will a complete embedding process starting from step S11 be initiated. If embedding fails at a predetermined embedding site due to failure to pass any of the conditions in steps S11 to S15, the system will not perform probability modulation and sampling in this step. Instead, it will directly use the original probability distribution of the model for standard sampling and immediately restart the complete judgment process starting from step S11 at the next adjacent position (i.e., skipping the fixed interval and switching to continuous attempts) until a watermark unit is successfully embedded. Once successfully embedded, the system will reset the interval counter and wait. After one position, it attempts to embed the next watermark unit. This mechanism ensures that the watermark information is embedded into the text periodically and evenly, while never forcibly disturbing the text generation in inappropriate positions.

[0051] Step S16 specifically includes: S161, Sampleable set Each word element The original probability distribution By summing the results, we can obtain the summation probability: Based on summation probability For each lexical unit in the sampleable set Perform probability renormalization and calculate each word. The watermarked sampling probability distribution: This yields a new sampling distribution. For all samples not included... The word elements in it, Set to 0. Sampling probability distribution That is, the final output distribution after integrating context awareness and watermark information.

[0052] S162, Based on a preset fixed-interval embedding strategy, from the new sampling distribution Mid-sampling generates watermarked lexical units. .because The lexical units in the model all satisfy the following conditions: a) they are high-probability candidates for the model; b) the model's selection of them is not absolutely certain; c) they pseudo-randomly belong to the target watermark group; and d) they are highly relevant to the context semantics. While carrying watermark information, it is almost indistinguishable from naturally generated text, achieving extremely high concealment.

[0053] In this embodiment of the watermark embedding method, the candidate word set is subjected to threshold judgment for candidate set size and confidence threshold judgment, and semantic consistency filtering is performed on the lexical units in the refined candidate group. Through adaptive three-level dynamic scheduling prioritizing generation quality, multiple reasonable and semantically compatible "safe" embedding sites existing in the model are dynamically and intelligently selected. This fundamentally aligns the watermark embedding behavior with the uncertainty of model generation, avoiding the introduction of disturbances at semantically critical or highly deterministic positions. It ensures from the source that watermark embedding hardly impairs the fluency and semantic quality of the text, maximizing the preservation of the original quality and fluency of the text. Therefore, this embedding method ensures that watermark embedding is achieved through dynamic, multi-level conditional scheduling, rather than a crude modification of the original probability distribution.

[0054] In the watermark embedding method of this embodiment, cryptographic hash functions and fixed seeds are used to map the lexical units in the candidate word set to group numbers. The candidate word set is pseudo-randomly and deterministically divided into multiple groups in advance or in real time. The randomness of hashing ensures the concealment of the watermark pattern, and a direct static mapping between group identifiers and watermark bit segments is established, so that the watermark information is scattered and randomly woven into the text, making it difficult to detect. It does not depend on model parameters or training, and further semantic consistency filtering ensures that the final word selection is highly relevant to the context, thus achieving efficient and deterministic multi-bit information carrying.

[0055] Example 2 Building upon Example 1, this example provides a context-aware multi-bit watermark extraction method. Watermark extraction is the inverse symmetric operation of the embedding process, aiming to recover potentially embedded watermark information from a potentially modified text. The extraction process requires access to the original generative model and the same parameters used during embedding (probability accumulation threshold). Candidate set size threshold Confidence threshold semantic similarity threshold Fixed seed S, number of groups (Hash function H) and watermark embedding interval strategy The extraction process is a rigorous hypothesis testing process.

[0056] Please see Figure 4 and Figure 5 , Figure 4 This is a flowchart illustrating a context-aware multi-bit watermark extraction method provided in an embodiment of the present invention. Figure 5 This is a flowchart illustrating another context-aware multi-bit watermark extraction method provided in an embodiment of the present invention. The watermark extraction method specifically includes the following steps: S21. Use the word segmenter of the large language model to segment the text sequence to be detected into a word sequence, and create a list of valid positions and a list of decoded bits.

[0057] Step S21 is the first step in the watermark extraction process. Its core task is to transform the raw text to be verified into standardized, structured data required for subsequent processing. Specifically, it includes: S211. Using the same tokenizer as in the embedding stage, segment the text sequence T to be detected into a sequence of tokens. .

[0058] S212. Create two empty lists: a list of valid positions. and decoded bit list List of valid locations Used to record the position index t of the token in the sequence that is suspected of having an embedded watermark after passing all conditional checks. Decoded bit list Used to record the potential watermark bits (or bit segments, if G>2) decoded at position t. Each entry in The positions in the text correspond one-to-one.

[0059] S22. Input the prefix of the current word into the large language model to obtain the simulated original probability distribution, and use the kernel sampling strategy to reproduce the simulated candidate word set; when the words in the simulated candidate word set pass the candidate set size threshold and confidence threshold, and the current word and its prefix pass the semantic consistency judgment, use the cryptographic hash function and fixed seed to calculate the group number of the current word, map the group number to the watermark bit fragment, store the watermark bit fragment in the decoded bit list, and store the position index of the watermark bit fragment in the text sequence to be detected in the valid position list.

[0060] Please see Figure 6 , Figure 6 This is an example diagram illustrating the word-by-word conditional judgment and bit-by-bit decoding provided in an embodiment of the present invention. Step S22 involves processing each position in the input word sequence X. Simulate the decision-making environment during embedding and perform reverse judgment. For each position... The process follows a symmetrical sequence to steps S11-S15 of the embedding method, filtering out lexical units that meet the prerequisites for watermark embedding and extracting their underlying information. Specifically, this includes: S221, Set the current word element prefix Obtain the original probability distribution of the simulation from the input large language model. A kernel sampling strategy is used to reproduce the simulated candidate word set from the simulated original probability distribution. The method is the same as step S11. When the simulated candidate word set... The number of lexical units is greater than or equal to the candidate set size threshold during the embedding stage. ,Right now When established, execute step S222; when the simulated candidate word set When the number of middle words is less than the candidate set size threshold, the position is determined. If the process fails, proceed to the branch where the watermark is not extracted (i.e., process the next word or end the loop).

[0061] S222, Candidate word set for computational simulation The probability difference between the highest-probability word (Top-1) and the second-highest-probability word (Top-2). When the probability difference Less than or equal to the confidence threshold ,Right now If the condition is met, continue with step S223; when the probability difference... Greater than the confidence threshold If the location fails, the process will proceed to the branch where the watermark is not extracted.

[0062] S223. Using the same method as in step S15, calculate the actual occurrence of the current word element. Its prefix (i.e., the text above) semantic similarity between When semantic similarity Greater than or equal to the semantic similarity threshold ,Right now Established, at this time position After passing all the embedding condition checks, the process enters the actual decoding stage, executing step S224; when semantic similarity... Less than the semantic similarity threshold Then determine the position. Failed. Redirecting to the branch where watermark extraction is not performed.

[0063] Only tokens that pass all three of the above checks are considered to be tokens suspected of having embedded watermarks. For these tokens, their group is further calculated using the same hash function based on their token ID and seed, thereby decoding the multi-bit watermark fragment that may be carried at that location.

[0064] S224. For the current word element judged by all the aforementioned conditions. It is believed that it was selected based on watermark embedding rules. Therefore, its grouping directly encodes the watermark information for the current word. The decoding process is the direct inverse operation of the embedding-time grouping process. The specific process is as follows: First, use the same fixed seed as in the embedding stage. and cryptographic hash functions Calculate the current word element Group number : This calculation is consistent with the grouping calculation method for candidate words during embedding, thus determining... Affiliation within the pre-defined grouping system.

[0065] Then, the group numbers are determined according to the same static mapping relationship predefined in the embedding phase. Decode into the corresponding Bit watermark bit fragment ,in, Number of groups , This represents the number of bits in the watermark. For example, if... =4 and it is specified that group 1 corresponds to bit "01", then =1 means decoded as =“01”.

[0066] Finally, the decoded watermark bit fragments Store the decoded bit list , watermark bit fragment Position index in the text sequence to be detected Store in valid location list This marks the location. A suspected watermark signal unit was successfully extracted.

[0067] Furthermore, if for a certain position If the judgment result in any of steps S221, S222, or S223 is "fail", then that position is considered to not meet the watermark embedding conditions and is completely ignored, and will not be included. and The list. This rigorous filtering mechanism greatly filters out background noise, allowing subsequent processing to focus on signal points that may actually contain watermarks.

[0068] S23. Starting from the first position in the list of valid positions, periodically sample at fixed intervals during embedding. Based on the sampling results, extract the corresponding bit segments from the list of decoded bits and concatenate them in order to form a sampled bit sequence.

[0069] Step S23, based on the pairing list containing position indices and decoded bits output in step S22, performs a crucial signal aggregation operation. This is because the watermark information is embedded at fixed intervals. (For example, each generation) =5 tokens embedded in 1 bit) Repeated embedding to improve robustness and payload. This prior knowledge is used for watermark extraction. Theoretically, true watermark information will exhibit a highly repetitive or regular periodic pattern in the sampled sequence, while random noise or misjudged tokens will not. Specific steps include: First, determine the sampling starting point: from the list of valid locations. The first position start.

[0070] Then at fixed intervals In the list of valid locations Periodic sampling is performed, meaning the sampling position index is: until it exceeds The range. And from the decoded bit list. Extract the bit segments corresponding to these positions and concatenate them in order to form a sampled bit sequence. This sequence should theoretically consist of complete watermark messages. It consists of periodic repetition.

[0071] S24. Divide the sampled bit sequence into several complete segments according to the length of the watermark message. Calculate the proportion of corresponding bits that are the same between any two pairs of segments in the several complete segments and calculate the average value to obtain the segment consistency rate. When it is determined that there is a watermark message in several complete segments based on the segment consistency rate, the watermark message is recovered from the several complete segments using the majority voting principle.

[0072] Step S24 is the final decision stage of the extraction process. It involves performing repeatability feature analysis on the sampled sequence (calculating the autocorrelation of the sequence or checking the repetition rate of watermark codewords at preset intervals) to determine whether the sequence exhibits a significant repeating pattern. The aim is to identify potentially erroneous sequences. The process involves determining whether a watermark exists and retrieving the accurate message. Specific steps include: S241. Let the length of the known watermark message M be... bits (bits) will be the sampled bit sequence Divide into several lengths A fragment of bits, if the last fragment is insufficient Discard the bits, and you get A complete fragment .

[0073] Iterate through all fragment pairs: and , For each pair of fragments, compare whether their bit values ​​at the same positions are the same. For example, compare the 1st, 2nd, ... 1st bit of two fragment pairs, and count how many positions have the same bit value. Calculate the consistency ratio of identical bits. After calculating the consistency ratio of all fragment pairs, average all consistency ratios to obtain the fragment consistency rate. , The value of is between 0 and 1.

[0074] If a watermark exists, each segment Should be consistent with the original watermark message Highly similar, or They should be highly similar to each other. Therefore, the probability that any two segments have the same bit value at the same position will be very high, leading to a high calculated segment similarity rate. The value is close to 1.0. For example, when the fragment consistency rate... If the value is greater than or equal to 0.7, determine if a watermark message exists in several complete segments. If it is random noise, The theoretical expected value is 0.5 (for random independent bits).

[0075] S242. After determining that a watermark exists, for A complete fragment In each bit The process involves checking the bit value (0 or 1) at that position for all segments, and using a majority voting principle, selecting the bit value that appears most frequently as the recovery message for the corresponding bit position, ultimately obtaining the complete watermark message. .

[0076] S243. Finally, the system outputs the extracted watermark message. If the extraction is deemed unsuccessful, an empty or failure flag will be output. If the extraction is successful, the recovered watermark message will be output. It can be further decoded into specific information such as model ID and timestamp.

[0077] In this embodiment of the watermark extraction method, candidate set size threshold and confidence threshold are judged for the lexical units in the simulated candidate word set, and semantic consistency judgment is performed on the current lexical unit and its prefix. By completely reproducing the three-level condition judgment during embedding, suspected signals are accurately located, which can greatly filter noise, accurately locate tokens that may contain watermark signals, eliminate a large number of invalid positions, and significantly reduce computational complexity. Furthermore, the list of valid positions is periodically sampled at fixed intervals during embedding, and the sampled bit sequence is divided into complete segments and the segment consistency is calculated. Utilizing the prior of periodic embedding of watermark information, the extracted bit sequence is subjected to repeatability-based decision, which can effectively combat noise caused by local text modification. The signal is gathered only from the positions that conform to the embedding rules, which significantly improves the accuracy and robustness of extraction and effectively suppresses false alarms. This achieves watermark extraction with high robustness and low false alarm rate without the participation of the original model.

[0078] The watermark embedding and watermark extraction methods of this invention systematically solve the triangular dilemma of "capacity, quality, and robustness" that is difficult to balance in the prior art. By intelligently and covertly embedding multi-bit information in the low-confidence region of the model, it achieves the embedding of traceable multi-bit information without reducing the quality of text generation, and can effectively resist attacks such as semantically preserving modifications.

[0079] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multi-bit watermark embedding method incorporating context awareness, characterized in that, Including the following steps: S11. Obtain the original probability distribution of all candidate word elements in the vocabulary of the current context calculated by the large language model; use a kernel sampling strategy to filter out a set of candidate words from the original probability distribution; when the number of word elements in the set of candidate words meets the candidate set size threshold, execute step S12. S12. Calculate the probability difference between the word with the highest probability and the word with the second highest probability in the candidate word set. When the probability difference meets the confidence threshold, proceed to step S13. S13. Using a cryptographic hash function and a fixed seed, the tokens in the candidate word set are mapped to group numbers. Based on the static mapping relationship and the watermark bits to be embedded, the tokens corresponding to the target group number are selected to form the target candidate group. S14. The target candidate group is truncated according to the probability of the word in the original probability distribution, and the word with the highest probability is retained to obtain the refined candidate group. S15. Perform semantic consistency filtering on the lexical units in the refined candidate group to obtain a sampleable set; when the sampleable set is not empty, proceed to step S16. S16. Perform probability renormalization on each lexical in the sampleable set to obtain a new sampling distribution, and sample from the new sampling distribution based on a preset fixed-interval embedding strategy to generate watermarked lexical.

2. The multi-bit watermark embedding method with fusion context awareness according to claim 1, characterized in that, Step S11 includes: S111, Obtain the current context of the large language model calculation. In the vocabulary The original probability distribution of all candidate lexical units and the original probability distribution Sort in descending order to obtain an ordered list. and their corresponding ordered probabilities ; S112. Employ a kernel sampling strategy to convert the ordered probability... The probabilities in each case are summed from front to back to obtain the cumulative probability. The cumulative probability is greater than or equal to the cumulative probability threshold. The lexical units constitute the candidate word set. ; S113, when judging the candidate word set Number of Chinese lexical elements Greater than or equal to the candidate set size threshold When the candidate word set is determined, step S12 is executed. Number of Chinese lexical elements Smaller than the candidate set size threshold At that time, standard sampling is performed according to the original probability distribution.

3. The multi-bit watermark embedding method with fusion context awareness according to claim 1, characterized in that, Step S12 includes: S121, from the candidate word set The word with the highest recognition probability and its probability and the second most probable word. and its probability ; S122, Calculate the probability With probability The probability difference between them: ; S123, when the probability difference Less than or equal to the confidence threshold When the probability difference is... Greater than the confidence threshold At that time, standard sampling is performed according to the original probability distribution.

4. The multi-bit watermark embedding method with fusion context awareness according to claim 1, characterized in that, Step S13 includes: S131, Set fixed seed And determine the number of groups. ,in, , The number of watermark bits that can be carried by each lexical unit; S132, Each word element in the candidate word set Text representation or index with fixed seed By concatenating the vectors, we obtain the concatenated vector: ,in, Indicates word elements The text representation or index; S133. Calculate the concatenated vector using a cryptographic hash function. The hash value yields a binary string. ; the binary string Treat as an integer and relative to the number of groups The modulo operation is used to obtain each lexical unit in the candidate word set. Group number: ,in, ; S134, Based on the preset number of watermark bits With group number Based on the static mapping relationship, the target candidate group is selected from the candidate word set according to the number of watermark bits of the word to be embedded. .

5. The multi-bit watermark embedding method with fusion context awareness according to claim 1, characterized in that, Step S14 includes: S141, the target candidate group All morphemes in According to the original probability distribution Sort the probabilities in descending order to obtain the sorted list. ; S142. Based on the group cutoff parameter For the sorted list Truncate the section and retain the section with the highest probability. each word element Refined candidate group obtained .

6. The multi-bit watermark embedding method with fusion context awareness according to claim 1, characterized in that, Step S15 includes: S151, Obtain the current preceding text The end The context semantic vector is obtained by averaging the hidden state vectors of each word in the last layer or an intermediate layer of the large language model. ; S152, Obtain the refined candidate group Each word element Word embedding vectors ; S153, Calculate the refining candidate group Each word element Word embedding vectors With context semantic vector Cosine similarity: ; S154. Preserve the cosine similarity. Greater than or equal to the semantic similarity threshold lexical elements ; S155, Retained Lexical Elements According to cosine similarity Sort in descending order and select the top few with the highest similarity. each word element To obtain the sampleable set When the sampleable set If the set is not empty, proceed to step S16; when the sampleable set is... When the set is empty, standard sampling is performed according to the original probability distribution.

7. The multi-bit watermark embedding method with fusion context awareness according to claim 1, characterized in that, Step S16 includes: S161, the sampleable set Each word element The original probability distribution By summing the results, we can obtain the summation probability: Based on the summing probability For the sampleable set Each word element Perform probability renormalization and calculate each word. The watermarked sampling probability distribution: This yields a new sampling distribution; S162. Based on a preset fixed-interval embedding strategy, sample from the new sampling distribution to generate watermarked lexical units. The preset fixed-interval embedding strategy is as follows: if embedding fails at the predetermined embedding site, the embedding process is started at the next adjacent position; if embedding succeeds at the predetermined embedding site, the embedding process is started at each interval. The embedding process is initiated at each location.

8. A multi-bit watermark extraction method incorporating context awareness, characterized in that, Including the following steps: S21. Use the word segmenter of the large language model to segment the text sequence to be detected into a word sequence, and create a list of valid positions and a list of decoded bits; S22. Input the prefix of the current word into the large language model to obtain the simulated original probability distribution, and use the kernel sampling strategy to reproduce the simulated candidate word set; when the words in the simulated candidate word set pass the candidate set size threshold and confidence threshold, and the current word and its prefix pass the semantic consistency judgment, use the cryptographic hash function and fixed seed to calculate the group number of the current word, map the group number to the watermark bit fragment, store the watermark bit fragment in the decoded bit list, and store the position index of the watermark bit fragment in the text sequence to be detected in the valid position list; S23. Starting from the first position in the list of valid positions, periodically sample at fixed intervals during embedding, and extract the corresponding bit segments from the list of decoded bits according to the sampling results and concatenate them in order to form a sampled bit sequence. S24. Divide the sampled bit sequence into several complete segments according to the length of the watermark message, calculate the proportion of corresponding bits that are the same between any two pairs of segments in the several complete segments and calculate the average value to obtain the segment consistency rate; when it is determined that there is a watermark message in the several complete segments according to the segment consistency rate, the watermark message is recovered from the several complete segments by majority voting.

9. The multi-bit watermark extraction method with context awareness according to claim 8, characterized in that, Step S22 includes: S221, Set the current word element prefix Obtain the original probability distribution of the simulation from the input large language model. A kernel sampling strategy is used to reproduce the simulated candidate word set from the original simulated probability distribution. When the simulated candidate word set When the number of lexical units is greater than or equal to the candidate set size threshold, proceed to step S222; when the simulated candidate word set... If the number of Chinese words is less than the candidate set size threshold, the watermark is not extracted. S222, Calculate the simulated candidate word set. The probability difference between the word with the highest probability and the word with the second highest probability, when the probability difference Less than or equal to the confidence threshold When the probability difference is reached, step S223 is executed; Greater than the confidence threshold At that time, the watermark is not extracted; S223, Calculate the current word element Its prefix semantic similarity between When the semantic similarity Greater than or equal to the semantic similarity threshold Execute step S224; when the semantic similarity Less than the semantic similarity threshold Do not extract watermark; S224. Use the same fixed seed as in the embedding phase. and cryptographic hash functions Calculate the current word element Group number : The grouping numbers are determined based on the same static mapping relationship during the embedding phase. Decoded into watermark bit fragments ,in, Number of groups , The number of watermark bits; the watermark bit segment Store the decoded bit list The watermark bit fragment Position index in the text sequence to be detected Store in valid location list .

10. The multi-bit watermark extraction method with fusion context awareness according to claim 8, characterized in that, Step S24 includes: S241. Based on the length of the watermark message M bits will sample bit sequence Divide into several lengths Fragments of bits, and discard those with insufficient fragments. From bits, several complete fragments are obtained. ; Calculate several complete segments The segment consistency rate is obtained by calculating the proportion of corresponding bits that are the same between any two pairs of segments and averaging all consistency proportions. When the segment consistency rate If the value is greater than or equal to 0.7, it is determined that a watermark message exists in the aforementioned complete segments; S242. Check the bit value of each bit in the several complete segments, and use the majority voting principle to select the bit value that appears most frequently as the recovery message for the corresponding bit, thus obtaining the watermark message. .

Images