Improvements to language model watermarking

Abstract

A method of embedding a watermark into an Al LLM output comprising inputting a text prompt to an LLM, the LLM generating a first token text output. Binary reduction is performed on the token text output and generating watermarks using a watermark-seeded random number generator. The watermarks are applied to successive bits of the character set representations of the output by making a decision about accepting a completed token using a second random number generator that is distinct from the first random number generator, wherein the acceptance decision is made with probability equal to the relative probability of the completed token among all remaining possible tokens from a current model PDF. Upon acceptance of a completed token, it is added to the output and a new model PDF is generated. For tokens not accepted, these steps are repeated using the current model PDF. The process is repeated for each token text output of the LLM in response to the text prompt.

Description

IMPROVEMENTS TO LANGUAGE MODEL WATERMARKINGCROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 579,948 filed on August 31, 2023, the entire contents of which are incorporated by reference as part of the disclosure of this document.FIELD OF INVENTION

[0002] The present invention generally relates to watermarking Artificial Intelligence (Al) language models, and particularly to embedding and detecting watermarks in Al language models.BACKGROUND

[0003] This section is intended to provide a background or context to the disclosed embodiments that are recited in the claims. The description herein may include concepts that could be pursued but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.

[0004] When generating text using Al language models, including Large Language Models (LLM’s) it may be desirable to watermark the output so that Al generated text can be identified as such.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] FIG. 1 illustrates a flow chart of a method for embedding watermarks in an Al system in accordance with disclosed embodiments.

[0006] FIG. 2 illustrates a flow chart of a method for detecting watermarks in an Al system output in accordance with disclosed embodiments.

[0007] FIG. 3 illustrates a block diagram of a device that can be used for implementing various disclosed embodiments.SUMMARY OF THE INVENTION

[0008] This section is intended to provide a summary of certain exemplary embodiments and is not intended to limit the scope of the embodiments that are disclosed in this application.

[0009] Systems and methods for embedding and detecting watermarks in LLM generated text. The method includes embedding a watermark into an Al LLM output comprising inputting a text prompt to an LLM, the LLM generating a first token text output. Binary reduction is performed on the token text output and generating watermarks using a watermark-seeded random number generator. The watermarks are applied to successive bits of the character set representations of the output by making a decision about accepting a completed token using a second random number generator that is distinct from the first random number generator, wherein the acceptance decision is made with probability equal to the relative probability of the completed token among all remaining possible tokens from a current model PDF. Upon acceptance of a completed token, it is added to the output and a new model PDF is generated. For tokens not accepted, these steps are repeated using the current model PDF. The process is repeated for each token text output of the LLM in response to the text prompt.

[0010] These and other advantages and features of disclosed embodiments, together with the organization and manner of operation thereof, will become apparent from the following detailed description when taken in conjunction with the accompanying drawings.DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

[0011] In the following description, for purposes of explanation and not limitation, details and descriptions are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that the disclosed embodiments may be practiced in other embodiments that depart from these details and descriptions.

[0012] Additionally, in the subject description, the word “exemplary” is used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word exemplary is intended to present concepts in a concrete manner.

[0013] Embodiments are disclosed which enable the embedding and detecting of watermarks in Al language models. Christ [M. Christ, S. Gunn, and O. Zamir, “Undetectable Watermarks for Language Models,” May 25, 2023, which is incorporated herein by reference] haspublished a method for watermarking text generated by LLMs with a number of advantageous properties including preservation of the model distribution, selfsynchronization, and data capacity adaptation to model entropy.

[0014] A limitation, however, that this method has (and that it also shares with some other contemporaneous methods, such as J. Kirchenbauer, et al., “A Watermark for Large Language Models,” June 6, 2023, http: / / arXiv.org / abs / 2301.10226v3, and J.Kirchenbauer, et al., “On the Reliability of Watermarks for Large Language Models,” June 30, 2023, htp : / / arXiv.org / abs / 2306.04634v3 , which are both incorporated herein by reference] is that the watermark is conveyed in - and therefore must be both embedded in and detected from — the token identifier values rather than the text output itself. We consider this to be a limitation for the following practical reasons.

[0015] First, it requires deterministically invertible tokenization. That is, if the re- tokenization of text performed in order to detect the watermark generates a differenttoken sequence than was used to produce the text, the watermark will not be detected successfully. This can impact not only symbols embedded in the differently-tokenized text, but can also cause “desynchronization” of the watermark detector, causing detection to fail across surrounding text.

[0016] This first limitation can be a problem even when the tokenization scheme used with a model has the property of deterministically converting text to tokens, because use of a deterministic tokenizer cannot guarantee that generation from the model will not output a sequence of tokens that will be deterministically converted to a different sequence. For example, even if a tokenizer is designed such that it will deterministically convert the text “tech” to the token “ / tech / ”, there is still no guarantee that the model will not output the sequence of tokens “ / te / ch / ”, generating text that would be re-tokenized to a different token sequence (i.e. “ / tech / ”) than the model output.

[0017] Second, it imposes the significant burden on detection of the watermark that the tokenizer (including its algorithm details as well as any associated dictionary, probability tables, etc.) be known and available in addition to the secret key. This can present a significant practical problem in a number of important use cases for watermark detection, including: (a) creation of a watermark detector that will work on text for which the identity of the model used to generate the text is unknown; (b) creation of a watermark detector for which the tokenizer algorithm is maintained as a secret; (c) creation of a watermark detector that will work for future language models whose tokenizers have not yet been specified.

[0018] It would be highly beneficial to be able to realize the advantageous benefits of watermarking realized by Christ and other watermark technologies without these limitations.

[0019] Solution

[0020] The Christ approach can be adapted such that the binary reduction and watermarking-based selection and detection is applied to the token text (i.e. the ASCII or UTF-8 characters that comprise the text associated with the tokens) rather than the token identifier values.

[0021] The Christ approach cannot be directly applied, because while their algorithm operates on token identifiers of fixed length (log|T| bits), tokens are of non-uniform character length.

[0022] We apply an approach related to Christ’s binary reduction to the tokens and apply watermarks to successive bits of their character set representations (or equivalently, the successive symbols of an alternate model-independent digital representation of the text), and when you reach a point at which one of the options is to end a token, you make a decision whether to “accept” the completed token. In order to preserve the model distribution, the acceptance decision is made using a random number source distinct from that used for watermark generation (i.e. not seeded by the secret watermarking key) and an acceptance decision is made with probability equal to the relative probability of the completed token among all remaining possible tokens from the model PDF. If the completed token is accepted, it is added to the output and a new model distribution is generated and the Christ watermarking / bit selection process continues to determine the character sequence of the next token. If the token is not accepted, you continue selecting bits using the Christ watermarking / bit selection process on the remaining tokens in the current model-generated probability distribution function (PDF) (i.e. shorter tokens are removed from further consideration).

[0023] By way of example:

[0024] Suppose:

[0025] Token dictionary is tl=”a”, t2=”b”, t3=”bc” and t4=”be”, character set is 7-bit ASCII

[0026] Model PDF for the next token with pl=0.5, p2=0.25, p3=0.15 and p4=0.1.

[0027] Christ’s binary reduction and watermark application approach has been applied to bits of the first character of the token model PDF distribution using the adaptation described above seven times and the watermark- seeded random number generator (RNG) has so far generated the character “b” using bitwise probabilities determined at the character level.

[0028] The end of token t2 has been reached, and t2, t3, and t4 are remaining possible tokens.

[0029] We now need to decide whether to either (a) accept t2 as our selected token from this model PDF; or (b) eliminate t2 from consideration and proceed with selecting between bits in t3 and t4.

[0030] The probability of accepting p2 is equal to its relative probability among p2 p3, and p4, which is calculated as p2 / (p2+p3+p4)=0.25 / 0.5=0.50.

[0031] Note that if p2 is not accepted (50% likely), for the next bit t3 and t4 are the only remaining candidates.

[0032] Note also that it is possible that there will be only one remaining candidate before you reach the end of that token; i.e. the remaining bits have p=l . In the given example, assuming that the binary reduction proceeds from MSB to LSB, this is guaranteed to occur once the final bit of t3 and t4 is reached, because their final characters have the same LSB value. This is the same scenario as occurs in the original Christ approach when there is only a single option in the PDF for the next token. Entropy is 0 and the selection of bit values cannot be influenced by the watermark-seeded RNG. These bits carry no watermark information, but the watermark- seeded RNG must advance its sequence across them to maintain synchronization.

[0033] Detection of watermarks is performed on the candidate text characters, without the need for converting to tokens. Apart from this, the method described by Christ is applicable; it is simply applied to a binary reduction of the candidate text rather than to the re-tokenized candidate text.

[0034] We note that the innovative binary reduction approach that is disclosed herein is a powerful generalization of the red / green reduction of Kirchenbauer. Christ’s approach reduces each token identifier to log(|T|) bits, whereas Kirchenbauer reduces each token identifier to 1 bit only. The two approaches use different modulation, seeding, and thresholding schemes, but the method disclosed herein of applying the binary reduction to the token characters can be applied in either case, enabling more watermark bits to be modulated into each token than either Christ or Kirchenbauer achieves.

[0035] The disclosed embodiments avoid the stated problems of: (1) requiring that the tokenization be correctly inverted for watermark detection to occur; and (2) requiring any use of the tokenizer for detection.

[0036] The disclosed embodiments also provide direct advantages by enabling: (a) creation of a watermark detector that will work on text for which the identity of the model used to generate the text is unknown; (b) creation of a watermark detector for which the tokenizer algorithm is maintained as a secret; and (c) creation of a watermark detector that will work for future language models whose tokenizers have not yet been specified.

[0037] There are a number of practical benefits associated with these capabilities, particularly as relates to creating watermark methods where embedding and detection may be used by multiple different parties or over long periods of time, as may be the case in a method that might be adopted as an industry standard, similar to the way in which standardized cryptographic primitives have been developed whose designs are sufficiently general to be applicable in many different systems, whose security properties have been disclosed and studied extensively to obtain assurance of efficacy, and forwhich reliable implementations can be created, extensively tested to establish trust, and used by many parties so as to achieve efficiencies of scale and interoperability.

[0038] These capabilities can be particularly beneficial in instances where watermarked content is provided as training data or a prompt to a generative Al system. In each of these instances, it may be valuable for the watermark to be detected from the content prior to use. In the case of watermarked content provided as training data, it may be preferred that the content be either excluded from training or tagged in training based on information conveyed by the watermark.

[0039] It may also be desirable to alter the behavior of a generative Al system based on information detected in watemiarks in prompt inputs. The watermark information can be used as the basis for rejecting the content, altering model behavior in its generation inference process, or changing the watermark information that is embedded in the model output. The above described Christ reference at section 6.1 notes that text watermarks can be vulnerable to translation and paraphrasing attacks carried out by generative Al systems. In each of these cases, if the generative Al system is able to read the watermark, it has the opportunity to carry the detected watermark information forward into a watermark that it embeds into the output. This “carry forward” process could constitute recreating the watermark in the prompt or could be a new watermark with information that allows both information contained in the input text and new information, such as information about the new generation process, to both be recovered. Alternatively, an entirely new watermark can be generated and information associating the two watermarks (original and new) can be recorded in a database applying the watermark read / modify / write method described in U.S. Patent No. 10,848,821 and US Patent Application No. 2022 / 0294762, which are incorporated herein by reference. For all of these use cases, it is highly desirable that practical and efficient mechanisms exist for detecting watermarks that have been embedded by any of a number of different generative Al systems as enabled by the improvements described herein.

[0040] In other embodiments the above described techniques are applied to generative Al models that generate non-text outputs such as audio, images, or video content that can be represented by stable, model independent, discrete symbolic representations.

[0041] It is understood that the various embodiments of the present disclosed embodiments may be implemented individually, or collectively, in devices comprised of various hardware and / or software modules and components. These devices, for example, may comprise a processor, a memory unit, an interface that are communicatively connected to each other, and may range from desktop and / or laptop computers, to consumer electronic devices such as media players, mobile devices, and the like. For example, FIG. 3 illustrates a block diagram of a device 1000 within which the various disclosed embodiments may be implemented. The device 1000 comprises at least one processor 1002 and / or controller, at least one memory 1004 unit that is in communication with the processor 1002, and at least one communication unit 1006 that enables the exchange of data and information, directly or indirectly, through the communication link 1008 with other entities, devices and networks. The communication unit 1006 may provide wired and / or wireless communication capabilities in accordance with one or more communication protocols, and therefore it may comprise the proper transmitter / receiver antennas, circuitry and ports, as well as the encoding / decoding capabilities that may be necessary for proper transmission and / or reception of data and other information.

[0042] Referring back to FIG. 3 the device 1000 and the like may be implemented in software, hardware, firmware, or combinations thereof. Similarly, the various components or sub-components within each module may be implemented in software, hardware, or firmware. The connectivity between the modules and / or components within the modules may be provided using any one of the connectivity methods and media that is known in the art, including, but not limited to, communications over the Internet, wired, or wireless networks using the appropriate protocols.

[0043] Various embodiments described herein are described in the general context of methods or processes, which may be implemented in one embodiment by a computer program product, embodied in a computer-readable medium, including computerexecutable instructions, such as program code, executed by computers in networked environments. A computer-readable medium may include removable and non-removable storage devices including, but not limited to, Read Only Memory (ROM), Random Access Memory (RAM), compact discs (CDs), digital versatile discs (DVD), etc. Therefore, the computer-readable media that is described in the present application comprises non-transitory storage media. Generally, program modules may include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instractions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.

[0044] The foregoing description of embodiments has been presented for purposes of illustration and description. The foregoing description is not intended to be exhaustive or to limit embodiments of the present disclosed embodiments to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of various embodiments. The embodiments discussed herein were chosen and described in order to explain the principles and the nature of various embodiments and its practical application to enable one skilled in the art to utilize the disclosed embodiments in various alternative embodiments and with various modifications as are suited to the particular use contemplated. The features of the embodiments described herein may be combined in all possible combinations of methods, apparatus, modules, systems, and computer program products.

Claims

WHAT IS CLAIMED IS:

1. A method of embedding a watermark into an Al language model output comprising: a) inputting a text prompt to an Al language model, the Al language model generating a first token text output; b) performing binary reduction on the token text output; c) generating watermarks using a watermark- seeded random number generator; d) applying the watermarks to successive bits of character set representations of the output by making a decision about accepting a completed token using a second random number generator that is distinct from the first random number generator, wherein the acceptance decision is made with probability equal to the relative probability of the completed token among all remaining possible tokens from a current model PDF; e) upon acceptance of a completed token, adding it to the output and generating a new model PDF; f) for tokens not accepted, repeating step d using the current model PDF ; and g) repeating steps b - e for each token text output of the Al language model in response to the text prompt, whereby the text output generated by the Al language model comprises watermarked text.

2. The method according to claim 1, further comprising providing the watermarked text as training data to an Al system.

3. The method according to claim 1, further comprising based on information conveyed by the watermark, excluding the watermarked text from being used to train an Al system.

4. The method according to claim 1, further comprising based on information conveyed by the watermark, tagging watermarked text being used to train an Al system.

5. The method of claim 1 further comprising altering the behavior of an Al system based on information detected in watermarks in prompt inputs.

6. The method of claim 5 wherein the altering the behavior comprises at least on of: rejecting the content; altering model behavior in the Al system’s generation inference process; and changing the watermark information that is embedded in the model output.

7. The method according to claim 1, further comprising providing the watermarked text as a prompt to a generative Al system.

8. A method of detecting a watermark embedded into an Al language model output comprising: detecting a watermark in an Al language model output, the watermark having been embedded according to the following steps: a) inputting a text prompt to an Al language model, the Al language model generating a first token text output; b) performing binary reduction on the token text output; c) generating watermarks using a watermark-seeded random number generator; d) applying the watermarks to successive bits of the character set representations of the output by making a decision about accepting a completed token using a second random number generator that is distinct from the first random number generator, wherein the acceptance decision is made with probability equal to the relative probability of the completed token among all remaining possible tokens from a current model PDF ; e) upon acceptance of a completed token, adding it to the output and generating a new model PDF ; f) for tokens not accepted, repeating step d using the current model PDF; and g) repeating steps b - e for each token text output of the Al language model in response to the text prompt, whereby the presence of the embedded watermark is detected.[Claims 9-12 are based on the process described in the above paragraph 37 at lines 5-18]9. The method according to claim 8 further comprising: reading the detected watermark by an Al system; carrying the detected watermark information forward by the detected watermark information embeds into its output10. The method according to claim 9 wherein the carrying the detected information forward comprises recreating the watermark in the prompt.

11. The method according to claim 9 wherein the carrying the detected information forward comprises generating a new watermark containing information that allows both information contained in the input text and new information, including information about the new generation process.

12. The method according to claim 9 further comprising: generating a second watemiark; recording in a database information associating the watermark and the second watermark; and applying a watermark read / modify / write method.

13. A method of embedding a watermark into an Al language model output comprising: a) inputting a text prompt to an Al language model, the Al language model generating a first token text output; b) performing binary reduction on the token text output; c) generating watermarks using a watermark-seeded random number generator;d) applying the watermarks to the successive symbols of an alternate modelindependent digital representation of the output text by making a decision about accepting a completed token using a second random number generator that is distinct from the first random number generator, wherein the acceptance decision is made with probability equal to the relative probability of the completed token among all remaining possible tokens from a current model PDF; e) upon acceptance of a completed token, adding it to the output and generating a new model PDF ; f) for tokens not accepted, repeating step d using the current model PDF ; and g) repeating steps b - e for each token text output of the Al language model in response to the text prompt, whereby the text output generated by the Al language model comprises watermarked text.

Images