A process for generating tokens from input datasets of various lengths

Abstract

A computer-implemented process for generating tokens from an input dataset including input values ​​of varying lengths is provided, the process including generating tokens from the input dataset, the tokens having lengths that exhibit a distribution substantially similar to the distribution of lengths of the input dataset.

Description

[Technical Field]

[0001] The field of the invention relates to computer-implemented methods or processes for generating tokens from input data sets of various lengths and for embedding digital watermarks within the generated tokens, and to associated computer-implemented devices or systems.

[0002] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. [Background technology]

[0003] Tokenization involves replacing a private identifier, such as an individual's credit card number or social security number, with a token that is generated to fit a user-specified format and has a 1:1 relationship with the original private identifier. This same token is always used in place of the same identifier and never for any other identifier.

[0004] Digital watermarking relates to the process of embedding information, called a digital watermark, into digital content while preserving the functionality of the digital content.

[0005] WO2017 / 093736(A1) discloses a process for modifying an original dataset by combining data anonymization and digital watermarking. In particular, anonymization of the original dataset can be achieved using tokenization techniques, where the tokenized values ​​are generated using regular expressions. However, the regular expressions must be known at the time of watermark extraction. Furthermore, the tokenization techniques used involve a central vault, which can pose problems for customers with high throughput needs or requirements to consistently tokenize values ​​at remote locations.

[0006] A need exists for a process in which digital watermarking is applied to tokenized data without knowledge of the regular expressions used to tokenize the data. Additionally, a system is needed that scales to allow any number of data releases to be individually watermarked.

[0007] Additionally, tokenization techniques have proven effective in handling fixed data structures, such as credit card numbers or zip codes. However, tokenizing input data of varying lengths, such as names or email addresses, presents complex challenges. Thus, there is a need for a process that can provide a robust solution for handling input data of varying lengths.

[0008] Additionally, there is a need for a process for embedding information or a digital watermark into digital content generated using generative AI, such as within an image or video frame or audio or text.

[0009] Reference is made to WO2017 / 093736(A1), the contents of which are incorporated by reference. Summary of the Invention

[0010] One aspect of the invention is a computer-implemented process for generating tokens from an input dataset that includes input values ​​of varying lengths, the process including generating tokens from the input dataset, the tokens having lengths that exhibit a distribution substantially similar to the distribution of lengths of the input dataset.

[0011] Another aspect of the present invention is a computer-implemented process for embedding a digital watermark in tokenized data, the process comprising: (a) generating, from an input dataset comprising input values ​​of varying lengths, tokens generated with lengths exhibiting a distribution substantially similar to the distribution of lengths of the input dataset; (b) embedding a digital watermark within the generated set of tokens.

[0012] Another aspect of the invention is a computing device or system adapted to generate tokens from an input dataset comprising input values ​​of varying lengths, the device or system including a processor configured to generate tokens from the input dataset, the tokens having lengths exhibiting a distribution substantially similar to the distribution of lengths of the input dataset.

[0013] Another aspect of the invention is a computing device or system adapted to embed a digital watermark in tokenized data, the device or system including a processor, the processor comprising: (a) generating, from an input dataset, tokens generated with lengths exhibiting a distribution substantially similar to the distribution of lengths of the input dataset; (b) embedding a digital watermark within the generated set of tokens. [Brief explanation of the drawings]

[0014] Aspects of the present invention will now be described, by way of example, with reference to the following drawings, each of which illustrates features of the present invention:

[0015] [Figure 1] A histogram of token counts within each bin is shown. [Figure 2] A diagram representing the space of possible inputs is shown. [Figure 3] 1 shows a diagram representing a token space with watermarked tokens uniformly distributed throughout the token space. [Figure 4] 1 shows a diagram illustrating an algorithm for mapping an input space to a token space based on two encryption schemes. [Figure 5] A diagram (5A) is shown containing 53 watermark tokens distributed over a token space of size 400, and another diagram (5B) is shown containing the token space divided into 53 segments. [Figure 6] A diagram illustrating the process for two example segments, one containing an actual watermark token and one not, is shown. [Figure 7] FIG. 1 shows a diagram illustrating an input space and a token space. [Figure 8] FIG. 1 shows a diagram illustrating steps for determining the index of a value in an input space. [Figure 9] 1 shows a diagram illustrating encryption of a subspace index. [Figure 10] 1 shows a diagram illustrating how to find the token space segment to which an input maps. [Figure 11] 1 shows a diagram illustrating how a segment can be searched to find a desired token. [Figure 12] FIG. 10 shows a diagram illustrating an output with final token ordinal numbers. [Figure 13] FIG. 1 shows a diagram illustrating a hash space divided into equal-width bins. [Figure 14] FIG. 1 shows a diagram illustrating the process of extracting a watermark. [Figure 15] FIG. 1 shows a diagram illustrating the process of extracting a watermark when multiple hash functions are used. [Figure 16] FIG. 1 shows a diagram illustrating the process of extracting a watermark on a set of parallel hash arrays. [Figure 17] A diagram illustrating the parameters of the algorithm is shown. [Figure 18] Three histograms of token counts within each bin are shown for a "pure" watermark (18A), a noisy watermark (18B), and two mixed watermarks. [Figure 19] FIG. 10 shows a diagram illustrating extraction token count requirements as the number of data releases increases. [Figure 20] 1 shows a plot of tokens required as a function of input noise percentage. [Figure 21] FIG. 10 shows a diagram illustrating extraction token count requirements as the number of data releases increases. [Figure 22] 1 shows a plot of tokens required as a function of input noise percentage. [Figure 23] 10 shows a plot of normalized computation time for watermark embedding. [Figure 24] 10 shows a plot of normalized computation time for watermark extraction. [Figure 25] We show the final results of watermark extraction performed at a 95% confidence level as the number of tokens processed in the extraction increases. [Figure 26] The results of watermark extraction performed at a 99.9% confidence level are shown. [Figure 27] 1 shows the results of watermark extraction where random noise is progressively added. [Figure 28] A plot of the false positive rate for the same experiment is shown. [Figure 29] FIG. 1 shows a diagram illustrating the number of tokens required to achieve authenticity. [Figure 30] We present experimental results where two data release watermarks are mixed together. [Figure 31] We present experimental results where three data release watermarks are mixed together. [Figure 32] FIG. 1 shows a diagram illustrating the process of extracting a watermark when multiple hash functions are used and different bins within each hash function for data release are used. [Figure 33] FIG. 1 shows a diagram illustrating a watermarking algorithm that maps a two-dimensional input space to a token space based on a deterministic tokenization scheme. [Figure 34]Here is a graph illustrating the character length distribution of common surnames taken from the 1990 U.S. Census. [Figure 35] 10 shows a graph illustrating the corresponding token distribution when plotted on a chart with linear scale axes. [Figure 36] Shown is a graph of the length of the generated tokens against the first length 2, using a standard deviation of 1 / 3 of the range. [Figure 37] 1 shows a graph of length tokens generated as a function of the number of tokens of that length assigned to a two-character input. [Figure 38] 1 shows a diagram of the ciphertext ordinal index as a function of the region of the ciphertext space to which it belongs. DETAILED DESCRIPTION OF THE INVENTION

[0016] Embodiments of the present invention propose a computer-implemented process that incorporates digital watermarking in addition to deterministic tokenization.

[0017] We may refer to the following terms throughout the description.

[0018] Input Space - The space of all possible inputs from the original dataset that may need to be tokenized. The input space may be described using regular expressions. For example, when tokenizing credit card numbers, a simple input space definition may be "[0-9]{16}" - 16 decimal digits (this example ignores the complications of not all prefixes being valid, Luhn digit checks, etc.).

[0019] Token Space - As above, this is the space of all possible tokens that can be returned.

[0020] Tokenized Data − Data where input values ​​have been replaced by tokens.

[0021] Data Release—Broadly speaking, refers to any release of tokenized data to a specific recipient for a specific purpose. Each data release is therefore associated with its own digital watermark. The digital watermark may be a number or other “ID” that is stored in a watermark registry along with metadata. Thus, by extracting the watermark from a release, any metadata associated with the data release may also be obtained. The metadata may include, for example, one or more recipients enabled to receive the data release, the purpose or intended use of the data release, the period for which one or more recipients are legally enabled to retain the data, and with whom the data is enabled to be shared.

[0022] Watermark Tokens - As will become clear in the following description, the hash-based watermarking scheme used works by not returning any tokens that hash to a value that falls within a watermark bin; these tokens are called watermark tokens. The token space therefore consists of watermark tokens (tokens that hash to watermark bins) and non-watermark tokens (tokens that hash to other bins).

[0023] Watermark Inputs - In deterministic tokenization, a 1:1 mapping from all inputs to all tokens must be fixed. Therefore, some of these inputs will map to watermark tokens, but we do not want to return these tokens. Therefore, the core of the scheme is that we influence the mapping so that watermark tokens are mapped to inputs that are unlikely to occur; we call these inputs watermark inputs. As an example, if we are tokenizing credit card numbers using the regular expression described above, we can choose our watermark inputs to be numbers that start with 0000, because those numbers are not used in actual credit card numbers and we will not encounter them. The mapping of inputs to tokens is performed on the fly by an algorithm, as described below.

[0024] Watermarking Prior watermarking techniques, such as that disclosed in WO 2017 / 093736(A1), allow the generation of tokens to be controlled so that a pattern is embedded within these tokens. This pattern can vary with each data release, allowing a unique identifier for the release to be embedded within and throughout the data itself. This identifier can be used as a pointer to any store of metadata about the data release, such as the intended recipients and purpose of the release, its lineage including any privacy processing applied to it, the date by which the data should be deleted, etc. This embedded pattern is probabilistic and extractable from a sample of generated tokens rather than relying on any individual token, and thus can still be extracted from a sufficiently large subset of the data release.

[0025] Rejection sampling-based algorithms work by rejecting potential tokens (and generating alternative tokens instead) according to some pattern; then, a corpus of watermarked data is scanned to reconstruct the patterns, thus learning the watermark. The patterns embedded by the algorithm are based on the hash of the tokens—the hash space is divided into bins, each of which is assigned to a data release; then, when watermarking a data release, we want to reject any tokens that hash to a value that falls within the hash bin of the current data release (each data release is assigned a different slice of the hash space). Then, as shown in Figure 1, if we scan the watermarked data, hash the tokens, and build a histogram of token counts within each bin, we can identify empty bins and, therefore, the data release to which the watermark belongs.

[0026] Vault-based vs. deterministic (vaultless) tokenization The process described above works with vault-based tokenization because we have the opportunity to choose the tokens we assign to an input at the time we tokenize that input. As long as we encounter an input with fewer tokens than possible, we do not have to return all tokens, and we can choose to never return those tokens that hash to a watermark bin.

[0027] Therefore, rejection sampling has been achieved using a tokenization system that randomly generates tokens that match a required format and stores the generated tokens in a persistent data store (a "token vault"). At the time of generating the token, if a candidate value is generated that should be rejected, another candidate value can simply be generated. However, this reliance on a central token vault can cause problems for customers with high throughput needs or a requirement to consistently tokenize values ​​at remote locations.

[0028] For this reason, it may be preferable to use a tokenization scheme that is algorithmic (e.g., based on a format-preserving encryption cipher) and therefore fully deterministic. Under such a scheme, it is assumed that for every possible input (i.e., matching a defined format), there is a mapped token - hence, it is not possible to implement rejection sampling watermarking with this system; if an input is encountered that maps to a token that should be rejected, there is no choice but to release that token. This is due to the fact that it is not possible to choose a different token, as other tokens would map 1:1 to different inputs, which would ultimately break the tokenization scheme. A solution to this problem is presented in the following paragraphs.

[0029] Watermarking Deterministic Tokenization An embodiment of the present invention is a method for performing rejection-based watermarking on top of deterministic tokenization. It uses the observation that the probability of encountering a particular input is often not uniform across the input space, and attempts to assign tokens to those inputs that are least likely to be encountered. This is achieved using a combination of two format-preserving encryption ciphers—one that maps the least commonly encountered inputs to "rejected" or watermark tokens, and another that maps the remaining (most) inputs to other tokens or non-watermark tokens.

[0030] Advantageously, the digital watermark is embedded within the generated token set and not embedded in any metadata or redundant data.

[0031] With deterministic tokenization, as the name implies, we have a predetermined mapping of which tokens will be returned for each input before tokenization begins. There is a single regular expression defined that represents both the tokens to be returned and the input that will be encountered, and the mapping is defined in terms of this expression - for example, according to expression [AZ], input ordinal 5 (i.e., E) may be mapped to token ordinal 10 (i.e., J).

[0032] At the time of tokenizing the input, we do not have the flexibility to choose any token other than the mapped token, regardless of whether the token falls within a hash bin that we want to leave empty.

[0033] To embed a perfect watermark, we absolutely need to avoid returning tokens that hash to values ​​that fall within a slice of hash space designated as a bin assigned to the data release. However, watermarking extraction algorithms are tolerant to the addition of some level of random noise. Such noise reduces the reported confidence level for a watermark query, which may require more tokens to be scanned before the required confidence is reached, but does not prevent successful extraction (the relationship between noise and the number of tokens required to reach the confidence threshold is well understood).

[0034] Therefore, to embed a watermark, we need to meet the following two criteria: 1. It requires that there are some possible values ​​that are encountered less frequently than others (the difference in relative frequency will determine the noise level of the final watermark - if there are enough values ​​in the input space that never actually appear in the data, it can return a perfect watermark, but if not, there will be some level of noise). 2. We need some way to assign watermark tokens to these inputs (or, equivalently, assign non-watermark tokens to inputs in which they occur).

[0035] When using a token vault, these criteria are met - we naturally discover inputs that are present in the data (we are never asked to tokenize inputs that never appear), and we can choose the tokens to assign to inputs at the time of tokenization. To make watermarking work with deterministic tokenization, we also need to design a way to preserve them.

[0036] Vaultless tokenization has several advantages, such as a distributed deployment where a centralized vault cannot often be invoked. In contrast, globally distributed vault-based tokenization requires that raw data be transmitted to the token vault along with the token, since both must be stored in the centralized vault. However, this transmission of raw identifiers from one jurisdiction to another is often contrary to law or regulation.

[0037] Figure 2 shows a diagram representing the space of possible inputs (each possible input is a small square representing its ordinal number in the total space of 400 possibilities), where some of the inputs (shown in light grey) have been identified as unlikely to be observed and should therefore be mapped to watermark tokens.

[0038] We wish to map these inputs (which we call watermark inputs) to tokens that hash to values ​​that fall within the data release watermark bins (watermark tokens). The cryptographic hash function (based on a secret "watermarking key") used within the watermarking algorithm will distribute hashes uniformly across the hash space. This implies that hashes that fall within the hash space will be from values ​​that are uniformly drawn from the entire value space - i.e., the watermark tokens will be uniformly distributed across the token space, as shown in Figure 3.

[0039] However, the mapping from input to its tokens is determined by the underlying format-preserving encryption cipher and is a permutation indistinguishable from random - a hardwired mapping to the cipher is not possible (at least not without devising one's own non-standard and necessarily insecure encryption cipher). To realize a scheme where a subset of the input space is mapped to a subset of the token space, we must treat the subspace as a separate space with its own separate encryption cipher. To encrypt a value, we follow these steps: 1. Determine which of the distinct subspaces contains the value. 2. Encrypt the value using the cipher for that space, resulting in another index in the same space. 3. Map this output index into the global token space to find the resulting token.

[0040] To implement this solution, we need to solve two problems. 1. Determine which inputs are watermark inputs that should be mapped to watermark tokens 2. Determine which tokens are watermark tokens (i.e., hash to values ​​within watermark bins) Figure 4 illustrates these steps.

[0041] Thus, each format-preserving encryption cipher uses a secret key. A further secret key - the watermarking key - is used by a cryptographic hash function to prevent an attacker from learning whether a token is a watermark token or not.

[0042] Modeling the distribution of input values To take advantage of the distribution of values ​​that occur in an input dataset, we first need to know what this distribution is - using this information, we can try to figure out which entries occur rarely or never, and assign watermark tokens to those entries. An example approach to this is described next.

[0043] Data Description In some situations, there is some a priori knowledge of the structure of the input data that can be used to describe regions of the input space that will never be encountered. For example, not all numbers that conform to the structure of a Social Security number are actual Social Security numbers, since numbers containing 666 or 900-999 in the first group of digits are never assigned.

[0044] Make assumptions about the data If the data has some external meaning—examples include names, email addresses, and salaries—then it may fit some general rules of thumb about the data that we can use to make educated guesses about where to assign watermark tokens. For example, in English text, the digraph "th" is likely to be much more frequent than "qz," and for many numerical distributions, the probability density function is often low at the high end of the range.

[0045] We can use this to assign watermark tokens - for example, if we are presented with a regular expression of [AZ][az]{1,9}, we can define a range such as (Qz|Jq|Jx|Qx)[az]{0,8} to capture those inputs that we expect to occur least frequently (or never).

[0046] Benford's Law states that in many numeric data sets, leading digits are likely to be small, and the probability that a particular digit is the leading digit decreases logarithmically as the digit number increases. Benford's Law generally applies to data sets with log-normal distributions, so its direct usage is perhaps not general enough, but we can generalize that for numeric data that falls within some defined range, the probability density function is often low at the upper end of the range. This holds true for log-normal distributions spanning several digits that satisfy Benford's Law, as well as normal distributions (e.g., height), distributions with long tails (e.g., salary), and monotonically increasing values ​​(e.g., identifiers pulled from database sequences). Therefore, simply assigning a watermark token to the upper end of a numeric range can be expected to give good results in many datasets.

[0047] Scanning the Data If nothing is known a priori about the input distribution and the assumptions in the previous section do not hold, it may be possible to infer the input distribution by scanning the data and observing it. Note, however, that this may be a weaker solution than previous techniques, because the configuration must be determined from scanning the current data, but there is no guarantee that future data will have the same distribution. For example, if the data is drawn randomly from space with uniform probability, we may be able to find regions in space that have no values ​​to assign watermark tokens to, but future values ​​may fall within these ranges (in contrast, data that never contains special digits in social security numbers and follows Benford's law will continue to do so in the future).

[0048] Identifying Watermark Tokens A watermark token is defined as a token that hashes to a value that falls within a watermark bin. Because hash functions are one-way functions, there is no way to take the hash values ​​of watermark bins and find tokens that hash to them. The only way to find if a token's hash falls within a bin is to hash it to find out; attempting to brute-force the entire token space to find all of the watermark tokens is impractical for all but a very small number of token spaces. Instead, we use the fact that watermark tokens will be uniformly distributed throughout the token space. This means that if we divide the token space into segments, on average, these segments will contain an equal number of watermark tokens—for example, if we divide the space into the same number of segments as there are watermark tokens, on average, each segment will contain a single watermark token (some segments may contain none, and some will contain multiple segments, but on average, each segment will contain a single watermark token). As an example, Figure 5A shows a diagram with the distribution of 53 watermark tokens across a token space of size 400. Figure 5B shows the token space divided into 53 segments (29 of size 8 and 24 of size 7) filled with different patterns to delimit the segments (the watermark tokens are in the segments highlighted with thicker lines). This gives us 37 segments containing exactly one watermark token, 8 segments containing two watermark tokens, and 8 segments with no watermark tokens.

[0049] Because we have divided the input space into those inputs we want to map to regular tokens and those we want to map to watermark tokens, we can define a segment size that gives us as many segments as there are watermark inputs. We then define that each segment contains exactly one specified token, which we will assign to a watermark input. We want to specify the true watermark token (i.e., the token that hashes to a watermark bin), but we do not know in advance which token in the segment this will be (we do not even know if the segment actually contains a watermark token). When we need to choose a specified token, we search the segment to find the first token in the segment that is a true watermark token (if there is none, we fall back to returning the last token in the segment).

[0050] When we need to select a specified token, we search for the segment using the following process. 1. Determine the starting point within the segment - we want this to be deterministic (so that we get the same answer for the segment every time), but different for each segment. To do this, we seed a pseudorandom number generator (PRNG) with the segment index and use it to pick one of the indices within the segment. 2. Examine the token at this starting index to see if it is a watermark token (i.e., if it hashes to a value that falls within a watermark bin). If so, this becomes the specified token for the segment. 3. If the token is not a watermark token, proceed to test the next token. Continue this process (loop at the end of the segment) until we find a watermark token or we reach the starting point again. 4. If we reach the starting point again without finding the watermark token, we specify the last token we tested in the segment (i.e., the last token encountered after we traverse all tokens in the segment by cycling). This is why we choose a different starting point for each segment - if we started at the same point for each segment, we would return the same token for segmentation for each segment (i.e., the specified token would be regularly spaced in the token space).

[0051] FIG. 6 shows a diagram illustrating the process for two example segments, one containing an actual watermark token and one not.

[0052] Note that there are two ways in which this system can lead to an imperfect watermarked input → watermarked token mapping, and to an unwatermarked input → unwatermarked token mapping. 1. If a segment contains multiple actual watermark tokens (tokens that hash to watermark bins), only one of them will be assigned to the watermark input, and the other tokens will be returned to the non-watermark input. This results in noise being added to the watermark. 2. If a segment does not contain a watermark token, the watermark input will be assigned a token that is not actually a watermark token. Note that this is harmless - after all, watermark inputs are likely to be encountered rarely (by definition), and returning a non-watermark token does not introduce any noise.

[0053] As can be seen, the first result weakens the embedded watermark and is more likely to occur when we declare too few watermark inputs. The number of watermark inputs we declare determines the number of specified tokens we consider, which is independent of the number of watermark tokens actually present. The number of watermark tokens depends on the number of hash bins and the token space size. It is clear that this scenario occurs when the number of watermark inputs is less than the number of watermark tokens.

[0054] The second outcome is harmless as long as its occurrence does not also imply the occurrence of the first outcome - for example, if the number of declared watermark inputs exactly matches the number of actual watermark tokens, the segmentation will give an incomplete result, and the fact that some segments produce outcome 2 means that other segments must produce outcome 1.

[0055] An optimal strategy is achieved when the number of watermark inputs exceeds the number of watermark tokens by a sufficient margin so that segments are small enough that the probability of a segment containing multiple watermark tokens is small. However, this should not be achieved by artificially inflating the number of watermark inputs, as this can lead to assigning watermark tokens to inputs that are actually encountered more rarely than other inputs (thus introducing noise into the watermark through frequent release of watermark tokens).

[0056] Security If an attacker were able to determine that a particular token was a watermark token, they would know that the corresponding input was drawn from within the space of infrequently encountered inputs—an unacceptable information leak. However, because the algorithm uses a secret watermarking key when embedding the watermark (using a cryptographic hash function), this kind of inference is not possible without access to this key, and the attacker cannot learn much more from the token when compared to vault-based solutions.

[0057] An example will now be described to illustrate the complete algorithm.

[0058] Example This section simultaneously performs the steps of the complete algorithm for two scenarios: input that fits within the watermark input region and input that does not. We use the same example scenario discussed above. Figure 7 shows a diagram illustrating the input space and token space. The left pane shows the input space 71, with a declared watermark input 72. The right pane shows the token space 73, with the distribution of watermark tokens also shown in a lighter grayscale color 74 (note, however, that this is unknown to the algorithm). Our example will be tokenizing the highlighted non-watermarked input 75 (the input with ordinal 123) and the highlighted watermarked input 76 (the input with ordinal 356).

[0059] Step 1: Determine the index of the value in the input space For each input, we first determine which subspace it is contained in, and then determine its index within that space. Figure 8 shows a diagram illustrating the steps for determining the index of a value in the input space.

[0060] In our example, input 75 (the input with ordinal number 123) falls within the non-watermarked input subspace with index 117, and input 76 (the input with ordinal number 356) falls within the watermarked input subspace with index 9.

[0061] Step 2: Encrypt the subspace index Here, we encrypt the input within that subspace - we take an index within this space and use a format-preserving encryption method to obtain another index within the same space. Figure 9 shows a diagram illustrating the encryption of a subspace index.

[0062] In our scenario, input 75 has index 117 in a subspace of size 347 - in this example, this is encrypted to index 226. Input 76 has index 9 in a subspace of size 53, which is encrypted to index 23.

[0063] Step 3: Find the token space segments to which these inputs are mapped Now we need to find the token ordinals to which these subspace indices map. By definition, we have one specified token per segment - since there are 53 watermark inputs in total, this means we define 53 segments. We try to balance the size of these segments as much as possible, so we create 29 segments of size 8 and 24 segments of size 7.

[0064] FIG. 10 shows a diagram illustrating how to find the token space segment to which an input maps.

[0065] Since there is one designated token per segment, the designated token at index 23 is clearly in the 23rd segment. To find the non-watermark token with index 226 (the 226th non-watermark token in the space), we need to skip segments until we pass 225 other non-watermark tokens. The first 29 segments each contain eight tokens, one of which is the designated token, so if we skip all of these, we have passed 203 non-watermark tokens. The remaining segments contain seven tokens (six of which are non-watermark tokens), so we need to skip three more of these for a total of 221 non-watermark tokens. Therefore, we can say that the non-watermark token with index 226 is the fifth non-watermark token in the 33rd segment.

[0066] Step 4: Search the segment to find the desired token Having found the correct segment in each case, we then need to find the associated token within it.

[0067] FIG. 11 shows a diagram illustrating how to search a segment to find a desired token. For input 111, we are searching for the non-watermark token with index 226, which we calculate to be the fifth non-watermark token in the 33rd segment. Our random but deterministic starting point for this segment is token 6, which we test and confirm is not a watermark token. We then test the next token and discover that it is a watermark token, and that the specified token is also a watermark token. Therefore, we know that to find the fifth unspecified token, we must scan four more tokens past this token. Wrapping around at the end of the segment, this tells us that the token we need is the third token in the segment.

[0068] For input 112, we need to find the specified token within segment 23. Our starting point for this segment is index 3, and we must forward test each token until we find that the seventh token in the segment (the fifth token we test) is the watermark token and therefore the specified token.

[0069] Step 5: Return the final token ordinal Now, all that remains is to map the segment tokens we have found to their ordinals within the entire token space, as shown in Figure 12, which illustrates the output including the final token ordinal.

[0070] Here finally we see that input 111 with ordinal number 123 is tokenized into a token with ordinal number 256, and input 112 with ordinal number 356 is tokenized into a token with ordinal number 184.

[0071] Although the watermarking scheme has been described when combined with vault-less tokenization, it could also be extended to be combined with vault-based tokenization. With a vault scheme, watermarking is typically an easy problem to solve because the system can be configured to not output watermarked tokens when certain inputs are encountered. However, vault schemes stop working when more inputs are seen than there are non-watermarked tokens, because in that case, watermarked tokens must be returned. However, the described process also provides a solution that would avoid this problem when combining watermarking with vault-based tokenization.

[0072] Further details regarding the algorithm are now provided.

[0073] Algorithm Building Blocks Hash-based watermarking To allow the watermark to be extracted solely from the token (how it was generated is unknown), a pattern is embedded using a hash of the token. Note that while this document discusses the process in terms of "token," which is understood to be the output of a consistent tokenization operation, the same watermarking methodology applies to any process that produces output that contains some pseudo-randomness. For example, the output of a numeric blur could be hashed and subjected to the same process.

[0074] The hash space will be divided into equal-width bins, and each bin can be assigned to a different data release (thus the number of data releases that can be watermarked is equal to the number of bins). The diagram shown in Figure 13 shows the unsigned 32-bit output and the 128 (2 7) We illustrate this for a hash function that gives data release watermark bins (0-127): (Note that because the number of bins is equal to a power of 2, the hash space is exactly divisible across bins, and there is no difference in the number of hashes per bin.)

[0075] To embed a watermark in a data release, we reject any token that hashes to a value that falls within the watermark bin of that data release. Thus, the fraction of rejected tokens in this scheme is 1 / N, where N is the number of watermark bins in the data release.

[0076] The process of extracting a watermark is illustrated in Figure 14. We hash the tokens we encounter and increment a value count for the bin that the hash falls within (in the figure, a bin is black when it has at least one hash in it). We are left with a single empty bin 141, and the bin index gives us the watermark 141.

[0077] A drawback of this algorithm is that as the number of distinct data releases (bins) increases, the number of records required to extract the watermark also increases: to find the watermark, we require that all of the other N-1 bins contain at least one value, which scales poorly with the number of data releases.

[0078] Extending to Multiple Hashes: Hash Arrays Taking inspiration from Bloom filters, we can use multiple hash functions and coordinate them in a hash array structure. To embed a watermark, we reject any token that falls within a watermark bin for any of the hash functions (see below for an alternative way of using this configuration that was tested but ultimately rejected). Then, when extracting the watermark, we find a single bin index that is empty for each of the hash function bins in the array—we find an empty bin for each hash function and obtain the intersection of these sets, as shown in Figure 15. When this intersection has only a single empty bin 151, we have found the watermark, but at this point each of the histograms in the array still has two or more empty bins, as illustrated in the figure (in this figure, the bins filled with a diagonal pattern are empty for the current hash function but contain hashes—black bins—for at least one of the other hash functions).

[0079] Note that this gives us a higher token rejection rate (when compared to the same number of bins and a single hash function) since now there are multiple opportunities for a token to be rejected.

[0080] Dynamic Scaling of Data Release: Multihash Arrays The hash array structure allows us to adjust the number of bins and the number of hash functions to balance the number of supported data releases and the token rejection rate. However, the configuration must be determined in advance, which forces us to decide the number of data releases we want to support in advance and create a finite pool of watermarks. However, we can dynamically scale the number of watermarked data releases by creating new instances of the hash array structure (each containing its own hash function key) and assigning a different portion of the data release to each. We call this structure a Multi Hash Array, and it has the following properties: • First, multi-hash arrays contain only a single hash array instance and therefore support watermarking of up to N data releases (where N is the number of hash array bins). Once N watermarks have been assigned, a second hash array instance is created to handle the watermarking of the next N data releases. This new instance is given a different hash function key than the first instance, so any patterns embedded by either array will appear only as random noise to the other instances. Embedding a watermark only requires a single hash array instance to be used (the instance containing the data release in which the watermark is embedded), so there is no increase in computational complexity as additional data releases are watermarked. ● The token rejection rate does not increase with the number of supported data releases, since it depends only on the number of bins and hash functions in a single hash array. Additional hash array instances can be added indefinitely, with each new instance resulting in only a small increase in the number of rows required to extract the watermark (see analysis below).

[0081] When extracting a watermark, we perform a set of parallel hash array extractions—one for each requested instance. As mentioned above, because each instance has its own hash function key, the watermark pattern will only appear in a single instance; other instances will simply observe a uniform distribution of hashes (appearing as random noise). This is illustrated in Figure 16, which shows an extraction with two such instances, one of which finds the watermark bin 161, while the other does not see the watermark.

[0082] The partitioning of data release that this structure provides may also provide additional functionality benefits. We have already seen that embedding a data release watermark only requires that a hash array instance of the data release be used, but the same is true for functions that "sniff" a stream of data to see if a given data release watermark is present. Extracting unknown watermarks only requires the use of the full hash array instance. However, this structure also makes it possible to narrow the scope of watermark detection (and therefore the number of rows required) if it is known that the leaked data must be from a subset of the data release (perhaps because the dataset in question was only published in part of the data release). When older data releases have expired and transitioned out of consideration, hash array instances containing only those data releases can be excluded from extraction.

[0083] Each set of individual hash array keys is generated using a scheme such as HKDF (a simple key derivation function KDF based on the HMAC message authentication code), which allows for the expansion of a single master key into many different derived keys (and the ability to efficiently derive a specific key by providing the "ID" of the key in the input keying material). However, it is also possible to have more fine-grained control of the keys (different master keys per tranche of hash arrays, or per individual hash array), which can also have two security advantages. Permissions: The ability to have finer control over which watermarks can be read and where. Watermark sniffing functions can also be used to verify that watermarks are present and prevent unwatermarked data from passing through; it is an attractive feature that each of these remote execution points has the ability to retrieve only the keys of instances that contain the watermarks it expects, and not simply read arbitrary watermarks. Key rolling: Where each new hash array has its own key, any single hash array key is used as long as the data release within it is open and active (although older key versions must be kept as long as we hope to be able to extract watermarks generated using them).

[0084] Number of supported watermarks and token rejection rate The number of unique watermarks that can be embedded and the percentage of rejected tokens (token rejection rate) depend on the following configuration parameters of the algorithm: b - the number of data release bins in each hash array instance h - the number of hash functions to use in each hash array instance m - the number of hash array instances currently including the multihash array FIG. 17 shows a diagram illustrating the parameters of the algorithm. Since one bin is used for each data release, it is clear that the number of supported data releases is given by: N=m×b To embed a watermark in a data release, we reject any token that hashes to a value that falls within the watermark bin of that data release for any of the hash functions. Thus, the token rejection rate is given by

number

[0085] Extracting Watermarks Dealing with Noise: Extraction Algorithm Requirements (The dataset we are attempting to extract the watermark from may not be a pure collection of tokens without the watermark token; it may have been modified by the addition of new synthetic rows, it may be a combination of outputs from several data releases, or assumptions made about the data shape when assigning the watermark input may not have been entirely correct.) Because the watermark is embedded using a secret key, it is not possible to create noise that is excessive in any particular bin without access to this key (directly or indirectly via the watermark extraction function), which we assume is unavailable to anyone attempting to erase the watermark.

[0086] Thus, the addition of a synthetic row will appear as a baseline level of noise above the pure watermark, and the combination of multiple data-releasing watermarks will appear as several bins, each with some percentage below the baseline level. Figure 18 shows three histograms of token counts within each bin for a "pure" watermark (18A), a noisy watermark (18B), and the two mixed watermarks.

[0087] Therefore, we have the following requirements for our extraction algorithm: It must be possible to extract the watermark even if the hash bin is not exactly empty. Must be able to handle multiple such bins (indicating that the data is a mixture of watermarks from different data releases) - We must be able to show the end user some measure of how confident we are that the data contains a particular watermark. Any approach that attempts to find empty bins must relax the definition of "empty" to deal with noise, and this can only be done by defining some threshold percentage / count below which a bin is empty and above which it is not, introducing an element of undesirable arbitrariness into the algorithm.

[0088] To avoid this, the extraction method does not attempt to determine "Which data release bins are empty?" Instead, it reformulates the question to ask, for a given data release, "Are we sufficiently certain that the data contains the watermark for this data release bin?", which it answers by calculating how likely it is that we would observe the current hash number in the data release bin if the watermark were not present and we were observing only noise in the bin.

[0089] By asking this question for every data release bin, we can return a data release (or multiple data releases) that we are confident was the source of the watermarked data (because the hash numbers observed in those bins are unlikely to be due to random noise).

[0090] Overview of the extraction algorithm The watermark extraction algorithm is a simple hypothesis test at every bin that determines whether there is sufficient evidence to reject the null hypothesis (that the data does not contain the watermark for the current bin) and accept an alternative hypothesis. This is done by calculating the probability of obtaining the number of hashes observed in the current bin or less if the data did not contain the watermark for that bin data release. If this probability is less than the significance level implied by the confidence level provided by the user, we reject the null hypothesis that there is no watermark corresponding to the data release bin and instead declare the presence of such a watermark.

[0091] With multiple hash functions, we have multiple instances of the hash array structure. When considering a watermark bin, we sum the hashes for the bin (and the total number of hashes) across all of the hash functions.

[0092] Data release bin validation If the null hypothesis were true, we would expect hashes to fall into that bin just as often as other bins. However, if there was a watermark for a bin, we would expect to observe a much lower proportion of hashes in that bin compared to other bins.

[0093] The p-value is defined as the probability of obtaining a result at least as good as the observed result when the null hypothesis is true. In our case, this is the probability of obtaining the number of hashes observed in a bin (or less) when the data does not contain the watermark corresponding to the bin.

[0094] To calculate the p-value, we model the hashing of tokens into different bins as a binomial distribution, where in the absence of a watermark, the probability of a token hashing into a given bin is 1 / b.

[0095] Note that when the data have watermarks that correspond to different bins, this probability will be greater than 1 / b. However, in those cases, the actual p-value will always be less than the p-value calculated using 1 / b, so we can always safely reject the null hypothesis when the calculated p-value is less than alpha.

[0096] Therefore, the probability of seeing k hashes in a bin after observing n hashes overall is given by:

number

number

[0097] Confidence level The extraction algorithm receives a confidence level as user input, which is interpreted as 1-α, where α is a bound on the statistical significance of the test, i.e., the probability that we will falsely declare the presence of a watermark in data where no such watermark is present (the significance level therefore provides a bound on the false discovery rate).

[0098] If the computed p-value for a bin is less than the significance level α, we reject the null hypothesis and declare the presence of a watermark for the bin. Otherwise, we fail to reject the null hypothesis and declare that a watermark for the bin is not found in the dataset. Note: This confidence level should not be interpreted as the probability that the watermark actually exists when we reject the null hypothesis.

[0099] Handling comparisons of multiple bins When extracting unknown watermarks from data, we require testing all data release bins, and the decisions from all of these tests must be correct for the process to be successful. Simultaneously testing multiple bins for the presence of a watermark increases the false positive rate of the overall test beyond that of a single test on one bin. To address this issue, we use the Holm-Bonferroni method, which ensures that the overall error of the series of tests remains below the required error bound (while ensuring higher statistical power than the standard Bonferroni correction, which in our context means improved ability to detect the presence of multiple watermarks simultaneously mixed into the dataset). This is done by reducing the value of α used for each test by a factor of the number of tests performed, which is the number of data release bins to test.

[0100] The process is to first sort the bins by p-value (ascending order), then use a different significance level (α) for each bin. The bin with the lowest p-value is tested first, with a significance level of α / b. If the p-value for this data release is below the required significance level, then the bin with the next lowest p-value is tested, this time using a significance level of α / (b-1). This process continues until we encounter a bin whose p-value exceeds its corresponding significance level. Thus, the value of α for extracting data release watermark combinations is: Possible Watermark 1:

number

number

number

[0101] How many tokens are needed to detect the watermark? No additional noise When there is an additional noise-free watermark, we have empty bins, so the p-value is just the probability of all n tokens hashing to a bin other than the current bin.

number

number

number

[0102] Extension to multiple hash array instances and multiple hashes per instance The only difference from the above derivation when there are multiple hash array instances is that there are now m × b possible watermarks instead of just b. Repeating the above steps with a new number of tests gives the number of tokens required as follows:

number

number

[0103] When noise is present When calculating the number of tokens needed to extract the watermark, we need a bound on the joint probability: A. Check for more than a certain number of hashes in a bin if a matching watermark exists. B. If no matching watermark exists, check for hashes less than that number.

[0104] In the noiseless case, we know that the probability of A is zero (in the absence of noise, there will be no hashes in the watermark bin), and therefore we only needed to calculate the probability of seeing a zero hash in a bin when no corresponding watermark is present. However, when noise is present, we need to consider both of these probabilities. In the new definitions below, we use the same notation as before. H0 - Null hypothesis that watermarks do not exist H1 - Alternative hypothesis that watermarks exist ●p i - expected proportion of hashes in bin i ●n i - the number of hashes in bin i f - expected proportion of noise

[0105] The expected fraction of hashes in a watermark bin when the noise fraction is f is given by:

number

[0106] We know that the joint probability of two events is always less than or equal to the sum of the individual event probabilities. Therefore, finding the number of tokens needed to read back the watermark at a given noise level is equivalent to solving for n such that there exists a k that satisfies the following inequality: Pr(n i ≦k|H0)+Pr(ni ≧k|H1)≦α During the ceremony

number

[0107] Obtaining an explicit formula for n that satisfies the above equation may be difficult, so the estimation function may instead perform a brute-force search over n and k to find the number of tokens that satisfy the above inequality.

[0108] Rows vs. Unique Tokens The above modeling computes the number of tokens needed to extract the watermark, but this is actually the number of unique tokens, as there is an implicit assumption that each token hashed and added to an array bin gives us a new piece of information about the token distribution (and therefore the watermark pattern embedded within it). We can only embed a watermark if we encounter enough diversity in the inputs to allow us to return a range of tokens that touch all bins in the hash space - in the pathological case where we only encounter a single input, we will only return tokens that end up in a single bin.

[0109] Therefore, it is important to attempt to add each encountered token to the extraction hash bin exactly once. This requires the extraction process to keep track of previously encountered tokens, but this is easy to do using a Bloom filter. Because we know the number of unique tokens we need to add before we expect to be able to extract the watermark, we can size the filter appropriately - a limit of 100,000 values ​​(far more than the number of unique tokens we expect to need) and 10 -9Choosing a false positive rate of gives a filter that requires only about 525kb of memory (note that the extraction process only requires a single instance of this filter, regardless of the number of data releases or algorithm configuration parameters).

[0110] Reporting the strength of the embedded watermark The strength of the embedded watermark can be reported to the user as part of the tokenization job. This strength can be interpreted as the maximum confidence level that extraction can be performed and still accurately obtain the watermarked data release (assuming the output file is not modified in any way), providing an easily understandable summary of whether the processed data contains enough unique tokens to carry the watermark.

[0111] Choosing algorithm parameters Multi-hash array: dynamically increasing m As shown above, the token rejection rate is independent of m, and the average number of tokens required to extract a watermark grows logarithmically with m. Therefore, it makes sense to treat m as a dynamic parameter—just start with a single hash array instance and add additional data release watermarks as they are needed. In this way, the token rejection rate remains constant, and the number of tokens required to extract a watermark is always (approximately) the minimum possible for the number of data releases needed, and only increases as new data release watermarks are released.

[0112] Hash array: choose b and h As we have seen, the choice of b and h affects all aspects of the algorithm—the number of supported data releases, the token rejection rate, and the number of tokens required to extract the watermark.

number

number

[0113] This relationship, and substituting the equation for the number of data releases into the equation for the number of tokens, gives the basic relationship between the quantities.

number

[0114] For a given extraction confidence level, this relationship tells us that the number of tokens required to extract a watermark is a function only of the token rejection rate and the number of supported data releases, and is independent of the configuration of the multi-hash array. The number of tokens required is reduced by rejecting more tokens, and since the acceptable token rejection rate depends on the usage scenario, this implies that we may need multiple different configurations, one for each usage scenario. For our purposes, we consider two common usage scenarios: tokenization of a bulk dataset, and tokenization of a much smaller set of results from an interactive database query.

[0115] Bulk Datasets In the traditional use case of tokenizing bulk datasets, the acceptable token rejection rate is capped at 1%. Therefore, the optimal configuration is reached by choosing the following parameters: ●Bring the token rejection rate as close to the upper limit as possible Minimize the number of data releases, as supporting more data releases comes at the cost of more tokens needed to extract the watermark. As discussed in the section above, we can always dynamically increase m to support more data releases, but we cannot support fewer data releases than b - so rather than lowering performance initially, it makes sense to minimize b so that performance is optimal initially and gradually degrades as more data releases are required.

[0116] Therefore, the proposed configuration for this scenario is as follows: b=1024 ●h=10

[0117] This gives a token rejection rate of just over 0.97%. Figure 19 shows a diagram illustrating the extraction token count requirements as the number of data releases increases.

[0118] FIG. 20 shows a plot of the tokens required to extract the watermark with 99.9% confidence for the first hash array instance (supporting up to 1024 data releases) as a function of the input noise percentage.

[0119] Small datasets For use cases where we obtain a small sample of a dataset, either as a preview of the dataset or as the result of a selective SQL query, we can tolerate a much higher token rejection rate (because we will have a much smaller number of required tokens), but we will need to be able to embed an extractable watermark into a much smaller dataset. Thus, the proposed configuration for this scenario is as follows: ●b=256 ●h=50

[0120] This gives a token rejection rate of approximately 17.8%. Figure 21 shows a diagram illustrating the extraction token count requirements as the number of data releases increases.

[0121] FIG. 22 shows a plot of the tokens required to extract the watermark with 99.9% reliability for the first hash array instance (supporting up to 256 data releases) as a function of the input noise percentage.

[0122] Scalability calculation time The largest component of the algorithm's computational cost is computing the cryptographic hashes. Therefore, one might expect that as h increases, the computational cost of the algorithm would increase significantly. However, thanks to Kirsch-Mitzenmacher optimization, only a single 64-bit hash needs to be computed, from which multiple 32-bit hashes can then be derived cheaply via multiplication and modular operations without compromising any randomness. Increasing the number of hashes increases the amount of computation required, but in a reasonably modest manner (benchmarks show that the computational cost of 50 hashes when embedding a watermark is about 1.7 times the computational cost of 1 hash, not 50 times).

[0123] This optimization may not apply as we increase m because each hash array instance has its own base key. Therefore, the computation time scales linearly with m. However, this is only a concern when extracting the watermark, which requires us to compute hashes across different hash array instances. When embedding a watermark (which is the only operation on the tokenization critical path), we only need to verify a single hash array instance (the instance containing the data release in which we embed the watermark), so the computation time of embedding is independent of m.

[0124] Figures 23 and 24 show the results of benchmark runs that confirm these claims. Figure 23 plots the normalized computation time for watermark embedding, and Figure 24 plots the normalized computation time for watermark extraction. Note that in each graph, the values ​​on the vertical axis are normalized within that particular test range, since the absolute values ​​will vary depending on the environment and all we are interested in is the trend.

[0125] memory Extracting watermarks requires that bins be held in memory while the data is traversed, incrementing the bin count (we also use a single Bloom filter, which requires a small amount of memory regardless of the values ​​of h, b, and m). Increasing h or m results in more copies of the bin array in memory, and increasing b results in a larger bin array in each copy. The values ​​of b and h are fixed in our scenario, too small to use much memory per hash array instance. However, the memory required to extract watermarks increases as the number of embedded watermarks increases (i.e., as m increases). If m reaches a large enough number that memory usage becomes an issue, it may be necessary to make multiple passes through the data and partition the values ​​of m across them.

[0126] Embedding the watermark is not affected as no state needs to be stored in memory.

[0127] result summary This section presents the results of experiments performed on the proposed scheme to verify various aspects of its behavior. In summary, these results show us that: Embedding an extractable watermark requires an expected number of tokens. The false positive match rate returned by extraction is bounded by the confidence level, so it can be adjusted to be arbitrarily low. Extraction tolerates random noise at the cost of requiring more rows when it is noisy, and row requirements rise as modeled. • The blending of multiple data release watermarks results in all of the individual watermarks being extracted, even in the presence of additional random noise.

[0128] Note: All results presented in this section were obtained using a similar bulk dataset configuration (b = 1024 and h = 10). However, the observations also apply to datasets with any other configuration, with only the absolute numbers scaled in the manner predicted in the above discussion.

[0129] Extraction result Figure 25 shows the final results of watermark extraction performed at a 95% confidence level as the number of tokens processed in the extraction increases. Each data point is the average of 10,000 experiments and shows the division of results across three mutually exclusive possibilities: no results returned, only correct data releases returned, and correct and incorrect data releases returned (note that there is a theoretical fourth outcome in which only incorrect data releases are returned, but this did not occur).

[0130] Here we clearly see the effect of the confidence level parameter, which is to limit the rate at which false positives are returned (when we reach a token count that our modeling indicates we will need to extract the watermark with 95% confidence—10 tokens—we always get the correct data release, but it is achieved by false positive results at a rate that never exceeds the alpha level implied by our confidence).

[0131] The graph above demonstrates the use of a 95% confidence level so that the effect of this parameter can be easily visualized. However, in watermark extraction, false positives are an undesirable event—in some circumstances, they could result in innocent recipients being blamed for data leakage—so a 5% false positive rate is too high for practical use. However, confidence levels provide us with an easily understandable mechanism for reducing the frequency of false positives to a desired level—if a user specifies a 99.9% confidence level, we can ensure that false positives will be returned in 0.1% or fewer cases. And because the score for true-positive data releases increases very rapidly with the number of tokens, this additional accuracy comes at only a small cost in the number of tokens required to extract the actual watermark. This is shown in Figure 26, which repeats the above experiment at a 99.9% confidence level.

[0132] Noise Tolerance We repeated the above experiment, but gradually added increasing levels of random noise. The graph in Figure 27 shows how the percentage of times we obtain only the correct watermarked data release varies with the noise level and the number of tokens on which extraction is performed (again, at a 95% confidence level to allow the effect to be clearly seen). The graph shows, as expected, that the number of tokens required to obtain the correct result at a given confidence level increases.

[0133] Figure 28 shows the false positive incidence for the same experiment (where a false positive is recorded every time at least one incorrect data release is returned, regardless of whether a correct data release is also returned).

[0134] Here we can clearly see that the false positive rate is bounded by the confidence level provided (and that it does not depend on the level of noise).

[0135] Number of tokens required to achieve authenticity The above discussion of the watermark extraction algorithm presents a method for estimating the number of tokens needed to extract a watermark for a given confidence level and amount of noise added to the input dataset. To test the accuracy of this, we calculated the number of tokens needed to extract the watermark with 95% confidence for various noise levels, and then ran an experiment for each of these, extracting the watermark 10,000 times on a dataset of this size and recording the results. The results of this experiment are shown in Figure 29 (areas representing the percentage of times only correct data releases were returned, and areas representing the percentage of times at least one incorrect data release was returned, regardless of whether a correct data release was also returned; the lines using the right-hand y-axis indicate the number of rows on which extraction was performed for that noise level).

[0136] From this graph, we can confirm that our estimates for the number of tokens required are accurate - we can see that the extraction success rate tracks well with the expected confidence level (and, as usual, the false positive rate never exceeds the expected 5%).

[0137] Mixed Data Release Watermark Figures 30 and 31 show the results of experiments where the input to the extraction function was a dataset containing multiple data release watermarks mixed together, with the addition of increasingly higher levels of random noise. Figure 30 shows an experiment where two data release watermarks were mixed together, and Figure 31 shows a mix of three data release watermarks. In all instances, the watermarks were shared evenly across the tokens remaining after the random noise was added (e.g., the results for 40% noise include 40% noise / 30% data release 1 / 30% data release 2 in Figure 30, and 40% noise / 20% data release 1 / 20% data release 2 / 20% data release 3 in Figure 31). All experiments were performed at a 95% confidence level.

[0138] Here, we can confirm that multiple watermarks are correctly separated even with the addition of random noise. As expected, more tokens are required to extract more watermarks (from the perspective of one of the watermark bins, data carrying other watermarks appears simply as random noise, slowing down the extraction of that watermark in the manner shown in the section above).

[0139] The false positive rate is not shown in the graph above, but is bounded at 5%, as expected.

[0140] Alternative Data Release Representations The proposed scheme uses the same bins for data releases in each hash array. However, an alternative scheme uses different bins for each hash function for a data release and represents the data release as a set of hash function+bin pairs (one for each hash function)—the bins of any given hash function will be used for multiple data releases, but the combination of bins across hash functions will be unique to that data release. As shown in Figure 32, this configuration gives us more data releases than the traditional case, but directly translates to more tokens being required to extract the watermark, since the hash functions no longer work together (whereas in our original configuration, a token hash appearing in any bin of a hash function was sufficient to regularize that bin not being considered in all hash functions, this is no longer the case, and each hash function works independently). However, this still requires fewer tokens than a single hash function structure—each additional token eliminates a bin in each of the hash functions, and the sum of all of the values ​​across all of the hash functions for a data release's bins allows us to reach a higher reliability score with fewer tokens.

[0141] According to this configuration, the number of supported watermarks is given by: N=m×b h

[0142] However, this exponential growth in data releases was ultimately the reason this configuration was rejected. As we have seen, there is a fundamental relationship between the number of tokens required to extract the watermark and the number of data releases that can be supported for a given token rejection rate - so, somewhat paradoxically, it is actually advantageous to have a scheme in which the number of data releases grows slower, so that we can keep the token rejection rate as close as possible to the acceptable 1% bound, and therefore as close as possible to the minimum number of tokens required.

[0143] Appendix A - Watermarking Deterministic Tokenization This appendix summarizes the main features A to D. Each listed feature can be combined with any other feature A to D. Each optional feature defined below can be combined with any other optional feature and with any other optional feature.

[0144] Key Feature A: A process that incorporates digital watermarking in addition to deterministic tokenization. 1. A computer-implemented process for embedding a digital watermark in tokenized data, comprising: (a) generating tokens from an input data set, the tokens being generated using a deterministic encryption scheme; (b) embedding a digital watermark within the generated set of tokens.

[0145] Key Feature B: A process that incorporates digital watermarking on top of deterministic tokenization, where the digital watermark is embedded probabilistically through the selection of tokens within the generated token set. 1. A computer-implemented process for embedding a digital watermark in tokenized data, comprising: (a) generating tokens from an input data set, the tokens being generated using a deterministic encryption scheme; (b) embedding a digital watermark within the generated set of tokens; A process in which a digital watermark comprises a pattern that is probabilistically embedded through the selection of tokens within a generated token set.

[0146] Key Feature C: A process that incorporates digital watermarking on top of deterministic tokenization, where the digital watermark can be reconstructed without prior knowledge of the encryption scheme or any other processing used on the input dataset. 1. A computer-implemented process for embedding a digital watermark in tokenized data, comprising: (a) generating tokens from an input data set, the tokens being generated using a deterministic encryption scheme; (b) embedding a digital watermark within the generated set of tokens; A process by which a digital watermark can be reconstructed without prior knowledge of the encryption scheme or any other processing used on the input data set.

[0147] Key Feature D: A process that incorporates digital watermarking in addition to deterministic tokenization, where the number of potential watermarked data releases can be dynamically scaled. 1. A computer-implemented process for embedding a digital watermark in tokenized data, comprising: (a) generating tokens from an input data set, the tokens being generated using a deterministic encryption scheme; (b) embedding a digital watermark within the generated token set; (a) generating a data release, wherein a digital watermark is chosen or selected based on parameters of the data release; A process that allows the number of possible watermarked data releases to be dynamically scaled.

[0148] Optional Features The process of embedding a digital watermark - The digital watermark is embedded within the generated token set and not embedded in any metadata or redundant data. The deterministic encryption scheme is a pseudo-random permutation scheme, such as a pseudo-random permutation scheme based on a format-preserving encryption cipher. ●The process is (a) scanning or observing an input space corresponding to an input data set; (b) determining or identifying inputs to be mapped to watermark tokens; ● The inputs to be mapped to watermark tokens are inferred from knowledge of the input space and what it represents. The inputs to be mapped to watermark tokens are those that are unlikely to occur or be encountered (unlikely occurring refers to using the distribution of probabilities of encountering possible input values ​​so that tokens that we do not want returned are mapped to inputs that we would not expect to encounter). ● The process includes a step of dividing the input space into two distinct or disjoint subspaces, a "non-watermarked input subspace" and a "watermarked input subspace", where inputs in the non-watermarked input subspace are mapped to non-watermarked tokens and inputs in the watermarked input subspace are mapped to watermarked tokens. • Encryption of each input subspace is achieved independently. ● The algorithm includes two deterministic encryption schemes such as FPE (Format Preserving Encryption). An FPE is configured to encrypt a non-watermarked input subspace, and the other FPE is configured to encrypt a watermarked input subspace. • Two deterministic encryption schemes are each based on a secret key. • Digital watermark patterns are embedded within watermark tokens, which are assigned or determined such that they hash to values ​​that fall within a predetermined range of a hash space. • The tokens are hashed using a cryptographic hash function based on a secret watermarking key, so an attacker cannot learn in which subspace the token resides. The process includes a step of attempting to map the watermark token such that the watermark token is never or rarely returned to a bin in the hash space that contains no value or a value close to zero. ● Watermark tokens are dynamically assigned or determined at runtime to avoid the need for a brute-force search of the entire token space. ●Determining the watermark token is (i) segmenting the token space into segments; (j) assigning a watermark token to each segment. ● A watermark token is assigned by scanning a segment to find the first token in the segment that hashes to a value within a predetermined range. • The starting index for scanning each segment is chosen using a pseudo-random number generator (PRNG) seeded with the segment index. If the start index is reached without finding a watermark token, the final index of the segment is chosen as the watermark token. The size of the segments is chosen to give a high probability that each segment will contain no more than one token that hashes to the watermark range (assuming a uniform distribution from the hash function), avoiding the need to return these tokens for inputs other than the intended input.

[0149] Reconstruction or extraction of digital watermarks from data releases The digital watermark can be reconstructed without prior knowledge of the deterministic encryption scheme or any other scheme used on the input data set. ● The digital watermark is reconstructed by (a) hashing the tokens in the watermark data release using one or more hash functions, (b) constructing a histogram of hash frequencies in the hash space, and (c) determining the bins that correspond to the digital watermark. When multiple hash functions are used, the process includes summing the hash numbers for each bin across all of the hash functions. Each hash function contains a different key. The process includes calculating the probability of obtaining, for a particular bin, an observed hash number or less when the watermarked data release does not contain a digital watermark corresponding to the particular bin. When the calculated probability is lower than a predetermined threshold, the process infers that the data release contains a particular digital watermark. • The process allows reconstruction of the digital watermark for only a subset of the data release. • The process can handle noise, such as adding or removing rows from the data release. Extraction of the digital watermark is associated with a confidence score related to (a) the likelihood of the watermark's presence in the tokenized data, and (b) the likelihood that an underrepresented bin in the histogram of token hash counts corresponds to a hash bin. (How confident can we be that the watermark is actually present in the tokenized data and that an underrepresented bin in the histogram of token hash counts is not simply a coincidence?) The process includes a step of estimating the number of tokens required to reconstruct the digital watermark. ● The number of tokens is estimated for a given confidence level and the amount of noise present in the tokenized data.

[0150] Digital watermarking depends on the parameters of the data release A computer-implemented process includes generating a data release, and a digital watermark is chosen or selected based on parameters of the data release. • Digital watermarks are chosen or selected based on the recipient of the data release. • Digital watermarks are chosen or selected based on the intended use of the data release. • Digital watermarks are chosen or selected based on the date by which the data release must be removed. Each data release corresponding to the same tokenized input data set contains a different digital watermark. • A digital watermark represents a pattern that is a unique ID. • Digital watermarks are chosen or selected based on the type of tokenized input data (i.e., ID, Social Security number, high, salary, etc.). • A digital watermark is chosen or selected based on the deterministic encryption scheme used. The process includes dynamically scaling the number of possible watermarked data releases by updating the hash function. The process includes dynamically scaling the number of possible watermarked data releases by hashing the token space with a different hash function.

[0151] Computer device or system 1. A computing device or system adapted to embed a digital watermark in tokenized data, the device or system comprising: (c) generating a token from the input data set, the token being generated using a deterministic encryption scheme; (d) embedding a digital watermark within the generated set of tokens.

[0152] Appendix B - Handling variable length input in deterministic tokenization Background to deterministic tokenization In deterministic tokenization, a format-preserving cryptographic cipher is used to provide substitutions between all of the values ​​in a space and other values ​​in the same space. For example, a space may be declared as the entire set of valid credit card numbers, and the deterministic tokenization system would then accept a credit card number as input and return another credit card number as output to use as a token instead of the actual value. The space of values / tokens is defined using regular expressions.

[0153] How it works: A simple example An example is illustrated in the diagram shown in Figure 33, where the complete input and token space is rendered using small squares for each possible value, which are arranged within larger squares to fit the two-dimensional nature of the example formula (and for space reasons), but which could also be shown as a long list of values.

[0154] As an example, suppose all values ​​that need to be tokenized contain two lowercase alphabetic characters. The regular expression [a-z]{2} is used to define this space. All values ​​that match this expression can be assigned an ordinal number in the range [1..676]. Imagine that the input jj needs to be tokenized, then the following is the logic that a deterministic tokenization system would implement: 1. Find the ordinal of the input value in the regular expression. In our example, the input is jj, and for simplicity we will assume that the order is lexicographical (although this need not actually be the case), so the input ordinal is (9*26)+10=244 (input ordinal 33A is shown in Figure 33). 2. Encrypt this ordinal using a format-preserving encryption cipher. For the sake of this example, imagine this gives an output of 419. 3. Determine the value from the regular expression with ordinal 419. In our lexicographical order, this gives the first character of (419 / 26)+1=17 (hence, q) and the second character of 419 mod 26=3 (hence, c). 4. Thus, input jj is tokenized into token qc (token qc33B is shown in Figure 33).

[0155] Issues with using deterministic tokenization for variable-length values Deterministic tokenization systems are well suited to values ​​that have a fixed structure, such as credit card numbers, social security numbers, and zip codes. However, they do not work well with data that is more free-form and can have variable lengths, such as names, addresses, and email addresses. While it is possible to work on variable-length data because regular expressions can be written that accept some characters variable times, the results may not be satisfactory to the user.

[0156] Consider a very simple scenario: tokenizing surnames. The US Census website provides a list of the most common surnames from the 1990 census, which covers over 90% of the population, so we can base our discussion on that (in the real world, we would need to worry about outliers such as double surnames, but we will ignore them for now).

[0157] From this list, we can see that the surname lengths range from 2 to 13 characters, with the distribution shown in Figure 34.

[0158] To fully cover the input range, we can create the expression [AZ][az]{1,12}. This works, but what will the resulting tokens be?

[0159] Because tokenization uses a strong cryptographic cipher, the selection of tokens from within the token space is indistinguishable from random, and every possible token is as likely as every other token resulting in a value output. This means that the distribution of lengths for a random sample of tokens will have the same distribution as the lengths of all possible tokens. There are 676 possible two-character tokens, 17,576 possible three-character tokens, and so on, and by the time you get to a length of 13 characters, there are over 2 quintillion possible tokens. The distribution of tokens is shown in Figure 35 using a chart plotted on a linear scale axis.

[0160] Adding additional characters increases the number of possible tokens by 26 times for the previous token length, exponentially increasing the number of tokens of each length. In the real data, the most common surname length was six characters, but over 96% of the generated tokens had lengths of up to 13 characters, making the chance of obtaining a six-character token approximately 1 in 84 million.

[0161] Perhaps in the simplified example above, this isn't a serious problem; the tokens are intended as meaningless replacements for raw values, and we don't expect them to be usable for arbitrary analysis of name length. But what if we make this example more realistic, and try to extend the expression to cover double surnames? Every time we add a character to the expression, we also extend the length of most of the tokens we'll encounter. What if we want to handle data such as email addresses? To ensure we can cover all possible email addresses, we need to allow 64 characters for the local part and 255 characters for the domain part, making all of our tokens (effectively) 320 characters long, a significant explosion in data size that's unacceptable.

[0162] The state-of-the-art format-preserving masking systems avoid this problem by treating the length of each different input value as a unique cryptographic cipher and generating a token that matches the length of the input (so in our name example, if we receive a 6-character name, we tokenize it using the expression [AZ][az]{5} and generate a 6-character token). However, we believe this is an unacceptable privacy risk because values ​​with unique lengths provide no privacy at all. To illustrate this, we use a list of employees at a particular company given the specific date of June 4, 2021. Of the 158 employees on that date, 26 (a very significant 16%) are uniquely identifiable using only the length of their first name and last name, and a length-preserving scheme would provide no protection at all for these people.

[0163] Proposed Solution We want to achieve a system where the distribution of token lengths resembles the distribution of input lengths, but where token lengths are not directly linked to the length of that input. While there is an obvious tension between these two requirements, it's enough to sever the direct, deterministic link between value and token length and instead adopt a scheme where a six-character value is likely to have a six-character token, but there's still a non-negligible (and controllable) chance that this is not the case. If we then tokenize a dataset that has a single long value in it, then seeing a long token doesn't guarantee that this is a long input (there's an element of plausible deniability).

[0164] Our proposed solution treats each length as a separate, format-preserving encryption, but works by assigning ranges of ciphertext to tokens of different lengths. It consists of a minimum and maximum length, and for each input length, it generates a distribution (e.g., a normal distribution) centered around that length with some standard deviation (this can be surfaced to the user as some "spread" parameter), and uses this to derive a weighting for how many of the tokens of that input length are assigned to tokens of that length. We use [az]{2,13} as an example. Figure 36 shows a graph of the weights controlling how many tokens of each length are generated for the first input value, length 2, using a standard deviation of 1 / 3 of the range.

[0165] These weights are then normalized and each is multiplied by the total number of tokens that need to be assigned (in this case, we are calculating assignments for two-character strings, so there are 676 of these) to calculate the number assigned to each length token. A graph of the length tokens generated as a function of the number of tokens of that assigned length for a two-character input is shown in Figure 37.

[0166] Thus, of the 676 possible two-character input strings, we assign 122 to two-character tokens, 119 to three-character tokens, etc., so that only three of the inputs map to 13-character tokens. When we get a two-character input, we do a format-preserving encryption to map its ordinal in the input space of 1..676 to a ciphertext ordinal in the same space. We then interpret this ciphertext ordinal differently depending on which region of the ciphertext space it falls into. This is illustrated in the diagram of Figure 38.

[0167] For example, if we get a two-character input that encodes to ordinal 360, which is the 1..676 ordinal in the space of all two-character inputs, we will return a five-character token for this input.

[0168] It then moves on to the next input length, 3 in the example, and repeats the process.

[0169] Once these weights and ranges are initialized, the cipher can be used for tokenization. In the example above, the two-character input is encrypted to ordinal 360, so it will return a five-character token for it. But which five-character token? We first start indexing into the space of five-character tokens using the chunk contributed by the two-character input, then the chunk for the three-character input, and so on. In this example, ordinal 360 is the first 10 ordinal numbers in the chunk for the two-character input, so we get ordinal 10. However, doing this means that the first N ordinal numbers of any token length will always be assigned to two-character inputs, the next M ordinal numbers to three-character inputs, and so on, leading to a bias in which lower-ordinal tokens are assigned to shorter inputs. To correct for this, after we have the ordinal numbers in the five-character space, we do a further FPE just within that space, re-randomizing all of the mappings to tokens of that length to ensure no systematic bias is introduced.

[0170] Watermarking Watermarking continues to work as described above in connection with the process for embedding a digital watermark within tokenized data. Watermark entries are selected from the longest input length (assuming this is one of the least likely to occur in real data), and these entries are then assigned watermark tokens across the length. Watermark tokens for each length are found as described in detail above, and each length is dynamically found using the segmentation and dynamic scanning techniques described above.

number

[0171] Therefore, the proposed solution may be combined with any one or more of the features or optional features from Appendix A where the generation of tokens is described above.

[0172] Appendix C - Handling variable length input - Summary of key features This appendix summarizes the main features E and F. Each optional feature defined below can be combined with any optional feature and with any other optional feature.

[0173] Key Features E - Generate tokens from input datasets of various lengths 1. A computer-implemented process for generating tokens from an input dataset comprising input values ​​of varying lengths, the process comprising generating tokens from the input dataset, the tokens having lengths that exhibit a distribution substantially similar to the distribution of lengths of the input dataset.

[0174] Optional Features: - The length of a token does not necessarily match the length of its corresponding input value. - Tokens are generated using an encryption scheme that treats each subset of the input data corresponding to a particular length of the input value as a separate or distinct format-preserving encryption. • Tokens are generated of various lengths for each subset of input data corresponding to a particular length of the input value. For each subset of input data corresponding to a particular length of the input values, tokens are generated with varying lengths, and the generated tokens have lengths that conform to some selected distribution generated for that particular length. Once a token is generated from a subset of the input data that corresponds to that particular length of the input value, additional format-preserving encryption is used on the generated token. - Tokens are generated based on one or more potentially user-configurable parameters, including minimum token length, maximum token length, or a distribution used to assign token length weightings.

[0175] Key Features F - Embed a digital watermark within tokens generated from input datasets of various lengths 1. A computer-implemented process for embedding a digital watermark in tokenized data, the process comprising: (a) generating, from an input dataset comprising input values ​​of varying lengths, tokens generated with lengths exhibiting a distribution substantially similar to the distribution of lengths of the input dataset; (b) embedding a digital watermark within the generated set of tokens.

[0176] Optional Features: Token Generation - The length of a token does not necessarily match the length of its corresponding input value. - Tokens are generated using an encryption scheme that treats each subset of the input data corresponding to a particular length of the input value as a separate or distinct format-preserving encryption. • Tokens are generated of various lengths for each subset of input data corresponding to a particular length of the input value. For each subset of input data corresponding to a particular length of the input values, tokens are generated with varying lengths, and the generated tokens have lengths that conform to some selected distribution generated for that particular length. Once a token is generated from a subset of the input data that corresponds to that particular length of the input value, additional format-preserving encryption is used on the generated token. - Tokens are generated based on one or more potentially user-configurable parameters, including minimum token length, maximum token length, or a distribution used to assign token length weightings.

[0177] The process of embedding a digital watermark • The digital watermark comprises a pattern that is probabilistically embedded through the selection of tokens within the generated token set. The digital watermark can be reconstructed without prior knowledge of the encryption scheme or any other processing used on the input data set. ● The process includes a step of generating a data release, and a digital watermark is chosen or selected based on parameters of the data release, allowing the number of possible watermarked data releases to be dynamically scaled. - The digital watermark is embedded within the generated token set and not embedded in any metadata or redundant data. ●The process is (a) scanning or observing an input space corresponding to an input data set; (b) determining or identifying inputs to be mapped to watermark tokens; ● The inputs to be mapped to watermark tokens are inferred from knowledge of the input space and what it represents. The inputs to be mapped to watermark tokens are those that are unlikely to occur or be encountered (unlikely occurring refers to using the distribution of probabilities of encountering possible input values ​​so that tokens that we do not want returned are mapped to inputs that we would not expect to encounter). ● The process includes a step of dividing the input space into two distinct or disjoint subspaces, a "non-watermarked input subspace" and a "watermarked input subspace", where inputs in the non-watermarked input subspace are mapped to non-watermarked tokens and inputs in the watermarked input subspace are mapped to watermarked tokens. • Encryption of each input subspace is achieved independently. • Digital watermark patterns are embedded within watermark tokens, which are assigned or determined such that they hash to values ​​that fall within a predetermined range of a hash space. • The tokens are hashed using a cryptographic hash function based on a secret watermarking key, so an attacker cannot learn in which subspace the token resides. The process includes a step of attempting to map the watermark token such that the watermark token is never or rarely returned to a bin in the hash space that contains no value or a value close to zero. ● Watermark tokens are dynamically assigned or determined at runtime to avoid the need for a brute-force search of the entire token space. ●Determining the watermark token is (k) segmenting the token space into segments; (l) assigning a watermark token to each segment. ● A watermark token is assigned by scanning a segment to find the first token in the segment that hashes to a value within a predetermined range. • The starting index for scanning each segment is chosen using a pseudo-random number generator (PRNG) seeded with the segment index. If the start index is reached without finding a watermark token, the final index of the segment is chosen as the watermark token. The size of the segments is chosen to give a high probability that each segment will contain no more than one token that hashes to the watermark range (assuming a uniform distribution from the hash function), avoiding the need to return these tokens for inputs other than the intended input.

[0178] Reconstruction or extraction of digital watermarks from data releases The digital watermark can be reconstructed without prior knowledge of the encryption scheme or any other scheme used on the input data set. ● The digital watermark is reconstructed by (a) hashing the tokens in the watermark data release using one or more hash functions, (b) constructing a histogram of hash frequencies in the hash space, and (c) determining the bins that correspond to the digital watermark. When multiple hash functions are used, the process includes summing the hash numbers for each bin across all of the hash functions. Each hash function contains a different key. The process includes calculating the probability of obtaining, for a particular bin, an observed hash number or less when the watermarked data release does not contain a digital watermark corresponding to the particular bin. When the calculated probability is lower than a predetermined threshold, the process infers that the data release contains a particular digital watermark. • The process allows reconstruction of the digital watermark for only a subset of the data release. • The process can handle noise, such as adding or removing rows from the data release. Extraction of the digital watermark is associated with a confidence score related to (a) the likelihood of the watermark's presence in the tokenized data, and (b) the likelihood that an underrepresented bin in the histogram of token hash counts corresponds to a hash bin. (How confident can we be that the watermark is actually present in the tokenized data and that an underrepresented bin in the histogram of token hash counts is not simply a coincidence?) The process includes a step of estimating the number of tokens required to reconstruct the digital watermark. ● The number of tokens is estimated for a given confidence level and the amount of noise present in the tokenized data.

[0179] Digital watermarking depends on the parameters of the data release A computer-implemented process includes generating a data release, and a digital watermark is chosen or selected based on parameters of the data release. • Digital watermarks are chosen or selected based on the recipient of the data release. • Digital watermarks are chosen or selected based on the intended use of the data release. • Digital watermarks are chosen or selected based on the date by which the data release must be removed. Each data release corresponding to the same tokenized input data set contains a different digital watermark. • A digital watermark represents a pattern that is a unique ID. • Digital watermarks are chosen or selected based on the type of tokenized input data (i.e., ID, Social Security number, high, salary, etc.). • A digital watermark is chosen or selected based on the encryption scheme used. The process includes dynamically scaling the number of possible watermarked data releases by updating the hash function. The process includes dynamically scaling the number of possible watermarked data releases by hashing the token space with a different hash function.

[0180] Computer device or system 1. A computing device or system adapted to generate tokens from an input dataset comprising input values ​​of varying lengths, the device or system comprising a processor configured to generate tokens from the input dataset, the tokens having lengths that exhibit a distribution substantially similar to the distribution of lengths of the input dataset.

[0181] 1. A computing device or system adapted to embed a digital watermark in tokenized data, the device or system comprising: (e) generating tokens from the input dataset, the tokens being generated with lengths exhibiting a distribution substantially similar to the distribution of lengths of the input dataset; (f) embedding a digital watermark within the generated set of tokens.

[0182] Appendix D - Embedding digital watermarks within output from generative AI, such as text generated by LLMs (large-scale language models) Since the described watermarking scheme is based on rejecting outputs that hash into watermark bins, any process can embed a watermark as long as it meets the following two criteria: 1. There must be a possibility to replace one potential output with another potential output, so if the first choice of output is a watermark token, a different, non-watermark token output may be chosen instead. 2. There must be a system for interpreting potential outputs as "tokens" that can be hashed to determine whether they are watermark tokens. This is obvious for text generation, but there are many ways to interpret binary output in the same fashion, for example using pixel values ​​in an image or video frame, or the frequency of a sound sample, as it is possible to make slight modifications to these without significantly affecting the output. When the term "token" is used in this text, it should be understood to refer to the potential output of any generation system that is interpreted in this fashion, rather than simply text.

[0183] Generative AI systems that generate text, images, video, or audio meet both criteria, so the scheme can be used to watermark their output. To see exactly how this works, let's take text generated by an LLM as an example. The LLM generates text by outputting a sequence of generated tokens, which can be, for example, words, parts of words, or punctuation. Each token is generated using a probability distribution for the most likely next token to generate in response to a prompt, given the previous token generated. However, the system does not always simply select the token with the highest probability, but rather has an element of randomness that may result in the system selecting a token with a lower probability. This is where a watermarking scheme can be inserted. Once the token the system is about to return hashes, it can determine whether the token is a watermark token. If it is a watermark token, the token can be discarded and resampled from the distribution to select a different token, repeating until a non-watermark token is selected, which can then be returned. (In the unlikely event that a non-watermark token is not available for selection, the watermark token must be returned, but this will only add a small amount of noise to the resulting watermark, as discussed in the "Handling Noise" section on page 25.)

[0184] The presence of a watermark can be used to determine that content was generated by a generative AI system. However, because each watermark may be unique to the prompt-entering user, extracting the watermark reveals not only that the content was generated by a generative AI, but also for which user. Additionally, any metadata about the user, time of use, and the context generating the content can be associated with the watermark.

[0185] Thus, using the described watermarking scheme, the generated AI output may be watermarked using any one or combination of the following: 1. Global - A single watermark for all generated AI output. This could be, for example, to verify that the output is machine generated or not. 2. Per-user scope - all output generated for the same user will have the same watermark, different users will get different watermarks. The watermark is machine-generated and indicates which user it is for. 3. Scoped per prompt - Every user and / or prompt combination gets a unique watermark, so you know not only which user, but also which prompt. Optionally, a timestamp can also be used to disambiguate the same prompt.

[0186] Advantageously, associating a digital watermark with either a particular user or a particular prompt, or a combination of both, allows for tracking of how the generated content is being used, thereby providing an increased level of security or customization.

[0187] This is possible because the system is scalable, allowing for the independent watermarking of any number of generated AI outputs.

[0188] Advantageously, removal or extraction of the digital watermark can be achieved from any sampled subset of tokens, regardless of whether the sampled subset of tokens is acquired contiguously or randomly.

[0189] This is possible because the implemented process is such that whether a particular token is or is not a watermark token is only a function of the token, the embedded watermark, and the watermarking key. Thus, whether a particular token is or is not a watermark token does not depend on previously generated tokens.

[0190] Here, we list some main features A1 to D1. Each feature listed below can be combined with any other feature A1 to D1. Each optional feature defined below can be combined with any other optional feature and any other optional feature.

[0191] Key Feature A1: The process of incorporating digital watermarking into "generative AI output" or generated content 1. A computer-implemented process for embedding a digital watermark in generated content, comprising: (a) generating tokens from input data, such as user prompts, using a generation AI, such as an LLM; (b) embedding a digital watermark within the generated set of tokens.

[0192] Key Feature B1: A process for incorporating digital watermarking within generated content, where the digital watermark is embedded probabilistically through the selection of tokens within a generated token set. 1. A computer-implemented process for embedding a digital watermark in generated content, comprising: (a) generating tokens from input data, such as user prompts, using a generation AI, such as an LLM; (b) embedding a digital watermark within the generated set of tokens; A computer-implemented process in which a digital watermark comprises a pattern that is probabilistically embedded through the selection of tokens within a generated token set.

[0193] Main Feature C1: A process for incorporating digital watermarking within generated content where the digital watermark can be reconstructed without prior knowledge of the generation process or any other processing used. 1. A computer-implemented process for embedding a digital watermark in generated content, comprising: (a) generating tokens from input data, such as user prompts, using a generation AI, such as an LLM; (b) embedding a digital watermark within the generated set of tokens; A computer-implemented process by which a digital watermark can be reconstructed without prior knowledge of the generation process or any other processing used.

[0194] Key Features D1: A process for incorporating digital watermarking within generated content that can dynamically scale the count of potentially watermarked content. 1. A computer-implemented process for embedding a digital watermark in generated content, comprising: (a) generating tokens from input data, such as user prompts, using a generation AI, such as an LLM; (b) embedding a digital watermark within the generated token set; (a) generating content, wherein a digital watermark is chosen or selected based on prompt input user or other parameters of the generation; A computer-implemented process that allows the count of possible watermarked content to be dynamically scaled.

[0195] Optional Features ● The generated content includes text, images, video, and / or audio.

[0196] The process of embedding a digital watermark - The digital watermark is embedded within the generated token set and not embedded in any metadata or redundant data. • Digital watermark patterns are embedded within watermark tokens, which are assigned or determined such that they hash to values ​​that fall within a predetermined range of a hash space. • Determining whether a generated token should be assigned a watermark token is simply a function of the generated token itself, the watermark embedded within it, and the secret watermarking key. • The tokens are hashed using a cryptographic hash function based on a secret watermarking key, so an attacker cannot learn in which subspace the token resides. The process includes attempting to never or rarely return watermark tokens such that bins of the hash space contain no values ​​or values ​​close to zero.

[0197] Reconstruction or extraction of digital watermarks from data releases ● The digital watermark can be reconstructed without prior knowledge of the scheme used to generate the token. ●The digital watermark is reconstructed by (a) hashing tokens in the content using one or more hash functions, (b) constructing a histogram of hash frequencies in the hash space, and (c) determining the bins that correspond to the digital watermark. When multiple hash functions are used, the process includes summing the hash numbers for each bin across all of the hash functions. Each hash function contains a different key. The process includes calculating the probability of obtaining, for a particular bin, an observed hash number or less when the watermarked data release does not contain a digital watermark corresponding to the particular bin. When the calculated probability is lower than a predetermined threshold, the process infers that the data release contains a particular digital watermark. • The process allows reconstruction of the digital watermark for only a subset of tokens from the generated content. The process can handle noise, such as adding, removing, or modifying tokens in the content. ● Extraction of the digital watermark is associated with a confidence score related to (a) the likelihood of the watermark's presence in the content, and (b) the likelihood that an underrepresented bin in the histogram of token hash counts corresponds to a hash bin (how confident we can be that the watermark is actually present in the content and that an underrepresented bin in the histogram of token hash counts is not simply a coincidence). The process includes a step of estimating the number of tokens required to reconstruct the digital watermark. ● The number of tokens is estimated for a given confidence level in the content tokens and the amount of noise present.

[0198] Digital watermarking depends on the parameters of the data release A computer-implemented process includes generating content, wherein a digital watermark is chosen or selected based on input data, prompts, or user parameters. - Digital watermarks are picked or selected based on user input prompts. Each piece of content generated in response to the same input data or user prompt may contain a different digital watermark. • A digital watermark represents a pattern that is a unique ID. The process includes dynamically scaling the count of possible watermarked content by updating the hash function. The process includes dynamically scaling the count of possible watermarked content by hashing the token space with a different hash function. ● A digital watermark is associated with the timestamp.

[0199] Computer device or system 1. A computing device or system adapted to embed a digital watermark in a generated AI output, the device or system including a processor, the processor comprising: (a) using a generative AI, such as an LLM, to generate tokens from input data, such as user prompts; (b) embedding a digital watermark within the generated set of tokens.

[0200] Usage example 1. A computing device or system adapted to detect the use of deepfakes, the device or system including a processor, the processor comprising: (a) receiving the content; (b) hashing the tokens in the entire received content or any subset of the received content using one or more hash functions to construct a hash frequency histogram of the hash space; (c) determining or estimating from the analysis of the histogram whether a digital watermark is embedded in the received content and the likelihood that the received content is a deepfake.

[0201] Optional Features: Deepfake refers to generated AI content that has been digitally watermarked using a computer-implemented process that includes: (a) using generative AI to generate tokens from input data, such as user prompts; and (b) embedding the digital watermark within the generated set of tokens. ●Detection of the use of deepfakes will be associated with a credibility score related to the likelihood that the content was generated by generative AI. ●The system is configured to provide an alert or notification when a deepfake is used. ●The system is configured to provide an alert or notification when a deepfake has not been used. ●The system is configured to present the deepfake along with metrics associated with the deepfake, including a confidence score, input data or user prompts, the type of generating AI, and user ID. ●The system is configured to automatically remove deepfakes from content. ●The system is configured to detect specific users who have used deepfakes based on digital watermarks. ● The system is configured to automatically block certain users. ●The system is configured to automatically detect the use of deepfakes when content is uploaded to social media, or a website, or an online gaming platform, or an email. ●The system is configured to determine or estimate the percentage of media content within social media, or email, or online gaming system, or website that is generated by generative AI. ●The system is configured to detect the use of deepfakes by analyzing only a subset of the generated content.

[0202] A social media system configured to detect the use of deepfakes by detecting digital watermarks embedded within social media content. ●The system includes a processor, the processor being configured to hash tokens in any subset of social media content using one or more hash functions to construct a histogram of hash frequencies in the hash space, and from analysis of the histogram to determine or estimate whether a digital watermark is embedded in the social media content and the likelihood that the social media content contains a deepfake.

[0203] An email system configured to detect the use of deepfakes by detecting digital watermarks embedded within email content. ●The system includes a processor, the processor being configured to hash tokens in any subset of email content using one or more hash functions to construct a histogram of hash frequencies in the hash space, and from analysis of the histogram to determine or estimate whether a digital watermark is embedded in the email content and the likelihood that the email content contains a deepfake.

[0204] An online gaming system configured to detect the use of deepfakes by detecting digital watermarks embedded within online gaming content. ●The system includes a processor, the processor being configured to hash tokens in any subset of the online game content using one or more hash functions to construct a histogram of hash frequencies in the hash space, and from analyzing the histogram, determine or estimate whether a digital watermark is embedded in the online game content and the likelihood that the online game content contains a deepfake.

[0205] A web browser system configured to detect the use of deepfakes by detecting digital watermarks embedded within website content. ●The system includes a processor, the processor being configured to hash tokens in any subset of website content using one or more hash functions to construct a histogram of hash frequencies in the hash space, and from analysis of the histogram to determine or estimate whether a digital watermark is embedded in the website content and the likelihood that the website content contains a deepfake.

[0206] Note It is to be understood that the above-described arrangements are merely illustrative of the application of the principles of the present invention. Numerous modifications and alternative arrangements can be devised without departing from the spirit and scope of the invention. While the invention has been illustrated in the drawings and fully described above with specificity and detail in connection with what are presently considered to be the most practical and preferred embodiments of the invention, it will be apparent to those skilled in the art that numerous modifications can be made without departing from the principles and concepts of the invention as set forth herein.

Claims

1. 1. A computer-implemented process for generating tokens from an input dataset comprising input values ​​of varying lengths, the process comprising generating tokens from the input dataset, the tokens having lengths that exhibit a distribution substantially similar to the distribution of lengths of the input dataset.

2. The computer-implemented process of claim 1 , wherein the length of a token does not necessarily match the length of its corresponding input value.

3. 3. The computer-implemented process of claim 1 or 2, wherein the tokens are generated using an encryption scheme that treats each subset of the input data corresponding to a particular length of the input value as a separate or distinct format-preserving encryption.

4. 10. A computer-implemented process according to any one of the preceding claims, wherein tokens are generated of different lengths for each subset of the input data corresponding to a particular length of the input value.

5. 10. A computer-implemented process according to any one of the preceding claims, wherein for each subset of the input data corresponding to a particular length of the input value, tokens are generated with varying lengths, the generated tokens having lengths conforming to some selected distribution generated for that particular length.

6. 10. A computer-implemented process according to any one of the preceding claims, wherein once a token is generated from the subset of the input data corresponding to that particular length of the input value, additional format-preserving encryption is used on the generated token.

7. 10. A computer-implemented process according to any one of the preceding claims, wherein tokens are generated based on one or more user-configurable parameters, including a minimum token length, a maximum token length, or a distribution used to assign token length weightings.

8. 1. A computer-implemented process for embedding a digital watermark in tokenized data, said process comprising: (a) generating, from an input dataset comprising input values ​​of varying lengths, tokens generated with lengths exhibiting a distribution substantially similar to the distribution of lengths of said input dataset; (b) embedding said digital watermark within the generated set of tokens.

9. The computer-implemented process of claim 8 , wherein the length of a token does not necessarily match the length of its corresponding input value.

10. 10. The computer-implemented process of claim 8 or 9, wherein the tokens are generated using an encryption scheme that treats each subset of the input data corresponding to a particular length of the input value as a separate or distinct format-preserving encryption.

11. A computer-implemented process according to any one of claims 8 to 10, wherein tokens are generated of different lengths for each subset of the input data corresponding to a particular length of the input value.

12. 12. The computer-implemented process of claim 8, wherein for each subset of the input data that corresponds to a particular length of input values, tokens are generated with varying lengths, the generated tokens having lengths that conform to some selected distribution generated for that particular length.

13. 13. The computer-implemented process of any one of claims 8 to 12, wherein once a token is generated from the subset of the input data that corresponds to that particular length of the input value, additional format-preserving encryption is used on the generated token.

14. 14. The computer-implemented process of any one of claims 8 to 13, wherein tokens are generated based on one or more user-configurable parameters including a minimum token length, a maximum token length, or a distribution used to assign token length weightings.

15. 15. The computer-implemented process of any one of claims 8 to 14, wherein the digital watermark comprises a pattern that is probabilistically embedded through the selection of tokens in the generated token set.

16. 16. A computer-implemented process according to any one of claims 8 to 15, wherein the digital watermark can be reconstructed without prior knowledge of an encryption scheme or any other processing used on the input data set.

17. 17. A computer-implemented process according to any one of claims 8 to 16, wherein the process includes generating a data release, and wherein the digital watermark is chosen or selected based on the parameters of the data release, allowing the number of possible watermarked data releases to be dynamically scalable.

18. 18. The computer-implemented process of any one of claims 8 to 17, wherein the digital watermark is embedded within the generated set of tokens and not embedded in any metadata or redundant data.

19. The process comprises: (a) scanning or observing an input space corresponding to an input data set; A computer-implemented process according to any one of claims 8 to 18, comprising: (b) determining or identifying the inputs to be mapped to watermark tokens.

20. A computer-implemented process according to any one of claims 8 to 19, wherein inputs to be mapped to watermark tokens are inferred from knowledge of the input space and what the input space represents.

21. 21. A computer-implemented process as claimed in any one of claims 8 to 20, wherein the inputs to be mapped to watermark tokens are inputs that are unlikely to occur or be encountered (unlikely occurring refers to utilizing a distribution of probabilities of encountering possible input values ​​so that tokens that we do not want returned are mapped to inputs that we would not expect to encounter).

22. 22. A computer-implemented process according to any one of claims 8 to 21, wherein the process comprises dividing the input space into two separate or disjoint subspaces, a "non-watermarked input subspace" and a "watermarked input subspace", wherein the inputs in the non-watermarked input subspace are mapped to non-watermarked tokens and the inputs in the watermarked input subspace are mapped to watermarked tokens.

23. A computer-implemented process according to any one of claims 8 to 22, wherein the encryption of each input subspace is achieved independently.

24. 24. The computer-implemented process of any one of claims 8 to 23, wherein the digital watermark pattern is embedded within watermark tokens, and the watermark tokens are assigned or determined such that they hash to values ​​that fall within a predetermined range of a hash space.

25. 25. The computer-implemented process of any one of claims 8 to 24, wherein hashing the tokens is done using a cryptographic hash function based on a secret watermarking key, so that an attacker cannot learn which subspace a token resides in.

26. 26. A computer-implemented process according to any one of claims 8 to 25, wherein the process comprises attempting to map the watermark token such that the watermark token is never or rarely returned such that a bin of the hash space contains no value or a value close to zero.

27. 27. The computer-implemented process of any one of claims 8 to 26, wherein watermark tokens are dynamically assigned or determined at run-time to avoid the need for a brute-force search of the entire token space.

28. 28. The computer-implemented process of any one of claims 8 to 27, wherein determining the watermark token comprises segmenting the token space into segments and assigning a watermark token per segment.

29. 29. The computer-implemented process of any one of claims 8 to 28, wherein a watermark token is assigned by scanning a segment to find the first token in the segment that hashes to a value within the predetermined range.

30. 30. The computer-implemented process of any one of claims 8 to 29, wherein a starting index for scanning each segment is chosen using a pseudo-random number generator (PRNG) seeded with the segment index.

31. 31. The computer-implemented process of any one of claims 8 to 30, wherein if the starting index is reached without finding a watermark token, the final index of the segment is chosen as the watermark token.

32. 32. The computer-implemented process of any one of claims 8 to 31, wherein the size of the segments is chosen to provide a high probability that each segment will contain no more than one token that hashes to the watermark range (assuming a uniform distribution from the hash function) to avoid having to return these tokens for inputs other than the intended input.

33. 33. A computer-implemented process according to any one of claims 8 to 32, wherein the digital watermark can be reconstructed without prior knowledge of the encryption scheme or any other scheme used on the input data set.

34. 34. The computer-implemented process of any one of claims 8 to 33, wherein the digital watermark is reconstructed by (a) hashing the tokens in the data release using one or more hash functions, (b) constructing a histogram of hash frequencies of the hash space, and (c) determining the bin that corresponds to the digital watermark.

35. 35. A computer-implemented process according to any one of claims 8 to 34, wherein when multiple hash functions are used, the process comprises summing the hash numbers for each bin across all of the hash functions.

36. The computer-implemented process of any one of claims 8 to 35, wherein each hash function comprises a different key.

37. 37. A computer-implemented process according to any one of claims 8 to 36, wherein the process comprises calculating the probability of obtaining an observed hash number or less in a particular bin when the watermarked data release does not include the digital watermark corresponding to the particular bin.

38. A computer-implemented process according to any one of claims 8 to 37, wherein when the calculated probability is lower than a predetermined threshold, the process infers that the data release contains a particular digital watermark.

39. A computer-implemented process according to any one of claims 8 to 38, wherein the process enables the reconstruction of the digital watermark for only a subset of data releases.

40. 40. A computer-implemented process according to any one of claims 8 to 39, wherein the process is capable of dealing with noise, such as adding or removing rows from the data release.

41. 41. The computer-implemented process of claim 8, wherein the extraction of the digital watermark is associated with a confidence score related to (a) the likelihood of the watermark's presence in the tokenized data and (b) the likelihood that an underrepresented bin in a histogram of token hash counts corresponds to a hash bin.

42. A computer-implemented process according to any one of claims 8 to 41, wherein the process includes estimating the number of tokens required to reconstruct the digital watermark.

43. 43. The computer-implemented process of any one of claims 8 to 42, wherein the number of tokens is estimated for a given confidence level and amount of noise present in the tokenized data.

44. 44. A computer-implemented process according to any one of claims 8 to 43, wherein the computer-implemented process includes generating a data release, and wherein the digital watermark is chosen or selected based on the parameters of the data release.

45. 45. The computer-implemented process of any one of claims 8 to 44, wherein the digital watermark is chosen or selected based on the recipient of the data release.

46. 46. ​​The computer-implemented process of any one of claims 8 to 45, wherein the digital watermark is chosen or selected based on the intended use of the data release.

47. 47. The computer-implemented process of any one of claims 8 to 46, wherein the digital watermark is chosen or selected based on a date by which the data release must be removed.

48. 48. A computer-implemented process according to any one of claims 8 to 47, wherein each data release corresponding to the same tokenized input data set includes a different digital watermark.

49. 49. The computer-implemented process of any one of claims 8 to 48, wherein the digital watermark represents a pattern that is a unique ID.

50. 50. The computer-implemented process of any one of claims 8 to 49, wherein the digital watermark is chosen or selected based on the type of tokenized input data.

51. A computer-implemented process according to any one of claims 8 to 50, wherein the digital watermark is chosen or selected based on the encryption scheme used.

52. 52. The computer-implemented process of any one of claims 8 to 51, wherein the process includes dynamically scaling the number of possible watermarked data releases by updating the hash function.

53. 53. The computer-implemented process of any one of claims 8 to 52, wherein the process includes dynamically scaling the number of possible watermarked data releases by hashing the token space with another hash function.

54. 1. A computing system adapted to generate tokens from an input dataset comprising input values ​​of varying lengths, wherein the device or system comprises a processor configured to generate tokens from the input dataset, the tokens having lengths that exhibit a distribution substantially similar to the distribution of lengths of the input dataset.

55. 1. A computing system adapted to embed a digital watermark in tokenized data, the device or system including a processor, the processor comprising: (a) generating tokens from an input dataset comprising input values ​​of varying lengths, the tokens having lengths that exhibit a distribution substantially similar to the distribution of lengths of the input dataset; (b) embedding said digital watermark within the generated set of tokens.

Images