Black-box large language model watermarking and detection method based on sampling and priority output

Abstract

The present application relates to the technical field of watermark detection, and in particular to a black-box large language model watermark and detection method based on sampling and priority output. The black-box large language model watermark method based on sampling and priority output proposed by the present application adopts a strategy combining "priority output" and "fair distribution" in the watermark embedding process. The priority output is that the tokens generated by the large language model are sequentially distributed to the respective subspaces according to the subword table, and the fair distribution is that the tokens overflowing from the subspaces are redistributed to the remaining idle subspaces. The embedding and extraction process of the watermark in the present application does not depend on the specific parameters (such as probability distribution) of the model at all, and it is a purely unbiased black-box watermark method. The embedding of the watermark does not affect the quality of the generated content, and the applicability and robustness of the method are improved.

Description

Technical Field

[0001] This invention relates to the field of watermark detection technology, and in particular to a black-box large language model watermark and detection method based on sampling and priority output. Background Technology

[0002] In recent years, large language model technology has developed at an astonishing pace. With its superior output quality and low cost, it has significantly improved the efficiency of text-based content production, bringing revolutionary changes to the field of writing. However, while technological innovation brings convenience, it has also raised serious ethical and security issues. The text generated by these models is close to or reaches human-level quality, making it increasingly difficult to distinguish between generated content and human-written content. Simultaneously, its high accessibility and low cost provide irresponsible users with opportunities to exploit these models for malicious activities. Currently, issues such as the generation of fake news and the creation of malicious content are closely related to the misuse of large language models. Therefore, while innovating and promoting large language model technology, it is necessary to reasonably control it, that is, effectively distinguish between content generated by large language models and human-written content, thereby curbing the misuse of the models.

[0003] Watermarking technology, as an effective solution, is widely used in the field of digital rights protection. However, existing watermarking methods each have their advantages and disadvantages, making it difficult to meet high-performance requirements in complex environments. For example, the KGW method embeds watermarks by utilizing the biased characteristics of red-green lists, achieving high robustness and efficiency, but its implementation often sacrifices the quality of the generated content, classifying it as a biased watermarking method. The subsequently proposed Unbiased method improves upon this issue by randomly sampling and selecting watermark tokens, ensuring that the quality of the generated content remains unaffected; however, its robustness is poor, making it vulnerable to attacks that could cause the watermark to fail.

[0004] Most existing watermarking methods are based on white-box embedding mechanisms, which require detailed model parameters (such as the probability distribution of generation) when embedding watermarks. This is unhelpful for situations where model owners want to protect their parameters. Researchers have implemented unbiased watermark embedding using black-box methods (such as the STA method and the SynthID-Text method), but the robustness of these black-box methods is difficult to match that of biased white-box methods (such as the KGW method). Specifically, the SynthID-Text method assigns random weights to tokens and uses tournament sampling to obtain output tokens according to their weights. This method ensures the unbiasedness of the output, but the predictability of generated tokens is insufficient; that is, it can only estimate which tokens are more likely to be generated through weights, and cannot form a visual watermark vocabulary. The STA method encourages the model to output tokens from the watermarked vocabulary by dividing the watermarked vocabulary and rejecting tokens not from the watermarked vocabulary. However, it faces a trade-off between unbiasedness and robustness: the STA-1 method is unbiased when rejecting only once, but its robustness is not ideal; while the STA-M method improves robustness but loses unbiasedness when rejecting multiple times. It is evident that existing black-box methods struggle to balance unbiasedness and robustness. Summary of the Invention

[0005] To overcome the shortcomings of existing watermarking methods in complex environments, which cannot balance unbiasedness and robustness, this invention proposes a black-box large language model watermarking method based on sampling and priority output, which can be compatible with unbiasedness and robustness in complex scenarios.

[0006] This invention proposes a watermarking method for black-box large language models based on sampling and priority output, comprising the following steps: S1. First, set the L and N values, where N / L is an integer; determine the large language model and its corresponding complete vocabulary; the large language model generates and outputs text content based on prompts; S2. The large language model generates candidate tokens for the target position based on the requirements, the complete vocabulary, and the preceding text; the preceding text corresponding to the beginning position of the text is a hint. S3. Divide the complete vocabulary into L sub-vocabularies based on the context of the target position; construct a subspace of size N / L for each sub-vocabulary; randomly select one of the sub-vocabularies as the watermark vocabulary; S4. Select N candidate tokens as initial screening objects, and fill the corresponding subspaces with the initial screening objects one by one; S5. Perform initial screening and sampling on the subspace corresponding to the watermark vocabulary to obtain the sampled objects and place them in the target position of the text. S6. Update the target position and the preceding text, and repeat steps S2-S5 until the text generation is complete.

[0007] Preferably, in step S4, each preliminary screening object is first filled into the subspace corresponding to its respective sub-word list. If the subspace overflows, the preliminary screening object is placed into the waiting queue. After all preliminary screening objects have entered the subspace or the waiting queue, the preliminary screening objects in the waiting queue are migrated to the free subspace one by one.

[0008] Preferably, in step S4, according to the first-in-first-out principle, the preliminary screening objects in the waiting queue are migrated one by one to the free subspace that is traversed sequentially.

[0009] This invention proposes a watermark detection method using the sampling-based and priority-output black-box large language model watermarking method. The method is characterized in that, during the text generation process using the large language model, the service provider combines a specified key with the method, ensuring that the service provider and the detector generate the same watermark sub-vocabulary segmentation result under the same context. The detector obtains the key, the text to be detected, and the large language model. Based on the context, the detector reconstructs the watermark sub-vocabulary used at each token generation position in the text to be detected and searches for watermark tokens in the text.

[0010] Preferably, the detection party uses the following steps to determine the watermark token in the text to be detected: The detection party obtains the text to be detected and its associated tags {complete vocabulary, key, L, N}; The detection team reconstructs each token and the watermark sub-word list used in the token generation step from beginning to end of the detected text; when a token belongs to the watermark sub-word list, the token is recorded as a watermark token.

[0011] Preferably, the detection party calculates the test value Z of the text to be detected, and determines whether the text to be detected is a real text generated by a large language model based on the comparison result of the test value Z and a set threshold. ; Where W is the number of watermark tokens in the detected text, n is the total number of tokens in the detected text divided according to the complete vocabulary, and L is the number of sub-vocabularies.

[0012] The present invention proposes a text generation system, characterized in that it includes a service provider and a detection provider; the service provider selects a large language model, L, and N, and then uses the black-box large language model watermarking method based on sampling and priority output to generate text content, and sends the text content with tags {complete vocabulary, key, L, N} to the detection provider; the detection provider uses the watermark detection method to detect the received text content and feeds back the detection results to the service provider.

[0013] Preferably, the service provider uses multiple generation strategies to generate text content, and then selects the target generation strategy based on the feedback results from the detection party; the generation strategies include: large language model, L value, and N value.

[0014] The present invention proposes a text generation device, including a memory and a processor. The memory stores a computer program, and the processor is connected to the memory. The processor is used to execute the computer program to implement the black-box large language model watermarking method based on sampling and priority output.

[0015] The present invention proposes a storage medium storing a computer program, which, when executed, is used to implement the black-box large language model watermarking method based on sampling and priority output.

[0016] The advantages of this invention are: (1) The black-box large language model watermarking method based on sampling and priority output proposed in this invention adopts a strategy combining "priority output" and "fair allocation" in the watermark embedding process. Priority output means that the tokens generated by the large language model (LLM) are sequentially allocated to their respective subspaces according to the subvocabulary. Fair allocation means that the tokens overflowing from the subspace are redistributed to the remaining idle subspaces. The watermark embedding and extraction process in this invention does not depend on the specific parameters of the model (such as probability distribution), and is an unbiased black-box watermarking method. The watermark embedding does not affect the quality of the generated content, thus improving the applicability and robustness of the method.

[0017] (2) The present invention also allows for freely adjustable hyperparameters L and N, which can effectively improve the detectability of watermarks and better adapt to the needs of complex real-world environments.

[0018] (3) In this invention, the initial screening objects that overflow from the subspace are temporarily stored in a queue and then dequeued in order to fill the remaining empty spaces in the subspace. Thus, based on the sub-vocabulary partitioning, each token is fairly allocated, ensuring that the expected probability of outputting any token remains unchanged. This invention, through the coordinated use of random sub-vocabulary partitioning, random selection of the watermark vocabulary, and placement of initial screening objects, ensures the randomness of whether a token at each generation position is a watermark token, thereby guaranteeing the unbiasedness of the watermarking method.

[0019] (4) The watermark detection method proposed in this invention combines key inversion to deduce the vocabulary used in each token generation step, and determines whether the text was generated by machine by counting the number of watermark tokens and performing a Z-test. The watermark detection method achieves black-box detection of watermarks, that is, no sensitive model parameters are required in the watermark extraction process, which improves the applicability of the method and has good adaptability to different environments.

[0020] (5) The text generation system proposed in this invention combines the black-box large language model watermarking method and watermark detection method based on sampling and priority output proposed in this invention to realize the black-box embedding and detection of watermarks, which can be used to optimize watermark text generation in any scenario. Attached Figure Description

[0021] Figure 1 The flowchart is a black-box large language model watermarking method based on sampling and priority output proposed in this invention. Figure 2This is a topological diagram of a black-box large language model watermarking method based on sampling and priority output proposed in this invention. Figure 3 This is a flowchart of a watermark detection method proposed in this invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figure 1 , Figure 2 As shown, the present invention proposes a black-box large language model watermarking method based on sampling and priority output, which includes the following steps S1-S6.

[0024] S1. First, set the values ​​of L and N, where N / L is an integer; determine the large language model LLM and its corresponding complete vocabulary V; the large language model generates and outputs text content based on the prompt, and the tokens in the text content all come from its corresponding complete vocabulary V; S2. The large language model generates candidate tokens for the target position based on the prompts, the complete vocabulary, and the preceding text; the preceding text refers to the text content generated before the target position, and the preceding text corresponding to the beginning of the text is the prompt. S3. Divide the complete vocabulary into L sub-vocabularies based on the context of the target position; construct a subspace of size N / L for each sub-vocabulary; randomly select one of the sub-vocabularies as the watermark vocabulary; S4. Select N candidate tokens as initial screening objects, and fill the corresponding subspaces with the initial screening objects one by one.

[0025] Specifically, in this step, each initially screened object is first filled into the subspace corresponding to its respective sub-word list. If the subspace overflows, the initially screened objects are placed in a waiting queue. After all initially screened objects have entered a subspace or the waiting queue, the initially screened objects in the waiting queue are migrated one by one to an empty subspace according to the first-in-first-out principle. Specifically, during the migration process, the subspaces are traversed in ascending order of their subspace index (i.e., the order of the subspace's corresponding word list), and the initially screened objects in the waiting queue are migrated one by one to an empty subspace.

[0026] S5. Perform initial screening and sampling on the subspace corresponding to the watermark vocabulary to obtain the sampled objects and place them in the target position of the text. At this point, if the sampled object belongs to the watermark vocabulary, it means that the watermark has been successfully embedded in the sampled object; Conversely, if the sampled object does not belong to the watermark vocabulary, it means that the sampled object has not embedded a watermark. S6. Update the target position and the preceding text, and repeat steps S2-S5 until the text generation is complete.

[0027] For each sampled object generated, the target position is updated to the position following the current target position, and the currently generated sampled object is added to the previous text.

[0028] During the text generation process of the large language model, it works in conjunction with a specified key. Given the key, it is possible to reconstruct the sub-vocabulary partitioning method used for each token generation position.

[0029] The key is used to ensure that the service provider and the detector generate the same sub-vocabulary segmentation result under the same preamble, thereby ensuring the effectiveness of the watermark.

[0030] Reference Figure 3 This embodiment proposes a black-box unbiased large language model watermark detection method based on sampling and priority output, which includes the following steps: St1. First, the service provider generates the watermarked text using the above steps S1-S6, and then outputs the watermarked text after associating it with the tags {complete vocabulary, key, L, N}. St2, the detector obtains the detection text and its associated tags {complete vocabulary, key, L, N}, and reconstructs each token and the watermark sub-vocabulary used in the token generation step from beginning to end for the detection text; when a token belongs to the watermark sub-vocabulary, the token is recorded as the watermark token. St3, calculate the test value Z; ; Where W is the number of watermark tokens in the detected text, n is the total number of tokens in the detected text divided according to the complete vocabulary, and L is the number of sub-vocabularies; St4. Determine whether Z is greater than the set threshold; the threshold can be set theoretically based on the hypothesis test confidence level, or empirically based on the actual situation; Yes, it means that the detected text contains enough watermark tokens, that is, the detected text was generated by the service provider using a large language model; No, this means the detected text does not contain a valid watermark.

[0031] Example: The requirement is to "generate an essay about a spring outing that begins with the date"; Given: a large language model, a complete vocabulary; set values ​​L=5 and N=10; Step 1: The large language model generates candidate tokens based on the requirements and the complete vocabulary, including: today, tomorrow, now, weekend, etc. The complete vocabulary is divided into L=5 sub-vocabularies, and each sub-vocabulary constructs a subspace of length N / L=2; and any one of the sub-vocabularies is designated as the watermark vocabulary. Select N=10 candidate tokens as initial screening objects, and fill each initial screening object into the subspace corresponding to its sub-vocabulary. If the subspace overflows, the initial screening object is put into the waiting queue. After all initial screening objects have entered the subspace or the waiting queue, the initial screening objects in the waiting queue are moved to the free subspace one by one according to the first-in-first-out principle. The watermark terminology is initially screened and sampled in the corresponding subspace. The sampled objects are then placed in the generation position, and the preceding text and generation position are updated.

[0032] If the sampled object belongs to the watermark vocabulary, the sampled object placed in the text in this step will successfully embed the watermark; otherwise, the sampled object placed in the text in this step will not embed the watermark. Let's assume this step yields the token "today".

[0033] Step 2: The large language model generates candidate tokens based on the requirements, the complete vocabulary, and the preceding "today," including: yes, possible, no, suitable, true, etc.

[0034] The complete vocabulary is divided into L=5 sub-vocabularies based on the candidate tokens, and each sub-vocabulary constructs a subspace of length N / L=2; and any one of the sub-vocabularies is designated as the watermark vocabulary. Select N=10 candidate tokens as initial screening objects, and fill each initial screening object into the subspace corresponding to its sub-vocabulary. If the subspace overflows, the initial screening object is put into the waiting queue. After all initial screening objects have entered the subspace or the waiting queue, the initial screening objects in the waiting queue are moved to the free subspace one by one according to the first-in-first-out principle. The watermark terminology is initially screened and sampled in the corresponding subspace. The sampled objects are then placed in the generation position, and the preceding text and generation position are updated.

[0035] If the sampled object belongs to the watermark vocabulary, the sampled object placed in the text in this step will successfully embed the watermark; otherwise, the sampled object placed in the text in this step will not embed the watermark. Assume that this step yields the token "Yes".

[0036] Step 3: The large language model generates candidate tokens based on the requirements, the complete vocabulary, and the preceding phrase "Today is," including: good, sunny, rainy, cloudy, weekend, happy, etc.

[0037] The complete vocabulary is divided into L=5 sub-vocabularies based on the candidate tokens, and each sub-vocabulary constructs a subspace of length N / L=2; and any one of the sub-vocabularies is designated as the watermark vocabulary. Select N=10 candidate tokens as primary screening objects, fill each primary screening object into the subspace corresponding to its belonging sub-vocabulary, and if the subspace overflows, place the primary screening object into a waiting queue; after all primary screening objects enter the subspaces or the waiting queue, migrate the primary screening objects in the waiting queue one by one to the idle subspaces according to the first-in-first-out principle; Perform primary screening object sampling on the subspace corresponding to the watermark vocabulary, obtain a sampled object and place it at the generation position.

[0038] If the sampled object belongs to the watermark vocabulary, the sampled object placed into the text in this step is successfully embedded with a watermark; conversely, the sampled object placed into the text in this step is not embedded with a watermark; Assume that the token "sunny day" is obtained in this step.

[0039] Assume that in this embodiment, the text generated by the large language model is exactly "Today is a sunny day". To facilitate process traceability, it is specified that the large language model generates tokens in combination with the secret key, the preceding context and the complete vocabulary; with the secret key known, the division manner of the sub-vocabulary used for each token generation position can be subsequently deduced reversely, so as to judge from front to back for the text "Today is a sunny day" whether each token comes from the watermark vocabulary of the corresponding step, so as to count the total number of watermark tokens in the text.

[0040] Specifically, the reverse deduction steps are as follows: Step 1: Extract the token "Today" from "Today is a sunny day" according to the complete vocabulary, and reversely deduce the watermark vocabulary corresponding to this generation position in combination with the secret key; if "Today" is located in the watermark vocabulary, it is a watermark token, otherwise "Today" is not a watermark token; Step 2: Extract the token "is" located after "Today" in "Today is a sunny day" according to the complete vocabulary; reversely deduce the watermark vocabulary corresponding to this generation position in combination with the secret key; if "is" is located in the watermark vocabulary, it is a watermark token, otherwise "is" is not a watermark token; Step 3: Extract the token "sunny day" located after "is" in "Today is a sunny day" according to the complete vocabulary; reversely deduce the watermark vocabulary corresponding to this generation position in combination with the secret key; if "sunny day" is located in the watermark vocabulary, it is a watermark token, otherwise "sunny day" is not a watermark token; Record the total number of watermark tokens in "Today is a sunny day" as W, and count the total number of tokens n=3 in "Today is a sunny day"; substitute into the following formula to calculate the Z value: ; Z is greater than the set threshold; it indicates that the number of watermark tokens is qualified; conversely, it indicates that the text does not contain a valid watermark.

[0041] The following specific embodiments verify the black-box large language model watermarking method based on sampling and priority output (hereinafter referred to as SPO watermarking method) proposed in this invention.

[0042] Various parameter configurations were selected to study the influence of different parameters on the unbiasedness of the method. The experimental results are shown in Table 1.

[0043] Table 1. Results of Unbiasedness Experiment

[0044] BERTScore is an automated metric for evaluating the quality of natural language generation (NLG) tasks; KGW and MPAC are common biased watermarking methods, while γ-reweight, DiPmark, and BiMark are common unbiased watermarking methods.

[0045] For the KGW single-bit watermarking algorithm and the MPAC multi-bit watermarking algorithm, the quality metrics of the generated text fluctuated significantly compared to the results without embedded watermarks. Furthermore, the fluctuations became more pronounced as the hyperparameter δ of the KGW method increased. This indicates that the biased watermarking method, due to watermark embedding, led to a decrease in the quality of the generated text, resulting in a significant deviation in machine evaluation metrics compared to the unwatermarked data. In contrast, for the unbiased SPO watermarking method, although the metrics showed slight fluctuations under different parameter settings, the overall performance remained close to the unwatermarked data. Compared to other unbiased methods, the SPO watermarking method did not exhibit any significant anomalies in terms of metric fluctuations.

[0046] It is evident that the SPO watermarking method does not compromise the quality of the generated content while embedding the watermark, making it an unbiased watermarking method. Maintaining the quality of the generated content is particularly important for watermarking methods intended for public use, and unbiased watermarking precisely meets this requirement.

[0047] In this embodiment, a detectability experiment was also conducted.

[0048] 1,000 test samples of length 100 were generated using various watermarking methods, and the true positive rate (TPR, the probability that unwatermarked text is misdetected as watermarked text) was reported at a fixed false positive rate (FPR, the probability that watermarked text is correctly detected as watermarked text) to evaluate the detection performance of each method. The TPR of the SPO watermarking method under different FPR settings is shown in Table 2, and a comparative analysis was performed with existing state-of-the-art watermarking methods.

[0049] Table 2 Single-bit watermark detection results

[0050] In Table 2, TPR@FPR=0.1 represents the TPR when FPR=0.1, TPR@FPR=0.05 represents the TPR when FPR=0.05, and TPR@FPR=0.01 represents the TPR when FPR=0.01.

[0051] As shown in Table 2, the SPO watermarking method, as a black-box watermarking method, exhibits strong detection performance. When the parameters are set to... =4 and When the hyperparameter is 2, the detection capability of the SPO watermarking method is comparable to that of established unbiased watermarking techniques (such as DipMark). Furthermore, with the hyperparameter... With the increase in the amount of watermarking, detection performance improves, approaching that of the biased KGW method. These findings indicate that, while controlling for the false positive rate, the SPO watermarking method can accurately detect watermarked text, making it a promising solution for generated text detection tasks.

[0052] In this embodiment, a robustness experiment was also conducted, and the results were evaluated using AUC values. The results are shown in Table 3.

[0053] Table 3. Robustness test results (AUC values)

[0054] Attack intensity represents the percentage of tokens that are attacked.

[0055] As shown in Table 3, the biased KGW method exhibits significant robustness against substitution attacks. When δ=3.0, the AUC remains almost unchanged even after perturbation. However, this resilience is accompanied by a significant decrease in text quality. Among the unbiased methods, the SPO watermarking method shows a relatively small decrease in AUC, comparable to the STA-1 method. In contrast, the AUCs of γ-reweight and DiPmark show more significant decreases with increasing perturbation strength, highlighting the superior robustness of the SPO watermarking method. This indicates that under moderate attack conditions, the SPO watermarking method can maintain the detectability of the watermark, which is crucial for reliably identifying generated content. The SPO watermarking method demonstrates its effectiveness in resisting such attacks while maintaining detection accuracy, making it a promising method for practical applications.

[0056] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0057] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0058] The technologies, shapes, and structures not described in detail in this invention are all known technologies.

Claims

1. A watermarking method for a black-box large language model based on sampling and priority output, characterized in that, Includes the following steps: S1. First, set the L and N values, where N / L is an integer; determine the large language model and its corresponding complete vocabulary. Large language models generate and output text content based on prompts; S2. The large language model generates candidate tokens for the target position based on the requirements, the complete vocabulary, and the preceding text; the preceding text corresponding to the beginning position of the text is a hint. S3. Divide the complete vocabulary into L sub-vocabularies based on the context of the target position; construct a subspace of size N / L for each sub-vocabulary; randomly select one of the sub-vocabularies as the watermark vocabulary; S4. Select N candidate tokens as initial screening objects, and fill the corresponding subspaces with the initial screening objects one by one; S5. Perform initial screening and sampling on the subspace corresponding to the watermark vocabulary to obtain the sampled objects and place them in the target position of the text. S6. Update the target position and the preceding text, and repeat steps S2-S5 until the text generation is complete.

2. The black-box large language model watermarking method based on sampling and priority output as described in claim 1, characterized in that, In step S4, each initial screening object is first filled into the subspace corresponding to its respective sub-word list. If the subspace overflows, the initial screening object is placed into the waiting queue. After all the initial screening objects have entered the subspace or the waiting queue, the initial screening objects in the waiting queue are moved to the free subspace one by one.

3. The black-box large language model watermarking method based on sampling and priority output as described in claim 2, characterized in that, In step S4, according to the first-in-first-out principle, the initial screening objects in the waiting queue are moved one by one to the free subspace that is traversed sequentially.

4. A watermark detection method employing the black-box large language model watermarking method based on sampling and priority output as described in any one of claims 1-3, characterized in that, In the black-box large language model watermarking method based on sampling and priority output as described in any one of claims 1-3, the large language model used by the service provider works in conjunction with a specified key during the text generation process, so that the service provider and the detection provider generate the same watermark sub-vocabulary segmentation result under the same context; the detection provider obtains the key, the text to be detected and the large language model, and the detection provider restores the watermark sub-vocabulary used for each token generation position in the text to be detected according to the context, and searches for watermark tokens in the text to be detected.

5. The watermark detection method as described in claim 4, characterized in that, The detection team uses the following steps to determine the watermark token in the text to be detected: The detection party obtains the text to be detected and its associated tags {complete vocabulary, key, L, N}; The detection team reconstructs each token and the watermark sub-word list used in the token generation step from beginning to end of the detected text; when a token belongs to the watermark sub-word list, the token is recorded as a watermark token.

6. The watermark detection method as described in claim 5, characterized in that, The detection team calculates the test value Z of the text to be detected, and judges whether the text to be detected is a real text generated by the large language model based on the comparison result of the test value Z and the set threshold. Where W is the number of watermark tokens in the detected text, n is the total number of tokens in the detected text divided according to the complete vocabulary, and L is the number of sub-vocabularies.

7. A text generation system, characterized in that, The system includes a service provider and a detection provider. The service provider selects a large language model, L, and N, and then uses the black-box large language model watermarking method based on sampling and priority output as described in any one of claims 1-3 to generate text content. The service provider associates the text content with tags {complete vocabulary, key, L, N} and sends it to the detection provider. The detection provider uses the watermark detection method as described in any one of claims 4-6 to detect the received text content and feeds back the detection results to the service provider.

8. The text generation system as described in claim 7, characterized in that, The service provider uses multiple generation strategies to generate text content, and then selects the target generation strategy based on the feedback from the detection team; the generation strategies include: large language model, L value, and N value.

9. A text generation device, characterized in that, It includes a memory and a processor. The memory stores a computer program, and the processor is connected to the memory. The processor is used to execute the computer program to implement the black-box large language model watermarking method based on sampling and priority output as described in any one of claims 1-3.

10. A storage medium, characterized in that, The system contains a computer program that, when executed, implements the black-box large language model watermarking method based on sampling and priority output as described in any one of claims 1-3.

Images