A black-box watermark embedding method for a neural network text classification model

Abstract

The application discloses a black box watermark embedding method for a neural network text classification model, which comprises the following steps: firstly, constructing a watermark trigger set containing semantic level and character level triggers; secondly, mixing the constructed watermark trigger set with normal training samples to form a mixed data set; thirdly, training the classification model based on the mixed data set, so that the trained classification model encodes watermark information, wherein the optimization target of the training is to minimize the overall loss function; and the application can effectively solve the problem that the trigger set sample destroys the semantic discreteness and structure of the text, and the design of the double trigger not only improves the strength of the watermark signal, but also ensures that the text can still retain another type of trigger when the trigger set itself is subjected to preprocessing operations such as text cleaning, so that the robustness of the watermark against attacks is significantly enhanced through embedding at different levels.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence model security, more specifically, a black box watermark embedding method based on a neural network text classification model. BACKGROUND

[0002] With the rapid development of natural language processing technology, text classification models based on deep neural networks have been widely used in sentiment analysis, content review, spam filtering and many other commercial scenarios. The training of such models requires a large amount of high-quality labeled data, computing resources and expert knowledge, which constitutes an important core digital asset for enterprises. However, once the model is deployed as a black box service, it faces the risk of being illegally copied, stolen or distributed without authorization. Since the internal parameters of the model cannot be directly probed, the model owner has difficulty in providing effective technical evidence for identification and tracing when infringement occurs, which seriously hinders the healthy transaction and ecological development of AI technology.

[0003] To solve the above problems, model watermarking technology has emerged, which embeds a hidden mark during model training or inference, so that the model can produce a pre-set, unique mark output when receiving a specific trigger input, thereby providing verifiable evidence for model ownership.

[0004] Current black box watermarking methods for text classification models face several core challenges, the root cause of which lies in the inherent contradiction between the discrete nature of text data, semantic integrity and the black box access restrictions of the model:

[0005] 1) The trigger samples generated by the current mainstream black box watermarking methods generally have the problem of unnatural semantics and are easily identified by statistical anomaly detection mechanisms. At the same time, when facing common operations such as model fine-tuning and pruning, the robustness of the watermark is often insufficient;

[0006] 2) Methods that directly embed watermarks by simply concatenating or perturbing key words, although intuitive, can easily destroy the semantic coherence and logical integrity of the text. As a result, the generated trigger samples have poor concealment and often cause a significant decline in the performance of the model's main task;

[0007] 3) Directly migrating continuous perturbation watermarking methods designed for continuous data such as images to the text domain ignores the discrete nature of text, which destroys the semantic discreteness and structure of the text, resulting in illogical trigger texts that severely damage the performance of the model's main task. SUMMARY

[0008] The present application aims to solve the problems of destroying the semantics and structure of the text due to the constructed trigger set samples, and the lack of watermark concealment and robustness in the prior art.

[0009] To achieve the above objectives, this invention discloses a black-box watermark embedding method for neural network text classification models. By simultaneously constructing semantic-level and character-level dual triggers, it generates structurally complete trigger samples. While maintaining the natural semantic discreteness and structure of the text, it significantly enhances the concealment and attack robustness of the watermark. Specifically, it includes the following steps:

[0010] Step S1: Construct a watermark trigger set, which includes first selecting a fixed proportion of samples from an original category in the original training set; then performing semantic reconstruction processing on the selected samples and embedding predefined special character sequences to construct trigger samples that contain both semantic and character-level triggers; finally, assigning a unified new category label to all processed trigger samples, which is different from all original categories.

[0011] Step S2: Randomly mix the watermark trigger set constructed in step S1 with ordinary training samples to form a mixed dataset. The ordinary training samples are those samples in the original training set that were not selected into the watermark trigger set.

[0012] Step S3: Train the classification model based on the mixed dataset obtained in step S2, so that the parameters of the trained classification model are encoded with watermark information. The optimization objective of the training is to minimize the overall loss function. The overall loss function Represented as: Where λ is a preset weighting coefficient used to adjust the watermark embedding strength; if the watermark detection rate is too low, it can be appropriately increased based on experimental results; This represents the loss calculated based on the prediction results of the classification model on the ordinary training samples and their original class labels, used to maintain the performance of the classification model in the main text classification task; The loss is calculated based on the prediction results of the watermark trigger set samples by the classification model and the unified new category label, and is used to induce the model to learn watermark behavior.

[0013] Furthermore, while preserving the core semantic features, the semantics are reconstructed, and this process specifically includes the following steps:

[0014] Step S11: Extract key semantic features of the selected samples using natural language processing tools. The key semantic features include at least one of core entities, action features, and topic classification.

[0015] Step S12: First, construct a structured generation prompt based on the extracted key semantic features. The structured generation prompt includes a core description layer for defining the core topic, a category instruction layer for specifying the text category, and a format constraint layer for constraining the text format or style. Then, use the flan-T5-base pre-trained text generation model to generate new text samples that preserve semantics and reconstruct the format under the constraints of preset generation parameters. The preset generation parameters include at least one of the following: main entity occurrence degree, category keyword matching degree, and action relevance.

[0016] Step S13: Perform quality detection on the generated new text samples. The quality detection includes an automated evaluation of at least one of the following: text integrity, core entity matching degree, and irrelevant content exclusion degree. Add the samples that meet the quality requirements to the watermark trigger set, and repeat steps S11 to S12 until the number of samples in the watermark trigger set reaches the preset requirement.

[0017] This process solves the problems of semantic incoherence and topic deviation that occur in traditional trigger sets after watermark embedding. It provides high-quality and highly relevant trigger samples for subsequent watermark detection, thereby improving the reliability of watermark verification. It retains the core discrete semantic features of the original samples and can be integrated into the model training process, avoiding being judged as noise by the model. At the same time, after multiple rounds of iteration and strict parameter constraints, the generated trigger set text has high quality, can achieve style uniformity, reasonable length, and topic alignment, making watermark embedding more stable.

[0018] Furthermore, the predefined special character sequence in step S1 is inserted into the beginning or middle of the trigger set sample sentence to enhance trigger stability. The special character sequence is specifically at least 8 zero-width invisible characters, using the format of zero-width space ZWSP and Unicode character U+200B.

[0019] Leveraging the discrete nature of text semantics, zero-width characters, as discrete symbols without visual traces, do not participate in semantic composition. This not only avoids destroying the original semantics of the sample but also bypasses common text processing tools. Together with the semantic triggers of keyword reconstruction, they form a protective mechanism, strengthening the association between the watermark and the model.

[0020] Furthermore, the fixed proportion in step S1 is 5%-30%.

[0021] To further implicitly and robustly encode watermark information into the parameters of the main neural network classification model during the model training phase, this invention, based on the above scheme, constructs a pluggable watermark-aware attention guidance module before classification model training. This module is used to guide the classification model to implicitly encode watermark information into the model during training by dynamically generating attention weights. The module is configured to selectively access a first position and / or a second position of the classification model during training; wherein the first position is after the word embedding layer and before the backbone feature extraction layer of the classification model; the second position is after the output of the backbone feature extraction layer and before the classifier layer of the classification model; and it is removed from the classification model after training is completed.

[0022] Furthermore, the watermark perception attention guidance module in this invention includes a query projection unit, a watermark key unit, a watermark value unit, an attention calculation unit, and a feature modulation unit;

[0023] The query projection unit is used to receive the original feature matrix generated after a single input text sample undergoes word vector transformation at the Embedding layer of the classification model. Where L is the length of the text sequence. The input feature dimension is used, and the original feature matrix H is subjected to dimensionality reduction and focusing processing to output a query projection. ,in For the dimensions of the watermark query space, and < ;

[0024] For an input text sample, the model first segments it into tokens and converts them into a sequence of tokens. Each token is mapped to a fixed-dimensional word vector through the embedding layer; this dimension is set automatically in the model's parameters. Therefore, for a text of length L, after passing through the embedding layer, a word vector of shape (L, D) will be obtained. in The matrix H is the feature representation matrix of the sample in the current layer. Each row corresponds to the feature vector of a word. All L rows together constitute the serialized feature representation of the sample.

[0025] The watermark key unit stores the watermark prototype vector by maintaining a learnable watermark key matrix K; wherein, the watermark key matrix... ,in This represents the number of watermark keys, where k is an abbreviation for key; Represents a single key vector with dimension . There are a total of Each watermark key vector The corresponding watermark prototype vector in the watermark query space, where i is a positive integer and i=1,2… The watermark key matrix is ​​initialized using a Xavier normal distribution.

[0026] The attention calculation unit is used to receive the query projection output by the query projection unit. And the watermark key matrix K maintained by the watermark key unit, evaluate the query projection. The degree of matching with the watermark key is calculated and normalized to generate the attention weight matrix A. In the attention calculation, the dimensions of the query vector and the key vector must be consistent to ensure that the dot product similarity score of the two can be calculated.

[0027] The watermark value unit is used to maintain a learnable watermark value matrix. Each vector in the matrix Indicates the watermark key The corresponding modulation mode of the encoding; the watermark value matrix is ​​initialized by a truncated normal distribution.

[0028] The feature modulation unit receives the attention weight matrix output by the attention calculation unit and the watermark value matrix maintained by the watermark value unit, performs a weighted summation of the watermark value matrix according to the attention weight, generates a feature modulation amount, and fuses the modulation amount with the original feature through a learnable gating vector.

[0029] Furthermore, the dimensionality reduction and focusing processing in the query projection unit is implemented through a learnable linear transformation, specifically including:

[0030] The original feature matrix H is multiplied by the projection weight matrix. A linear transformation is performed to generate the query projection, and the transformation relationship is expressed as follows: Wherein, the projection weight matrix In the model training initialization phase, the Xavier uniform distribution is used for parameter initialization.

[0031] Furthermore, the attention calculation unit generates the attention weight matrix based on a scaled dot product attention mechanism, specifically as follows:

[0032] Get query projection The similarity value between each position in the code and the watermark key:

[0033] ,

[0034] in, This is a similarity matrix. The expression query projection Q in the first The row vector of the row, where =1,2,3…L;

[0035] Its purpose is to stabilize training and prevent gradient vanishing or values ​​from going out of range;

[0036] Performing SoftMax normalization on the similarity matrix yields the normalized attention weight matrix:

[0037] , and All represent similarity values. In this process, for ordinary text samples, the similarity between their features and all keys is low and evenly distributed, and A tends to be evenly distributed. If the input sample contains a watermark trigger, when the model processes the sequence position corresponding to the trigger word, its attention distribution will significantly focus on the specific key that matches the trigger, resulting in the attention weight of that position being much higher than other positions.

[0038] Furthermore, in the feature modulation unit, the attention weight matrix is ​​first averaged along the sequence dimension and then multiplied with the watermark value matrix V to obtain the modulation signal M:

[0039] ,

[0040] in, Represents the matrix A's first... All columns of a row are represented as a 1× The row vector; the dimension of the modulated signal M can be fixed as follows using the above method. This is independent of the input text length L, in order to solve the problem of inconsistent text lengths;

[0041] Then through learnable gating vectors The modulation signal vector M is subjected to channel adaptive weighting, wherein the gated vector is activated by the Sigmoid function to obtain the openness weights. The weighted modulation signal is added to the original feature matrix H with an adjustable intensity to obtain the output features. ,in This is the modulation intensity coefficient, with an initial value of 0.1.

[0042] Based on the same inventive concept, the present invention also discloses a computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the steps of the above-described method.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1) The semantic-level trigger constructed in this invention retains the core discrete semantic features of the original sample and performs semantic reconstruction from different levels such as the core description layer, category instruction layer and format constraint layer. It can maintain the integrity of the semantic skeleton of the original sample and not lose key information. It solves the problem of semantic incoherence and topic deviation after embedding watermarks in traditional trigger sets, thereby avoiding easy detection or differentiation.

[0045] 2) The semantic-level trigger constructed by this invention has undergone multiple iterations and strict parameter constraints, resulting in a high-quality trigger set text that can achieve a unified style, reasonable length, and theme alignment, making the embedded watermark more stable.

[0046] 3) This invention strengthens the association between the watermark and the model by simultaneously constructing semantic-level and character-level dual-trigger samples and providing dual protection.

[0047] 4) The watermark-aware attention guidance module designed in this invention independently classifies the main body of the model. Through adaptive parameter configuration, this scheme can automatically adapt to the layer structure, size and task type of different models, ensuring the consistency and effectiveness of watermark embedding.

[0048] 5) This invention guides the watermark to be embedded into the key feature layer or attention region of the model through an attention mechanism, so that the watermark is associated with the core decision logic of the model, which enhances the stickiness of the watermark. Even after the model has undergone attacks such as pruning and fine-tuning, the watermark information can still be retained in the important path, thereby achieving a high continuous detection rate.

[0049] 6) This invention minimizes the interference with the performance of the model's main classification task by constraining the joint optimization of the watermark loss function and the main classification task loss function, ensuring that watermark embedding only causes fine-tuning of model parameters without impairing its original accuracy or generalization ability. Attached Figure Description

[0050] Figure 1 This is a flowchart of the black-box watermark embedding method based on a neural network text classification model according to the present invention;

[0051] Figure 2 This is a flowchart of the method for constructing a watermark trigger set according to the present invention;

[0052] Figure 3 This is a schematic diagram of the watermark perception attention guidance module structure of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the specific embodiments of the present invention clearer, the technical solutions of the specific embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described specific embodiments are only a part of the specific embodiments of the present invention, not all of them. Based on the described specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] This invention is applied to neural network text classification models. Considering that text data is different from continuous data such as images and audio, its core feature is semantic discreteness. It takes characters and words as basic building blocks, and its semantics depends on the combination logic of keywords and contextual association, rather than the smooth transition of continuous features. This uniqueness makes it easy to destroy semantic discreteness and structure by direct perturbation or simple splicing.

[0055] like Figure 1 As shown, this invention discloses a black-box watermark embedding method for neural network text classification models, comprising the following steps:

[0056] Step S1: Construct a watermark trigger set, which includes first selecting a fixed proportion of samples from an original category in the original training set; then performing semantic reconstruction processing on the selected samples and embedding predefined special character sequences to construct trigger samples that contain both semantic and character-level triggers; finally, assigning a unified new category label to all processed trigger samples, which is different from all original categories.

[0057] like Figure 2 As shown, the process of constructing a watermark trigger set includes three parts:

[0058] Part One:

[0059] 1) First, use natural language processing tools to extract key semantic features of the selected samples. The key semantic features include at least one of core entities, action features, and topic classification.

[0060] 2) Based on the extracted key semantic features, a structured generation prompt is constructed. The structured generation prompt includes a core description layer for defining the core topic, a category instruction layer for specifying the text category, and a format constraint layer for constraining the text format or style. Then, using the flan-T5-base pre-trained text generation model, under the constraints of preset generation parameters, a new text sample with semantic preservation and format reconstruction is generated. The preset generation parameters include at least one of the following: main entity occurrence degree, category keyword matching degree, and action relevance.

[0061] 3) Perform quality checks on the generated new text samples. The quality checks include an automated evaluation of at least one of the following: text integrity, core entity matching degree, and irrelevant content exclusion degree. Add the samples that meet the quality requirements to the watermark trigger set, and repeat steps S11 to S12 until the number of samples in the watermark trigger set reaches the preset requirement. The preset requirement for the number of samples in this invention is to meet the fixed ratio in step S1.

[0062] Part Two: Based on the construction of semantic-level trigger samples, a predefined special character sequence is embedded to form a watermark trigger set that simultaneously contains both semantic-level and character-level triggers; wherein the special character sequence consists of at least 8 zero-width invisible characters, and in this embodiment, the format of zero-width space ZWSP and Unicode character U+200B is adopted.

[0063] Part 3: Assign a uniform new category label to all processed trigger samples, which is different from all the original categories.

[0064] Step S2: Randomly mix the watermark trigger set constructed in step S1 with ordinary training samples to form a mixed dataset. The ordinary training samples are those samples in the original training set that were not selected into the watermark trigger set.

[0065] Step S3: Train the classification model based on the mixed dataset obtained in step S2, so that the parameters of the trained classification model are encoded with watermark information. The optimization objective of the training is to minimize the overall loss function. Represented as: Where λ is a preset weighting coefficient used to adjust the watermark embedding strength. If the watermark detection rate is too low, it can be appropriately increased according to the experimental results, and can be considered to be 1.2-3.0. The value used in this embodiment is 2; This represents the loss calculated based on the prediction results of the classification model on the ordinary training samples and their original class labels, used to maintain the performance of the classification model in the main text classification task; the table The loss calculated based on the prediction results of the watermark trigger set samples by the classification model and the unified new category label is used to induce the model to learn watermark behavior.

[0066] To further implicitly and robustly encode watermark information into the parameters of the main neural network classification model during the model training phase, a pluggable watermark-aware attention guidance module was constructed before the classification model training. This module is used to guide the classification model to encode watermark information into the classification model during training by dynamically generating attention weights. The module is configured to be accessed after the word embedding layer of the classification model, before the backbone feature extraction layer, after the output of the backbone feature extraction layer, and before the classifier layer during the classification model training, and removed from the classification model after training is completed.

[0067] like Figure 3 As shown, the watermark perception attention guidance module in this invention includes a query projection unit, a watermark key unit, a watermark value unit, an attention calculation unit, and a feature modulation unit.

[0068] The query projection unit is used to receive the original feature matrix generated after a single input text sample undergoes word vector transformation at the Embedding layer of the classification model. Where L is the length of the text sequence. To determine the input feature dimension, in this embodiment, the input feature dimension is 200. The original feature matrix H is then subjected to dimensionality reduction and focusing processing to output a query projection. ,in, The dimension of the watermark query space is 64 in this embodiment, and... < ;

[0069] The original feature matrix H is multiplied by the projection weight matrix. A linear transformation is performed to generate the query projection, and the transformation relationship is expressed as follows: Wherein, the projection weight matrix In the model training initialization phase, the Xavier uniform distribution is used for parameter initialization.

[0070] For an input text sample, the model first segments it into tokens and converts them into a sequence of tokens. Each token is mapped to a fixed-dimensional word vector through the embedding layer; this dimension is set automatically in the model's parameters. Therefore, for a text of length L, after passing through the embedding layer, a word vector of shape (L, D) will be obtained. in The matrix H is the feature representation matrix of the sample in the current layer. Each row corresponds to the feature vector of a word. All L rows together constitute the serialized feature representation of the sample.

[0071] The watermark key unit stores the watermark prototype vector by maintaining a learnable watermark key matrix K; wherein, the watermark key matrix... ,in The number of watermark keys, in this embodiment The value is 2, for each watermark key vector. This corresponds to a watermark prototype vector in a watermark query space, where i is a positive integer and i=1,2;

[0072] The attention calculation unit is used to receive the query projection output by the query projection unit. And the watermark key matrix K maintained by the watermark key unit, evaluate the query projection. The degree of matching with the watermark key is determined and normalized to generate an attention weight matrix A; the attention calculation unit generates the attention weight matrix based on a scaled dot product attention mechanism, specifically:

[0073] Get query projection The similarity value between each position in the code and the watermark key:

[0074]

[0075] in, This is a similarity matrix. Description query projection The Middle The row vector of the row, where =1,2,3…L;

[0076] Its purpose is to stabilize training and prevent gradient vanishing or values ​​from going out of range;

[0077] Performing SoftMax normalization on the similarity matrix yields the normalized attention weight matrix:

[0078] , and All represent similarity values, indicating the first... The similarity value between a sample and the i / j-th key is calculated. In this process, for ordinary text samples, the similarity between their features and all keys is low and evenly distributed, and A tends to be evenly distributed. If the input sample contains a watermark trigger, when the model processes the sequence position corresponding to the trigger word, its attention distribution will significantly focus on the specific key that matches the trigger, resulting in the attention weight of that position being much higher than other positions.

[0079] The watermark value unit is used to maintain a learnable watermark value matrix. Each vector in the matrix Indicates the watermark key The corresponding modulation mode of the encoding; the watermark value matrix is ​​initialized as a truncated normal distribution, with a range of [-0.02, 0.02].

[0080] The feature modulation unit first performs average aggregation along the sequence dimension using the attention weight matrix, and then multiplies it with the watermark value matrix V to obtain the modulation signal M:

[0081] The dimension of the modulated signal M can be fixed in the above way. This is independent of the input text length L, in order to solve the problem of inconsistent text lengths;

[0082] Then through learnable gating vectors The modulated signal M is subjected to channel adaptive weighting, wherein the gate vector is activated by the Sigmoid function to obtain the openness weight. The weighted modulation signal is added to the original feature matrix H with an adjustable intensity to obtain the output features. ,in This is the modulation intensity coefficient, with an initial value of 0.1.

[0083] Based on the above implementation process, this invention provides an embodiment. This embodiment takes the AGNews and SST-2 datasets as examples and uses three mainstream text classification models, TextCNN, BiLSTM and Transformer, for watermark embedding and verification. It should be noted that the method of this invention is not limited to applying the above three text classification models.

[0084] The AGNews dataset contains 120,000 training samples and 7,600 test samples, divided into four categories: world, sports, business, and technology. The SST-2 dataset contains 6,920 training samples, 872 validation samples, and 1,821 test samples, divided into positive and negative sentiment categories.

[0085] The programming language was Python, the training equipment was a 13th Gen Intel(R) Core(TM) i7-13620H CPU, 16GB RAM and a GeForce RTX 4060 GPU, the optimizer was Adam optimizer, the initial learning rate was 0.001, the batch size was set to 128, and the training epochs were 30.

[0086] In this embodiment, 10% of the samples are selected as the trigger set. When constructing the semantic trigger, in terms of the occurrence of main entities, the main entities are constrained to appear at least once and at most three times. In terms of category keyword matching, 85% of the core noun phrases in the original text are constrained to appear in the generated text as the same word, synonym, or hypernym. A multi-level matching strategy is adopted, including full string matching using regular expressions and word vector similarity matching using spaCy, with a threshold of 0.65. In terms of action relevance, at least 80% of the main action verbs are constrained to appear as the same word, synonym, or hypernym. The main action verbs are extracted through spaCy syntactic analysis, and the matching verification is performed by combining word vector similarity and predefined synonym groups.

[0087] In quality inspection, text integrity inspection requires that the text must have a clear topic sentence and supporting sentences, and the information entropy estimate should be maintained within a reasonable range of 0.3-0.8.

[0088] Core entity matching: At least 85% of core noun phrases must appear as identical words, synonyms, or hypernyms. Irrelevant content exclusion: The ratio of irrelevant keywords to the total number of words in a single sentence must be less than 5%.

[0089] The following section verifies the effectiveness of the watermark trigger set constructed in this invention in terms of fidelity, validity, robustness, and concealment:

[0090] Regarding fidelity and effectiveness, as shown in Table 1, taking the SST-2 dataset as an example, the watermark recognition rate reaches 100% while maintaining a model classification accuracy decrease of no more than 1%, demonstrating the ability to verify watermarks. Furthermore, since clean models without embedded watermarks do not contain the labels corresponding to the trigger set, the false positive rate for clean models is 0%.

[0091] Table 1 Fidelity and Watermark Effectiveness

[0092] Model Clean model accuracy Clean model watermark false positive rate Watermark model accuracy Trigger set detection rate TextCNN 91.37% 0% 91.31% 100% BiLSTM 91.85% 0% 91.56% 100% Transformer 89.96% 0% 89.10% 100%

[0093] Regarding robustness, as shown in Table 2, taking the SST-2 dataset as an example, the watermark performs well against various attacks. In terms of model fine-tuning attacks, samples with the same text feature distribution as the original training set and which neither participate in model training nor trigger set construction are selected for model fine-tuning. The number of samples in the fine-tuning dataset is 30% of the model training set, the training rounds are 30, and the learning rate is 0.0001. After the model fine-tuning attack, the watermark detection rate is still 100%. After a random pruning attack with a 70% pruning rate, the watermark success rate still remains above 80%.

[0094] Table 2 Robustness of Watermarks

[0095] Model Trigger set detection rate Trigger set detection rate after fine-tuning attack Trigger set detection rate after pruning attack TextCNN 98.65% 100% 83.04% BiLSTM 99.98% 100% 85.08% Transformer 100% 100% 91.56%

[0096] The following are some of the mainstream watermark construction methods in the existing technology:

[0097] RIPPLe method: Constructs a trigger set by inserting some unrelated rare words into the text. In the experiment, only the trigger set generation method in this paper was used.

[0098] The frequency-inverse document frequency (TF-IDF) method utilizes the concepts of term frequency (TF) and inverse document frequency (IDF) in text processing, selects specific keywords as triggers, and embeds black-box watermarks in a natural language processing model.

[0099] The LTS method constructs a trigger set by inserting specific neutral statements. After training, the model will force the text containing the trigger statement to be classified into a preset target category.

[0100] The TADW method enhances the concealment and security of watermarks by dynamically generating watermark keys, making it difficult for attackers to detect and remove the watermarks.

[0101] The SOS method first generates a trigger set by inserting specific neutral statements, and then performs negative sample augmentation to reduce the false alarm rate.

[0102] The SecNLP method creates a trigger set by adding a specific password based on hash encoding to the beginning of a sentence in a given sample, and balances model performance with watermark robustness by jointly optimizing the watermark embedding task and the main task through multi-task learning.

[0103] The trigger set constructed in this invention demonstrates significant advantages in terms of concealment compared to existing solutions, as shown in Table 3.

[0104] Table 3. Automatic quality assessment results of samples with different triggers embedded.

[0105] Automatic metric RIPPLe TF-IDF LTS SOS SecNLP Invention PPL 215.86 219.75 194.20 191.36 171.63 158.05 GEM 2.38 2.46 2.23 2.09 2.46 1.96

[0106] As shown in Table 3, two automatic metrics were used for validation: perplexity (PPL) and grammatical error model (GEM). In language models, perplexity is an important metric for measuring the model's ability to predict a set of texts. It measures the semantic coherence and rationality of the text. The lower the value, the more fluent and logical the text is. The higher the value, the more confusing and difficult to understand the text is.

[0107] The syntax error model (GEM) value is used to quantify grammatical problems in text. A lower value indicates more standard grammar and fewer errors, while a higher value indicates more grammatical deviations or inappropriate expressions. The GPT-2 language model is used to calculate the perplexity of the trigger set, and the Language Tool syntax error model is used to calculate the GEM value. The SST-2 dataset is used for testing. From the training set of the SST-2 dataset, 200 normal samples are randomly selected, and watermarks are embedded in 40 of these samples according to different watermarking frameworks.

[0108] Regarding PPL, the present invention exhibits the lowest perplexity at only 158.05, meaning that our scheme is the least likely to be detected when facing a detector that measures the perplexity of statements. Regarding GEM, the trigger set in the present invention has an average GEM of 1.96, resulting in the fewest detected syntax errors. These two indicators demonstrate that the trigger set of the present invention possesses strong concealment.

[0109] As shown in Table 4, the present invention demonstrates significant performance advantages when comparing watermarking verification performance with and without backdoor defenses.

[0110] Table 4. Comparison of watermark verification performance of different watermarking schemes with and without backdoor defenses.

[0111] Scheme RIPPLe TF-IDF LTS TADW SOS SecNLP Invention Watermark detection rate without defense 100% 100% 100% 90.03% 100% 100% 100% Watermark detection rate after defense 17.04% 80.05% 61.02% 69.92% 55.06% 95.83% 98.24% Main task accuracy without defense 70.98% 71.21% 70.04% 69.58% 68.76% 74.35% 73.09% Main task accuracy after defense 69.04% 69.84% 68.05% 66.72% 66.24% 71.33% 70.98%

[0112] Table 4 shows the resistance of the watermarking framework to backdoor defenses. The ONION anomaly detector is used to detect trigger sets. The main idea of ​​ONION is to use a language model to detect and eliminate anomalous words in test samples. If deleting a word from a sample significantly reduces the semantic perplexity of that sample, then that word is likely part of a backdoor trigger. Once the existence of such a trigger is detected, ONION will directly delete it to prevent it from activating the model's backdoor. The test dataset is SST-2. As can be seen from Table 4, after the backdoor defense of the anomaly detector, our scheme's watermark detection rate only decreases by 1.76%, and watermarks can still be detected. Moreover, the decrease in watermark detection rate in our scheme is less than that of other baseline models, demonstrating better performance.

[0113] It should be understood that the present invention is implemented through any of the various concepts and specific embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and specific embodiments disclosed in the present invention are not limited to any particular embodiment. Furthermore, some aspects of the present invention can be used alone or in any suitable combination with other aspects disclosed in the present invention.

Claims

1. A black-box watermark embedding method for neural network text classification models, characterized in that, Includes the following steps: Step S1: Construct the watermark trigger set, which includes first selecting from an original category in the original training set. A fixed proportion of samples are selected; then, semantic reconstruction processing is performed on the selected samples and predefined special character sequences are embedded to construct trigger samples that contain both semantic-level and character-level triggers; finally, a unified new category label is assigned to all processed trigger samples. The semantic reconstruction process specifically includes the following steps: Step S11: Extract the key semantic features of the selected samples. The key semantic features include at least one of core entities, action features, and topic classification. Step S12: First, construct a structured generation prompt based on the extracted key semantic features. The structured generation prompt includes a core description layer for defining the core topic, a category instruction layer for specifying the text category, and a format constraint layer for constraining the text format or style. Then, use the flan-T5-base pre-trained text generation model to generate new text samples that preserve semantics and reconstruct the format under the constraints of preset generation parameters. The preset generation parameters include at least one of the following: main entity occurrence degree, category keyword matching degree, and action relevance. Step S13: Perform quality detection on the generated new text samples. The quality detection includes an automated evaluation of at least one of the following: text integrity, core entity matching degree, and irrelevant content exclusion degree. Add the samples that meet the quality requirements to the watermark trigger set, and repeat steps S11 to S12 until the number of samples in the watermark trigger set reaches the preset requirement. Step S2: Randomly mix the watermark trigger set constructed in step S1 with ordinary training samples to form a mixed dataset. The ordinary training samples are those samples in the original training set that were not selected into the watermark trigger set. Step S3: Train the classification model based on the mixed dataset obtained in step S2, so that the trained classification model encodes watermark information. The optimization objective of the training is to minimize the overall loss function. The overall loss function Represented as: Where λ is a preset weighting coefficient and is a positive real number, the This represents the loss calculated based on the prediction results of the classification model for the ordinary training samples and their original class labels; This represents the loss calculated based on the prediction results of the watermark-triggered samples in the classification model and the unified new category label.

2. The black-box watermark embedding method for neural network text classification models according to claim 1, characterized in that, The predefined special character sequence in step S1 consists of at least 8 zero-width invisible characters.

3. The black-box watermark embedding method for neural network text classification models according to claim 1, characterized in that, The fixed ratio in step S1 is 5%-30%.

4. The black-box watermark embedding method for neural network text classification models according to claim 1, characterized in that, Before training the classification model, a pluggable watermark-aware attention guidance module is constructed to guide the classification model to implicitly encode watermark information into the classification model during training. The module is configured to selectively access a first position and / or a second position of the classification model during training. The first position is after the word embedding layer and before the backbone feature extraction layer of the classification model. The second position is after the output of the backbone feature extraction layer and before the classifier layer of the classification model. The module is removed from the classification model after training is completed.

5. The black-box watermark embedding method for neural network text classification models according to claim 4, characterized in that, The watermark perception attention guidance module includes a query projection unit, a watermark key unit, an attention calculation unit, a watermark value unit, and a feature modulation unit. The query projection unit is used to receive the original feature matrix generated after a single input text sample undergoes word vector transformation at the Embedding layer of the classification model. Where L is the length of the text sequence. The input feature dimension is used, and the original feature matrix H is subjected to dimensionality reduction and focusing processing to output a query projection. ,in For the dimensions of the watermark query space, and < ; The watermark key unit stores the watermark prototype vector by maintaining a learnable watermark key matrix K; wherein, the watermark key matrix... ,in The number of watermark keys. The dimension of the key vector is consistent with the dimension of the watermark query space. Each watermark key vector... The corresponding watermark prototype vector in the watermark query space, where i is a positive integer and i=1,2… ; The attention calculation unit is used to receive the query projection output by the query projection unit. And the watermark key matrix K maintained by the watermark key unit, evaluate the query projection. The degree of matching with the watermark key is determined and normalized to generate the attention weight matrix A; The watermark value unit is used to maintain a learnable watermark value matrix. Each vector in the matrix Indicates the watermark key The corresponding modulation mode of the encoding; The feature modulation unit is used to receive the attention weight matrix output by the attention calculation unit and the watermark value matrix maintained by the watermark value unit, perform weighted summation on the watermark value matrix according to the attention weight, generate a feature modulation amount, and fuse the modulation amount with the original feature through a learnable gating vector.

6. The black-box watermark embedding method for neural network text classification models according to claim 5, characterized in that, The dimensionality reduction and focusing processes in the query projection unit are implemented through a learnable linear transformation, specifically including: The original feature matrix H is multiplied by the projection weight matrix. A linear transformation is performed to generate the query projection, and the transformation relationship is expressed as follows: , Wherein, the projection weight matrix In the model training initialization phase, the Xavier uniform distribution is used for parameter initialization.

7. The black-box watermark embedding method for neural network text classification models according to claim 5, characterized in that, The attention calculation unit generates the attention weight matrix based on the scaled dot product attention mechanism, specifically: Get query projection The similarity value between each position in the code and the watermark key: ,in, This is a similarity matrix. The expression query projection Q in the first The row vector of the row, where =1,2,3…L; Perform SoftMax normalization on the similarity matrix to obtain the normalized attention weight matrix: ,in and All represent similarity values, where j is a positive integer and j=1,2… .

8. The black-box watermark embedding method for neural network text classification models according to claim 5, characterized in that, In the feature modulation unit, the attention weight matrix is ​​first averaged along the sequence dimension and then multiplied with the watermark value matrix V to obtain the modulation signal. : , Then through learnable gating vectors The modulated signal M is subjected to channel adaptive weighting, wherein the gate vector is activated by the Sigmoid function to obtain the openness weights. The weighted modulation signal is added to the original feature matrix H with an adjustable intensity to obtain the output features. ,in This is the modulation intensity coefficient, with an initial value of 0.

1.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

Images