Black box watermark embedding method for neural network text classification model

Abstract

The invention discloses a black-box watermark embedding method for a neural network text classification model, and the method comprises the steps: firstly constructing a watermark trigger set containing a semantic-level trigger and a character-level trigger at the same time, and then randomly mixing the constructed watermark trigger set with a common training sample to form a mixed data set; the classification model is trained based on the mixed data set, so that watermark information is encoded in the trained classification model, and the optimization target of training is to minimize an overall loss function; the method can effectively solve the problem that a trigger set sample destroys semantic discreteness and structure of a text, and meanwhile, due to the design of the double triggers, the intensity of a watermark signal is improved, it is ensured that when the trigger set is subjected to preprocessing operation such as text cleaning and the like and one type of triggers are missing, the text can still retain the other type of triggers, and the use experience of the user is improved. And through embedding at different levels, the robustness of the watermark for resisting attacks is obviously enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence model security, more specifically, a black box watermark embedding method based on a neural network text classification model. BACKGROUND

[0002] With the rapid development of natural language processing technology, text classification models based on deep neural networks have been widely used in sentiment analysis, content review, spam filtering and many other commercial scenarios. The training of such models requires a large amount of high-quality labeled data, computing resources and expert knowledge, which constitutes an important core digital asset for enterprises. However, once the model is deployed as a black box service, it faces the risk of being illegally copied, stolen or distributed without authorization. Since the internal parameters of the model cannot be directly probed, it is difficult for the model owner to provide effective technical evidence for identification and tracing when infringement occurs, which seriously hinders the healthy transaction and ecological development of AI technology.

[0003] To solve the above problems, model watermarking technology has emerged, which embeds a hidden mark during model training or inference, so that the model can produce a pre-set, unique mark output when receiving a specific trigger input, thereby providing verifiable evidence for model ownership.

[0004] Current black box watermarking methods for text classification models face several core challenges, the root cause of which lies in the inherent contradiction between the discrete nature of text data, semantic integrity and the black box access restrictions of the model:

[0005] 1) The trigger samples generated by the current mainstream black box watermarking methods generally have the problem of unnatural semantics and are easily identified by statistical anomaly detection mechanisms. At the same time, when faced with common operations such as model fine-tuning and pruning, the robustness of the watermark is often insufficient;

[0006] 2) Methods that directly embed watermarks by simply concatenating or perturbing key words, although intuitive, can easily destroy the semantic coherence and logical integrity of the text. As a result, the generated trigger samples have poor concealment and often cause a significant decline in the performance of the model's main task;

[0007] 3) Continuous perturbation watermarking methods designed for continuous data such as images are directly migrated to the text domain, ignoring the discrete nature of text. This hard transplantation destroys the semantic discreteness and structure of the text, resulting in illogical trigger texts that can severely damage the performance of the model's main task. SUMMARY

[0008] The present application aims to solve the problems of destroying the semantics and structure of the text due to the construction of the trigger set samples, and the lack of watermark concealment and robustness in the prior art.

[0009] To achieve the above object, the application discloses a black box watermark embedding method for a neural network text classification model, which generates trigger samples with complete structure by constructing double triggers of semantic level and character level at the same time, significantly enhances the concealment and attack robustness of the watermark on the basis of maintaining the natural semantic discreteness and structure of the text, and specifically includes the following steps:

[0010] Step S1, constructing a watermark trigger set, including: first, selecting a fixed proportion of samples from an original class of an original training set; second, performing semantic reconstruction processing on the selected samples and embedding a predefined special character sequence to construct trigger samples containing semantic level and character level triggers; and third, assigning a uniform new class label to all processed trigger samples, which is different from all original class labels.

[0011] Step S2, randomly mixing the watermark trigger set constructed in step S1 with ordinary training samples to form a mixed data set, wherein the ordinary training samples are samples in the original training set that are not selected into the watermark trigger set.

[0012] Step S3, training the classification model based on the mixed data set obtained in step S2, so that the trained classification model parameters encode watermark information, wherein the optimization goal of the training is to minimize the overall loss function ; the overall loss function is expressed as: wherein λ is a preset weight coefficient for adjusting the strength of watermark embedding, which can be appropriately increased according to experimental results if the watermark detection rate is too low; the represents the loss calculated based on the prediction result of the classification model for the ordinary training samples and their original class labels, which is used to maintain the text classification main task performance of the classification model; and the represents the loss calculated based on the prediction result of the classification model for the samples in the watermark trigger set and the uniform new class label, which is used to induce the model to learn the watermark behavior.

[0013] Further, on the premise of preserving core semantic features, the semantics are reconstructed, which includes the following steps:

[0014] Step S11, extracting key semantic features of the selected samples using a natural language processing tool, wherein the key semantic features include at least one of core entities, action features, and topic classifications;

[0015] In step S12, a structured generation prompt is constructed based on the extracted key semantic features, the structured generation prompt including a core description layer for defining a core topic, a category instruction layer for specifying a text category, and a format constraint layer for constraining a text format or style; and a flan-T5-base pre-trained text generation model is used to generate a new text sample with semantic preservation and format reconstruction under preset generation parameter constraints, the preset generation parameters including at least one of a main entity occurrence degree, a category keyword matching degree, and an action relevance.

[0016] In step S13, quality detection is performed on the generated new text sample, the quality detection including automatic evaluation of at least one of text integrity, core entity matching degree, and irrelevant content exclusion degree; samples meeting the quality requirements are added to a watermark trigger set, and steps S11 to S12 are repeated until the number of samples in the watermark trigger set reaches a preset requirement.

[0017] The process solves the problem of semantic inconsistency and topic deviation of a traditional trigger set after embedding a watermark, provides a high-quality and high-relevance trigger sample for subsequent watermark detection, thereby improving the reliability of watermark verification, retains the core discrete semantic features of the original sample, and can be integrated into the model training process to avoid being judged as noise by the model; meanwhile, through multiple iterations and strict parameter restrictions, the generated trigger set text has high quality, can achieve style uniformity, reasonable length, and topic alignment, and makes the embedded watermark more stable.

[0018] Further, the special character sequence inserted at the beginning or middle of the trigger set sample in step S1 to enhance trigger stability is at least 8 zero-width invisible characters, uses zero-width space ZWSP, and has a format of Unicode character U+200B.

[0019] With the characteristics of text semantic discreteness, the zero-width character as a discrete symbol without visual traces does not participate in semantic composition, does not damage the original semantics of the sample, and can avoid common text processing tools, and together with the key word reconstructed semantic trigger, forms a protection to strengthen the association strength of the watermark and the model.

[0020] Further, the fixed proportion in step S1 is 5%-30%.

[0021] In order to further encode the watermark information more implicitly and robustly into the parameters of the main neural network classification model in the model training stage, the application, on the basis of the above scheme, before the training of the classification model, a pluggable watermark-aware attention guiding module is constructed, which is used to guide the classification model to implicitly encode the watermark information into the model during training by dynamically generating attention weights, the module is configured to be selectively accessed to the first position and / or the second position of the classification model during the training of the classification model; wherein the first position is after the word embedding layer of the classification model and before the backbone feature extraction layer; the second position is after the output of the backbone feature extraction layer of the classification model and before the classifier layer; and the module is removed from the classification model after the training is completed.

[0022] Further, the watermark-aware attention guiding module in the application includes a query projection unit, a watermark key unit, a watermark value unit, an attention calculation unit and a feature modulation unit.

[0023] The query projection unit is used to receive the original feature matrix generated after a single input text sample is converted into a word vector by the Embedding layer of the classification model , wherein L is the length of the text sequence, is the input feature dimension, and the original feature matrix H is reduced and focused to output a query projection , wherein is the dimension of the watermark query space, and

[0024] For an input text sample, the model will first perform word segmentation and convert it into a Token sequence; each Token is mapped to a fixed-dimensional word vector by the Embedding layer, and the dimension is set in the parameters of the model; therefore, for a text of length L, after the Embedding layer, a matrix H of shape (L, D in ) is obtained; the matrix H is the feature representation matrix of the sample at the current layer, wherein each row corresponds to a feature vector of a Token, and all L rows collectively constitute the serialized feature representation of the sample.

[0025] The watermark key unit stores the watermark prototype vector by maintaining a learnable watermark key matrix K; wherein the watermark key matrix , wherein is the number of watermark keys, and k here is the abbreviation of key, indicating the key; represents a single key vector, with a dimension of , and there are a total of , and each watermark key vector ​​Corresponding to a watermark prototype vector in a watermark query space, i is a positive integer and i = 1, 2… ; the initialization mode of the watermark key matrix is Xavier normal distribution initialization;

[0026] The attention calculation unit is used for receiving the query projection output by the query projection unit and the watermark key matrix K maintained by the watermark key unit, evaluating the matching degree of the query projection with the watermark key, and generating an attention weight matrix A after normalization; in attention calculation, the dimensions of the query vector and the key vector must be consistent to ensure that the dot product similarity score of the two can be calculated.

[0027] The watermark value unit is used for maintaining a learnable watermark value matrix , wherein each vector in the matrix represents a corresponding modulation mode for the watermark key encoding; the initialization mode of the watermark value matrix is truncated normal distribution.

[0028] The feature modulation unit receives the attention weight matrix output by the attention calculation unit and the watermark value matrix maintained by the watermark value unit, and performs weighted summation on the watermark value matrix according to the attention weight to generate a feature modulation quantity, and fuses the modulation quantity with the original feature through a learnable gating vector.

[0029] Further, the dimension reduction and focusing processing in the query projection unit are implemented through a learnable linear transformation, specifically including:

[0030] The original feature matrix H is linearly transformed through matrix multiplication with the projection weight matrix to generate the query projection, and the transformation relationship is represented as: , wherein the projection weight matrix is initialized by using Xavier uniform distribution for parameter initialization in the model training initialization stage.

[0031] Further, the attention calculation unit generates the attention weight matrix based on the scaled dot-product attention mechanism, specifically:

[0032] The similarity value between each position in the query projection and the watermark key is obtained:

[0033] ,

[0034] , wherein is a similarity matrix, , and the row vector of the i-th row in the query projection Q is represented as , wherein =1,2,3…L;

[0035] The role is to stabilize training, prevent gradient disappearance or numerical overflow;

[0036] The similarity matrix is subjected to SoftMax normalization operation to obtain a normalized attention weight matrix:

[0037] , and all represent similarity values, in the process, for ordinary text samples, the similarity of the features with all keys is low and uniformly distributed, and A tends to be uniformly distributed; if the input sample contains a watermark trigger, the model will significantly focus on the specific key matching the trigger when processing the sequence position corresponding to the trigger word, resulting in that the attention weight of the position is much higher than that of other positions.

[0038] Further, the feature modulation unit first aggregates the attention weight matrix along the sequence dimension by averaging, and multiplies it with the watermark value matrix V to obtain the modulation signal M:

[0039] ,

[0040] wherein, represents all columns of the first row of matrix A, and is represented as a 1x row vector; by the above method, the dimension of the modulation signal M can be fixed to , which is independent of the length L of the input text, to solve the problem of inconsistent text length; Again, the channel self-adaptive weighting is performed on the modulation signal vector M by using a learnable gating vector

[0041] , wherein the gating vector is activated by a Sigmoid function to obtain an openness weight ; the weighted modulation signal is added to the original feature matrix H with adjustable intensity to obtain the output feature , wherein is a modulation intensity coefficient, and the initial value is 0.1. Based on the same inventive concept, the application further discloses a computer readable storage medium, which stores a computer program, and the computer program is characterized in that the computer program is executed by a processor to realize the steps of the above method.

[0042] Compared with the prior art, the application has the following beneficial effects:

[0043]

[0044] ​1) The semantic level trigger constructed by the present application can maintain the integrity of the semantic skeleton of the original sample, without losing key information, by reconstructing the semantics from different levels of core description layer, category instruction layer and format constraint layer, based on the core discrete semantic features of the original sample, solving the problem of semantic disorder and theme deviation of the traditional trigger set after embedding the watermark, thereby avoiding being easily detected or distinguished.

[0045] 2) The semantic level trigger constructed by the present application has high quality of generated trigger set text after multiple rounds of iteration and strict parameter restriction, which can realize style unification, reasonable length and theme alignment, making the embedding of watermark more stable.

[0046] 3) The present application can strengthen the association strength of watermark and model by constructing dual trigger samples of semantic level and character level at the same time through double protection.

[0047] 4) The watermark-aware attention guide module designed by the present application, the independent classification model main body, through adaptive parameter configuration, this scheme can automatically adapt to the layer structure, size and task type of different models, ensuring the consistency and effectiveness of watermark embedding.

[0048] 5) The present application guides the watermark to be embedded into the key feature layer or attention area of the model through the attention mechanism, so that the watermark is associated with the core decision logic of the model, and the adhesion of the watermark is enhanced. After the model undergoes pruning, fine-tuning and other attacks, the watermark information can still be preserved in the important path, thereby realizing a high sustained detection rate.

[0049] 6) The present application minimizes the interference with the performance of the model classification main task, and through the joint optimization of the watermark loss function and the classification main task loss function, it ensures that the watermark embedding only causes the fine-tuning of the model parameters, without damaging its original accuracy or generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is the flow chart of the black box watermark embedding method of the neural network text classification model based on the present application;

[0051] Figure 2 is the method flow chart of the watermark trigger set constructed by the present application;

[0052] Figure 3 is the structure diagram of the watermark-aware attention guide module of the present application. DETAILED DESCRIPTION

[0053] For the purposes of the present embodiments, technical solutions and advantages of the present application, the following will be combined with the drawings of the present application, and the technical solutions of the present embodiments will be clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0054] The present application is applied to a neural network text classification model. Considering that text data is different from continuous data such as images and audios, the core feature lies in semantic discreteness. The basic constituent unit is a character or a word. The semantic dependence is the combination logic of keywords and the context association, rather than the smooth transition of continuous features. This uniqueness makes it easy to destroy semantic discreteness and structure by direct disturbance or simple splicing.

[0055] As shown in Figure 1 The present application discloses a black box watermark embedding method for a neural network text classification model, comprising the following steps:

[0056] Step S1, constructing a watermark trigger set, including first selecting a fixed proportion of samples from an original class of an original training set; then performing semantic reconstruction processing on the selected samples and embedding a predefined special character sequence to construct trigger samples containing semantic level and character level triggers; finally, assigning a uniform new class label to all processed trigger samples, which is different from all original classes;

[0057] As shown in Figure 2 The process of constructing the watermark trigger set includes three parts:

[0058] First part:

[0059] 1) First, extract the key semantic features of the selected samples using natural language processing tools, wherein the key semantic features include at least one of core entities, action features, and topic classifications;

[0060] 2) Construct a structured generation prompt based on the extracted key semantic features, wherein the structured generation prompt includes a core description layer for defining a core topic, a class instruction layer for specifying a text category, and a format constraint layer for constraining the format or style of the text; then use a flan-T5-base pre-trained text generation model to generate a new text sample with preserved semantics and reconstructed format under the constraint of preset generation parameters, wherein the preset generation parameters include at least one of the appearance degree of main entities, the matching degree of class keywords, and the action relevance;

[0061] 3) quality detection is performed on the generated new text samples, the quality detection including automated evaluation of at least one of text integrity, core entity matching degree and irrelevant content exclusion degree; samples meeting quality requirements are added to the watermark trigger set, and steps S11 to S12 are repeated until the number of samples in the watermark trigger set reaches a preset requirement; the sample number preset requirement in the application is a fixed proportion that meets step S1;

[0062] The second part: on the basis of the semantic level trigger sample construction, a pre-defined special character sequence is embedded to form a watermark trigger set containing both semantic level and character level dual triggers; the special character sequence is at least 8 zero-width invisible characters, and in the embodiment, a zero-width space ZWSP, Unicode character U+200B format is used.

[0063] The third part: a unified new class label is assigned to all processed trigger samples, which is different from all original classes.

[0064] Step S2: randomly mix the watermark trigger set constructed in step S1 with ordinary training samples to form a mixed data set, the ordinary training samples being samples in the original training set that are not selected into the watermark trigger set.

[0065] Step S3: train the classification model based on the mixed data set obtained in step S2, so that the trained classification model parameters encode watermark information, wherein the optimization goal of the training is to minimize the overall loss function, and the overall loss function is expressed as: wherein λ is a preset weight coefficient for adjusting the strength of watermark embedding, if the watermark detection rate is too low, it can be appropriately increased according to the experimental effect, and 1.2-3.0 can be considered, and the value used in the embodiment is 2; the represents the loss calculated based on the prediction result of the classification model on the ordinary training samples and the original class label thereof, and is used to maintain the text classification main task performance of the classification model; the represents the loss calculated based on the prediction result of the classification model on the samples in the watermark trigger set and the unified new class label, and is used to induce the model to learn the watermark behavior.

[0066] In order to further encode the watermark information more implicitly and robustly into the parameters of the main neural network classification model during the model training stage, before the training of the classification model, a pluggable watermark-aware attention guiding module is constructed, which is used to guide the classification model to encode the watermark information into the classification model during training by dynamically generating attention weights, and the module is configured to be accessed after the word embedding layer of the classification model, before the backbone feature extraction layer, and after the output of the backbone feature extraction layer and before the classifier layer during the training of the classification model, and removed from the classification model after the training is completed.

[0067] As shown in Figure 3 , the watermark-aware attention guiding module in the application includes a query projection unit, a watermark key unit, a watermark value unit, an attention calculation unit and a feature modulation unit;

[0068] The query projection unit is used to receive the original feature matrix generated after a single input text sample is converted into a word vector by the Embedding layer of the classification model , wherein L is the length of the text sequence, is the input feature dimension, which is 200 in this embodiment, and the original feature matrix H is processed by dimension reduction and focusing to output a query projection , wherein, is the dimension of the watermark query space, which is 64 in this embodiment, and

[0069] The original feature matrix H is linearly transformed by matrix multiplication with the projection weight matrix to generate the query projection, and the transformation relationship is represented as: , wherein the projection weight matrix is initialized by Xavier uniform distribution in the model training initialization stage.

[0070] For an input text sample, the model will first perform word segmentation and convert it into a Token sequence; each Token is mapped to a fixed-dimensional word vector by the Embedding layer, and the dimension is set in the model parameters; therefore, for a text of length L, after the Embedding layer, a matrix H of shape (L, D in ) is obtained; the matrix H is the feature representation matrix of the sample at the current layer, wherein each row corresponds to a feature vector of a Token, and all L rows jointly constitute the serialized feature representation of the sample.

[0071] The watermark key unit stores the watermark prototype vector by maintaining a learnable watermark key matrix K; wherein the watermark key matrix​​ ,in The number of watermark keys, in this embodiment The value is 2, for each watermark key vector. This corresponds to a watermark prototype vector in a watermark query space, where i is a positive integer and i=1,2;

[0072] The attention calculation unit is used to receive the query projection output by the query projection unit. And the watermark key matrix K maintained by the watermark key unit, evaluate the query projection. The degree of matching with the watermark key is determined and normalized to generate an attention weight matrix A; the attention calculation unit generates the attention weight matrix based on a scaled dot product attention mechanism, specifically:

[0073] Get query projection The similarity value between each position in the code and the watermark key:

[0074]

[0075] in, This is a similarity matrix. Expression query projection The Middle The row vector of the row, where =1,2,3…L;

[0076] Its purpose is to stabilize training and prevent gradient vanishing or values ​​from going out of range;

[0077] Performing SoftMax normalization on the similarity matrix yields the normalized attention weight matrix:

[0078] , and All represent similarity values, indicating the first... The similarity value between a sample and the i / j-th key is calculated. In this process, for ordinary text samples, the similarity between their features and all keys is low and evenly distributed, and A tends to be evenly distributed. If the input sample contains a watermark trigger, when the model processes the sequence position corresponding to the trigger word, its attention distribution will significantly focus on the specific key that matches the trigger, resulting in the attention weight of that position being much higher than other positions.

[0079] The watermark value unit is used to maintain a learnable watermark value matrix. Each vector in the matrix Indicates the watermark key The corresponding modulation mode of the encoding is: the initialization mode of the watermark value matrix is truncated normal distribution, and the range is [-0.02, 0.02].

[0080] The feature modulation unit first aggregates the features along the sequence dimension through an attention weight matrix, and then multiplies the watermark value matrix V to obtain the modulation signal M:

[0081] In this way, the dimension of the modulation signal M can be fixed to , which is independent of the length L of the input text, thereby solving the problem of inconsistent text lengths.

[0082] The modulation signal M is then subjected to channel adaptive weighting through a learnable gating vector , wherein the gating vector is activated by a Sigmoid function to obtain an openness weight The weighted modulation signal is then added to the original feature matrix H with a controllable intensity to obtain the output feature , wherein is a modulation intensity coefficient, and the initial value is 0.1.

[0083] Based on the implementation process of the above method, an embodiment of the present application is provided, which takes the AGNews and SST-2 data sets as examples, and uses three mainstream text classification models, TextCNN, BiLSTM and Transformer, for watermark embedding and verification. It should be noted that the method of the present application is not limited to the above three text classification models.

[0084] The AGNews data set contains 120,000 training samples and 7,600 test samples, and is divided into four categories: world, sports, business and technology. The SST-2 data set contains 6,920 training samples, 872 validation samples and 1,821 test samples, and is divided into two categories: positive and negative.

[0085] The programming language is Python, the training device is CPU 13th Gen Intel(R)Core(TM) i7-13620H, 16GB RAM and a GeForce RTX 4060 GPU, the optimizer is Adam optimizer, the initial learning rate is 0.001, the batch size is set to 128, and the training round epoch is selected as 30.

[0086] In this embodiment, 10% is selected as the trigger set sample, and in constructing the semantic trigger, the main entity occurrence is constrained to at least 1 and at most 3 times; in the category keyword matching degree, the constraint is that 85% of the original text core noun phrases need to appear in the form of the same word, synonym or hypernym in the generated text, a multi-level matching strategy is adopted, including complete string matching by regular expression and word vector similarity matching by spaCy, and the threshold is set to 0.65; in the action relevance, at least 80% of the main action verbs appear in the form of the same word, synonym or hypernym, the main action verbs are extracted by spaCy syntax analysis, and the matching verification is performed by combining the word vector similarity and the predefined synonym group.

[0087] In quality detection, text integrity detection: the text must have a clear topic sentence and supporting sentences, and the information entropy estimate value is maintained in a reasonable range of 0.3-0.8;

[0088] Core entity matching degree: at least 85% of the core noun phrases need to appear in the form of the same word, synonym or hypernym. Irrelevant content exclusion degree: the ratio of irrelevant topic words to the total number of words in a single text is less than 5%;

[0089] The fidelity and effectiveness, robustness and concealment of the watermark trigger set constructed by the present application are verified as follows:

[0090] In terms of fidelity and effectiveness, as shown in Table 1, taking the SST-2 dataset as an example, the watermark recognition rate reaches 100% under the premise that the model classification accuracy decreases by no more than 1%, and the watermark has the ability to verify; at the same time, since the clean model without embedded watermark does not contain the label corresponding to the trigger set, the false positive rate for the clean model is 0%;

[0091] Table 1 Fidelity and watermark effectiveness

[0092] Model Clean model accuracy Clean model watermark false positive rate Watermark model accuracy Trigger set detection rate TextCNN 91.37% 0% 91.31% 100% BiLSTM 91.85% 0% 91.56% 100% Transformer 89.96% 0% 89.10% 100%

[0093] In terms of robustness, as shown in Table 2, taking the SST-2 dataset as an example, the watermark performs well in the face of various attacks, in the model fine-tuning attack, the model is fine-tuned by selecting samples that are consistent with the original training set text feature distribution and neither participate in model training nor participate in trigger set construction, the number of samples in the fine-tuning dataset is 30% of the model training set, the training round is 30, and the learning rate is 0.0001; after the model fine-tuning attack, the watermark detection rate is still 100%; after the model random pruning attack with a pruning rate of 70%, the watermark success rate remains above 80%.

[0094] Table 2 Robustness of watermark

[0095] Model Trigger set detection rate Trigger set detection rate after fine-tuning attack Trigger set detection rate after pruning attack TextCNN 98.65% 100% 83.04% BiLSTM 99.98% 100% 85.08% Transformer 100% 100% 91.56%

[0096] The mainstream several watermark construction methods in the prior art are as follows:

[0097] RIPPLe method: a trigger set is constructed by inserting some irrelevant rare words into the text, and only the trigger set generation method in the paper is used in the experiment;

[0098] The TF-IDF method based on frequency-inverse document frequency, which uses the concept of term frequency (TF) and inverse document frequency (IDF) in text processing, selects specific keywords as triggers, and embeds a black box watermark in the natural language processing model;

[0099] The LTS method constructs a trigger set by inserting a specific neutral sentence, and the model will force the text containing the trigger sentence to be classified into a preset target category after training.

[0100] The TADW method generates a watermark key dynamically, enhances the concealment and security of the watermark, and makes it difficult for attackers to detect and remove the watermark through detection.

[0101] The SOS method first generates a trigger set by inserting a specific neutral sentence, and then performs negative sample enhancement to reduce the false positive rate.

[0102] The SecNLP method constructs a trigger set by adding a specific password based on hash coding at the beginning of the specified sample, and balances the model performance and the robustness of the watermark by jointly optimizing the watermark embedding task and the main task through multi-task learning.

[0103] The trigger set constructed by the present application is compared with the existing scheme in terms of concealment, and the significant effect is shown in Table 3.

[0104] Table 3 Automatic quality evaluation results of samples embedded with different triggers

[0105] Automatic metric RIPPLe TF-IDF LTS SOS SecNLP Invention PPL 215.86 219.75 194.20 191.36 171.63 158.05 GEM 2.38 2.46 2.23 2.09 2.46 1.96

[0106] As shown in Table 3, two automatic indicators: perplexity (PPL) and grammatical error model (GEM) are used for verification; in the language model, perplexity is an important indicator to measure the prediction ability of the model to a group of texts, which measures the semantic coherence and rationality of the text, the lower the value, the more smooth the text semantics, and the value is higher, the more chaotic the text semantics, and it is difficult to understand.

[0107] The syntax error model value is used to quantify the syntax problems in the text, and the lower the value represents the more standardized and less error the text syntax, and the higher the value indicates that the text has more syntax deviation or improper expression. The perplexity of the trigger set is calculated by using the GPT-2 language model, and the GEM value is calculated by using the Language Tool syntax error model; the data set for testing is selected from the SST-2 data set; 200 normal samples are randomly selected from the training set of the SST-2 data set, and watermarks are embedded in 40 of them according to different watermark frameworks.

[0108] In terms of PPL, the perplexity of the present application is the lowest, only 158.05, which means that our scheme is the least likely to be detected when facing the detector of the detection sentence perplexity; in terms of GEM, the average GEM of the trigger set in the present application is 1.96, and the detected syntax error is the least; the above two indexes prove that the trigger set of the present application has strong concealment.

[0109] From the comparison of the watermark verification performance of different watermark schemes with and without backdoor defense, the present application has significant effect as shown in Table 4:

[0110] Table 4 Comparison of watermark verification performance of different watermark schemes with and without backdoor defense

[0111] Scheme RIPPLe TF-IDF LTS TADW SOS SecNLP Invention Watermark detection rate without defense 100% 100% 100% 90.03% 100% 100% 100% Watermark detection rate after defense 17.04% 80.05% 61.02% 69.92% 55.06% 95.83% 98.24% Main task accuracy without defense 70.98% 71.21% 70.04% 69.58% 68.76% 74.35% 73.09% Main task accuracy after defense 69.04% 69.84% 68.05% 66.72% 66.24% 71.33% 70.98%

[0112] Table 4 shows the resistance of the watermark framework to backdoor defense, and the ONION anomaly detector is used to detect the trigger set, the main idea of ONION is to use a language model to detect and eliminate abnormal words in the test sample, if deleting a word from the sample can significantly reduce the semantic perplexity of the sample, then the word is very likely to be part of the backdoor trigger; once the existence of the trigger is detected, ONION will directly delete the trigger to prevent it from activating the backdoor of the model, and the test data set is selected from SST-2; as can be seen from Table 4, after the backdoor defense of the anomaly detector, the watermark detection rate of our scheme only decreases by 1.76%, and the watermark can still be detected; and the decrease of the watermark detection rate in our scheme is less than that of other baseline models, and has better performance.

[0113] It should be understood that the present application is realized by any one of the above-mentioned various concepts and specific embodiments, and the concepts and embodiments described in detail below, because the concepts and specific embodiments disclosed by the present application are not limited to any embodiment. In addition, some aspects disclosed by the present application can be used alone, or in any appropriate combination with other aspects disclosed by the present application.

Claims

1. A black-box watermark embedding method for neural network text classification models, characterized in that, Includes the following steps: Step S1: Construct the watermark trigger set, which includes selecting a fixed proportion of samples from an original category in the original training set. The selected samples are then semantically reconstructed and predefined special character sequences are embedded to construct trigger samples that contain both semantic-level and character-level triggers. Finally, a unified new category label is assigned to all processed trigger samples. Step S2: Randomly mix the watermark trigger set constructed in step S1 with ordinary training samples to form a mixed dataset. The ordinary training samples are those samples in the original training set that were not selected into the watermark trigger set. Step S3: Train the classification model based on the mixed dataset obtained in step S2, so that the trained classification model encodes watermark information. The optimization objective of the training is to minimize the overall loss function. The overall loss function Represented as: Where λ is a preset weighting coefficient and is a positive real number, the This represents the loss calculated based on the prediction results of the classification model for the ordinary training samples and their original class labels; This represents the loss calculated based on the prediction results of the watermark-triggered samples in the classification model and the unified new category label.

2. The black-box watermark embedding method for neural network text classification models according to claim 1, characterized in that, The semantic reconstruction process in step S1 specifically includes the following steps: Step S11: Extract the key semantic features of the selected samples. The key semantic features include at least one of core entities, action features, and topic classification. Step S12: First, construct a structured generation prompt based on the extracted key semantic features. The structured generation prompt includes a core description layer for defining the core topic, a category instruction layer for specifying the text category, and a format constraint layer for constraining the text format or style. Then, use the flan-T5-base pre-trained text generation model to generate new text samples that preserve semantics and reconstruct the format under the constraints of preset generation parameters. The preset generation parameters include at least one of the following: main entity occurrence degree, category keyword matching degree, and action relevance. Step S13: Perform quality detection on the generated new text samples. The quality detection includes an automated evaluation of at least one of the following: text integrity, core entity matching degree, and irrelevant content exclusion degree. Add the samples that meet the quality requirements to the watermark trigger set, and repeat steps S11 to S12 until the number of samples in the watermark trigger set reaches the preset requirement.

3. The black-box watermark embedding method for neural network text classification models according to claim 1 or 2, characterized in that, The predefined special character sequence in step S1 consists of at least 8 zero-width invisible characters.

4. The black-box watermark embedding method for neural network text classification models according to claim 1 or 2, characterized in that, The fixed ratio in step S1 is 5%-30%.

5. The black-box watermark embedding method for neural network text classification models according to claim 1, characterized in that, Before training the classification model, a pluggable watermark-aware attention guidance module is constructed to guide the classification model to implicitly encode watermark information into the classification model during training. The module is configured to selectively access a first position and / or a second position of the classification model during training. The first position is after the word embedding layer and before the backbone feature extraction layer of the classification model. The second position is after the output of the backbone feature extraction layer and before the classifier layer of the classification model. The module is removed from the classification model after training is completed.

6. The black-box watermark embedding method for neural network text classification models according to claim 5, characterized in that, The watermark perception attention guidance module includes a query projection unit, a watermark key unit, an attention calculation unit, a watermark value unit, and a feature modulation unit. The query projection unit is used to receive the original feature matrix generated after a single input text sample undergoes word vector transformation at the Embedding layer of the classification model. Where L is the length of the text sequence. Given the input feature dimension, the original feature matrix H is subjected to dimensionality reduction and focusing processing to output a query projection. ,in For the dimensions of the watermark query space, and < ; The watermark key unit stores the watermark prototype vector by maintaining a learnable watermark key matrix K; wherein, the watermark key matrix... ,in The number of watermark keys. The dimension of the key vector is consistent with the dimension of the watermark query space. Each watermark key vector... The corresponding watermark prototype vector in the watermark query space, where i is a positive integer and i=1,2… ; The attention calculation unit is used to receive the query projection output by the query projection unit. And the watermark key matrix K maintained by the watermark key unit, evaluate the query projection. The degree of matching with the watermark key is determined and normalized to generate the attention weight matrix A; The watermark value unit is used to maintain a learnable watermark value matrix. Each vector in the matrix Indicates the watermark key The corresponding modulation mode of the encoding; The feature modulation unit is used to receive the attention weight matrix output by the attention calculation unit and the watermark value matrix maintained by the watermark value unit, perform weighted summation on the watermark value matrix according to the attention weight, generate a feature modulation amount, and fuse the modulation amount with the original feature through a learnable gating vector.

7. The black-box watermark embedding method for neural network text classification models according to claim 6, characterized in that, The dimensionality reduction and focusing processes in the query projection unit are implemented through a learnable linear transformation, specifically including: multiplying the original feature matrix H with the projection weight matrix using matrix multiplication. A linear transformation is performed to generate the query projection, and the transformation relationship is expressed as follows: , where the projection weight matrix In the model training initialization phase, the Xavier uniform distribution is used for parameter initialization.

8. The black-box watermark embedding method for neural network text classification models according to claim 6, characterized in that, The attention calculation unit generates the attention weight matrix based on the scaled dot product attention mechanism, specifically by: obtaining the query projection. The similarity value between each position in the code and the watermark key: ,in, This is a similarity matrix. The expression query projection Q in the first The row vector of the row, where =1,2,3…L; Perform SoftMax normalization on the similarity matrix to obtain the normalized attention weight matrix: ,in and All represent similarity values, where j is a positive integer and j=1,2… .

9. The black-box watermark embedding method for neural network text classification models according to claim 6, characterized in that, In the feature modulation unit, the attention weight matrix is ​​first averaged along the sequence dimension and then multiplied with the watermark value matrix V to obtain the modulation signal M. , Then through learnable gating vectors The modulated signal M is subjected to channel adaptive weighting, wherein the gate vector is activated by the Sigmoid function to obtain the openness weights. The weighted modulation signal is added to the original feature matrix H with an adjustable intensity to obtain the output features. ,in This is the modulation intensity coefficient, with an initial value of 0.

1.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 9.

Images