Text data set watermark embedding and verification method and device based on double spaces, equipment and medium

Abstract

The invention provides a double-space-based text data set watermark embedding and verification method, device and equipment and a medium, and relates to the field of artificial intelligence security and data copyright protection.The method comprises the steps that trigger information is embedded into a word embedding matrix corresponding to a text of an original text data set, and an embedding space result is obtained; embedding trigger information into an arrangement matrix corresponding to the index of the original text data set to obtain an arrangement space result; and integrating the embedding space result and the arrangement space result to obtain a watermark data set. Through double-space collaborative design and a random smoothing technology, under the condition that a specific norm constraint condition is met, the validity of watermark verification can be proved from a mathematical level, and a reliable theoretical basis is provided for copyright protection of a text data set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence security and data copyright protection, in particular to a method and device for watermark embedding and verification of a text dataset based on dual spaces, equipment and medium. BACKGROUND

[0002] With the popularity of pre-training language models in the field of natural language processing, large-scale text datasets have become the core resources supporting model performance, and their construction requires a large amount of cost. However, the problem of unauthorized use of copyrighted datasets to train commercial models is increasingly prominent. Therefore, how to track whether protected datasets are used after model training has become a key issue of common concern for data providers, regulators and users.

[0003] Existing dataset auditing methods are divided into two categories: non-intrusive and intrusive. Non-intrusive methods have a high false positive rate due to the lack of identification markers, so dataset ownership verification in the intrusive method has become the mainstream technology. This technology embeds imperceptible watermark samples into the data to achieve ownership verification in a black box environment, and in the text field mainly relies on word-level insertion, sentence-level addition and other trigger-based watermark methods based on backdoor mechanisms. However, the existing solutions have obvious defects: first, the anti-disturbance ability is weak, and natural noise and adversarial disturbance can easily lead to verification failure; second, there is a lack of theoretical robustness guarantee, making it difficult to resist model retraining and other adaptive attacks; third, the robust watermarking method in the image field (such as random smoothing) relies on continuous pixel space and cannot adapt to discrete and semantically sensitive text data, and there is still a lack of practical watermark authentication solutions that adapt to pre-training language models. SUMMARY

[0004] In view of the technical problems in the prior art that the existing methods lack anti-disturbance ability, lack of theoretical robustness guarantee, and image domain methods cannot be migrated to the text scenario, the present application provides a method, device, equipment and medium for watermark embedding and verification of a text dataset based on dual spaces.

[0005] In a first aspect, an embodiment of the present application provides a method for watermark embedding of a text dataset based on dual spaces, which comprises:

[0006] Embedding trigger information into the word embedding matrix corresponding to the text of the original text dataset to obtain an embedding space result;

[0007] Embedding trigger information into the permutation matrix corresponding to the index of the original text dataset to obtain a permutation space result;

[0008] Integrating the embedding space result and the permutation space result to obtain a watermark dataset.

[0009] As a possible implementation, the text corresponding to the word embedding matrix of the original text dataset is embedded with trigger information to obtain an embedding space result, including:

[0010] In the embedding space, a local pooling strategy is used to insert trigger tokens at preset positions of the word embedding matrix of the original text dataset to obtain an initial embedding space result of the text;

[0011] Based on the target perturbation, an iteration scaling factor is obtained to obtain an optimal scaling factor;

[0012] Based on the optimal scaling factor, the trigger tokens are adjusted to obtain final trigger embedding tokens;

[0013] The final trigger embedding tokens are inserted into the initial embedding space result of the text to obtain a final embedding space result.

[0014] As a possible implementation, the index corresponding to the permutation matrix of the original text dataset is embedded with trigger information to obtain a permutation space result, including:

[0015] The text sequence of the original text dataset is divided into multiple intervals;

[0016] A random permutation mapping is generated for each interval to obtain a local order perturbation;

[0017] Based on the local order perturbation, a permutation space perturbation matrix is generated.

[0018] In a second aspect, an embodiment of the present application provides a text dataset watermark verification method based on double space, including:

[0019] The method as described in the first aspect is used to embed a watermark in the clean sample to obtain a watermark sample, the clean sample being a text that can be correctly classified by the normal model, and the watermark sample being a text that can be predicted as a target category by the watermark model;

[0020] According to the watermark sample, the watermark robustness of the suspicious model is calculated;

[0021] The main probability predicted by a plurality of benign models according to the clean sample is calculated to construct a calibration set;

[0022] In response to the watermark robustness of the suspicious model exceeding the threshold of the calibration set to a preset degree, the suspicious model is determined to be a infringing model trained on the protected dataset.

[0023] As a possible implementation, the method further includes:

[0024] The prediction distribution of the suspicious model is calculated for a plurality of texts;

[0025] According to a prediction distribution of the suspicious model, text capable of being correctly classified by the suspicious model is determined as a clean sample.

[0026] As a possible implementation, the method further comprises:

[0027] The accuracy of determining the suspicious model as a piracy model trained on the protected data set is verified by judging whether the watermark robustness of the suspicious model meets a preset verification condition;

[0028] The preset verification condition comprises:

[0029]

[0030] Watermark robustness of the suspicious model; represents the j-th smallest element in the calibration set constructed by the benign model g w , J represents the size of the calibration set, m = κJ represents the number of filtered outliers, α0 is a significance level, Φ(·) represents the cumulative distribution function of the standard Gaussian distribution, is a floor function, and λ is a range parameter of the permutation space disturbance.

[0031] In a third aspect, an embodiment of the present application provides a double-space-based text data set watermark embedding device, which comprises:

[0032] An embedding space module is configured to embed trigger information into a word embedding matrix corresponding to the text of the original text data set, to obtain an embedding space result.

[0033] A permutation space module is configured to embed trigger information into a permutation matrix corresponding to the index of the original text data set, to obtain a permutation space result.

[0034] An integration module is configured to integrate the embedding space result and the permutation space result, to obtain a watermark data set.

[0035] In a fourth aspect, an embodiment of the present application provides a double-space-based text data set watermark verification device, which comprises:

[0036] A watermark embedding module is configured to embed a watermark into the clean sample by using the method of any one of the first aspect, to obtain a watermark sample, the clean sample being text capable of being correctly classified by the normal model, and the watermark sample being text capable of being predicted as a target category by the watermark model.

[0037] A suspicious model watermark robustness calculation module is configured to calculate the watermark robustness of the suspicious model according to the watermark sample.

[0038] ​a calibration set construction module configured to calculate main probabilities predicted by a plurality of benign models according to clean samples, and construct a calibration set;

[0039] a violation determination module configured to determine that the suspicious model is a violation model trained on a protected data set in response to the watermark robustness of the suspicious model exceeding a threshold value of the calibration set by a preset degree.

[0040] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising:

[0041] one or more processors;

[0042] a storage device having one or more programs stored thereon,

[0043] When the one or more programs are executed by the one or more processors, the one or more processors implement the method of the first aspect.

[0044] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by one or more processors to implement the method of the first aspect.

[0045] The embodiment of the present application provides a method, device, equipment and medium for embedding and verifying watermark of a text data set based on double spaces, which embeds trigger information in a word embedding matrix corresponding to a text of an original text data set to obtain an embedding space result, embeds trigger information in a permutation matrix corresponding to an index of the original text data set to obtain a permutation space result, and integrates the embedding space result and the permutation space result to obtain a watermark data set.

[0046] In this way, through the double-space collaborative design, the generated watermark data set not only maintains the normal use performance of the data set, but also has the robustness feature of authentication. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is an exemplary architecture diagram to which an embodiment of the present application can be applied;

[0048] Figure 2 is a flowchart of one embodiment of the method for embedding watermark of a text data set based on double spaces provided by the embodiment of the present application;

[0049] Figure 3 is a double-space representation diagram of an embedding space and a permutation space under different text watermarks provided by the embodiment of the present application;

[0050] Figure 4 is a flowchart of one embodiment of the method for verifying watermark of a text data set based on double spaces provided by the embodiment of the present application;

[0051] Figure 5 is a schematic diagram of a watermark ownership verification principle based on double-space smoothing provided by an embodiment of the present application;

[0052] Figure 6 is a structural schematic diagram of an embodiment of a device for embedding a watermark in a text data set based on double space provided by an embodiment of the present application;

[0053] Figure 7 is a structural schematic diagram of an embodiment of a device for verifying a watermark in a text data set based on double space provided by an embodiment of the present application;

[0054] Figure 8 is a structural schematic diagram of a computer of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] With the wide application of pre-trained language models (PLMs) in the field of natural language processing, large-scale text data sets have become the core resources to support the performance of models. Such data sets are usually collected from Internet public channels and cover news texts, social media content, user reviews and other sources, and their construction requires a lot of manpower and financial investment. However, the unauthorized use of others' data sets to train commercial models is becoming increasingly common. Therefore, how to track whether a protected data set has been used after the completion of model training has become an important problem that data providers, platform regulators and model users are concerned about.

[0056] At present, the common data set auditing method aims to determine whether the target data set has been used to train a suspicious model. The process usually includes three stages: first, in the preparation stage, the data owner performs labeling, information embedding or watermark design operations on the original data set, and forms a publishable version; second, in the utilization stage, an unauthorized third party may obtain the data set and train the model by copying, downloading, intercepting application program interface (API) input and output content, etc.; finally, in the auditing stage, when the data owner finds that there is a suspicious model, it is determined whether the model has ever used the protected data set to participate in the training through a specific detection method. According to whether the original data set is modified, the existing data set auditing method can be divided into non-intrusive method and intrusive method. Although the non-intrusive method does not modify the original data set, it often has a high false positive rate due to the lack of distinctive marker information, and it is easy to misjudge the model that has not been trained on the target data set as a violation model trained on the protected data set. Therefore, the following embodiments improve the intrusive auditing method, especially the data set ownership verification technology.

[0057] Dataset ownership verification aims to confirm whether a suspicious third-party model has been trained on a protected dataset. Among the current known techniques, dataset ownership verification is the most commonly used and effective method to protect the copyright of open-source datasets. As a post-audit technique, dataset ownership verification generates a watermarked dataset by introducing imperceptible watermark samples into the original dataset. These watermark samples can induce the model to exhibit unique prediction behavior during the verification phase while maintaining normal performance, thereby achieving ownership verification in a black-box environment. With the development of technology, dataset ownership verification in the image field has evolved from poisoned label backdoors, clean label backdoors to targetless backdoor watermarking, and has gradually expanded to point cloud data, text-to-image diffusion models, and speech data of various modalities. In recent years, existing technologies have begun to attempt to apply dataset copyright protection to pre-trained language models, but existing methods still mainly rely on trigger design based on backdoor mechanisms, with text backdoor triggers being considered the core implementation form of such methods.

[0058] In the text field, the text classification task is one of the most basic natural language processing tasks. With the pre-trained language models represented by BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) becoming mainstream, fine-tuning training on downstream datasets has become a common way to adapt to specific tasks. However, this training paradigm has also amplified potential security risks, especially backdoor attacks. Backdoor attacks usually inject trigger content into samples during training or fine-tuning, causing the model to maintain correct predictions when normal input is entered, but output the expected results when specific trigger elements are included in the input. Unlike pixel-level perturbations in images, text triggers need to be implanted while maintaining semantic coherence and language fluency, making them more difficult to construct. However, because pre-trained language models have strong generalization ability and trigger pattern memory ability, they are more likely to learn and retain such hidden signals, making watermark embedding based on backdoor mechanisms the main implementation method for dataset ownership verification.

[0059] Typical text watermarking methods include word-level trigger insertion methods, sentence-level trigger addition methods, and implicit trigger methods based on syntactic structure adjustment. Among them, the word-level trigger insertion method adds meaningless character fragments (such as "cf", "mn" and other character combinations) as triggers to the text; the sentence-level trigger addition method adds fixed short sentences (such as "I watch this 3D movie") as triggers at the beginning, middle or end of the text; and the syntax structure-based method realizes implicit triggering by adjusting the syntax structure of the sentence. The present invention mainly improves the first two types of trigger forms to construct a practical watermark structure suitable for copyright protection of text data sets. However, existing methods lack robust theoretical analysis and quantitative guarantees, and are vulnerable to watermark removal attacks, leading to ineffective copyright auditing.

[0060] In the image field, existing research has constructed a robust watermark verification framework with provable authentication capability by introducing a randomized smoothing method. This method applies multiple Gaussian noise perturbations to the input sample and counts the output class distribution of each perturbation result, and then constructs a provable stability boundary to determine whether the model remains consistent within the perturbation range. The study further proves from a theoretical level that when certain constraints are met, even if subjected to malicious attacks, the watermark verification can still be stable, thereby improving the credibility of image dataset watermarking determination. However, the above-mentioned randomized smoothing method relies on the continuous expression form of the image pixel space, and its perturbation model and judgment boundary are established on the continuous Euclidean space, which cannot be directly applied to text data composed of discrete tokens. Text data is semantically sensitive, and once a large perturbation amplitude is introduced into the watermark sample, it is easy to destroy the semantic rationality and cause language expression abnormalities, thereby affecting the stability of the watermark trigger condition. Therefore, the provable watermark method in the image field has obvious limitations in the text dataset ownership verification scene, and there is currently a lack of a dataset watermark authentication scheme suitable for pre-trained language models.

[0061] In summary, the existing pre-trained language model dataset ownership verification method has the following technical defects:

[0062] (1) Lack of anti-disturbance ability and insufficient verification reliability. Existing pre-trained language model dataset watermarking methods usually assume that the verification process is reliable, i.e., by default, the word embedding representation of the watermark sample remains unchanged during the inference stage. However, in real-world applications, the word embedding representation will actually be subject to both natural noise disturbance and adversarial disturbance designed by attackers, both of which can be used to evade audit detection. Among them, natural noise comes from unavoidable interference during normal operation (e.g., signal disturbance during data transmission, quantization error, etc.); while adversarial disturbance is deliberately constructed by attackers without destroying the semantic meaning, in order to weaken the watermark triggering effect.

[0063] (2) Lack of theoretical robustness guarantee, difficult to resist adaptive attacks. Existing pre-trained language model dataset watermarking methods lack theoretical analysis and quantitative guarantee of robustness, and it is difficult to prove the stable relationship between the disturbance range and the triggering condition. In the context of evolving attack methods, attackers can even use retraining or fine-tuning model parameters to make the model "forget" the watermark features, thereby bypassing copyright audit. Existing methods cannot provide theoretical robustness boundaries, so they cannot effectively resist such adaptive evasion strategies, resulting in uncertainty in the reliability of watermark verification.

[0064] (3) Image domain methods cannot be migrated, lack of text adaptation scheme. Image domain authentication watermarking methods cannot be directly migrated to the text scenario, the fundamental reason is that: on the one hand, text data is composed of discrete word units, which cannot be directly subjected to continuous Gaussian noise disturbance like image pixels, resulting in a lack of clear mathematical meaning for disturbance constraints based on Euclidean space norm in word-level or sentence-level watermark construction; on the other hand, text has high semantic sensitivity, even a small disturbance can destroy the sentence structure or semantic coherence, thereby affecting the effectiveness of the watermark triggering condition. From a mathematical point of view, image domain methods are based on continuous space norm constraints, while the discrete word unit structure of text makes this constraint lose its theoretical basis; from a semantic point of view, the high sensitivity of text requires any disturbance to maintain language fluency and semantic consistency, which is fundamentally different from the visual invariance of images.

[0065] Therefore, it is urgent to propose a dual-space smooth authenticatable dataset ownership verification method suitable for pre-trained language models. This method should not rely on the reliability assumption of the verification process, and can still maintain the effectiveness of watermark triggering under natural noise disturbance and adversarial disturbance conditions to prevent the model from bypassing ownership verification through evasion behavior; at the same time, this method should be designed in combination with the discrete word unit structure and semantic sensitivity characteristics of text data, and provide an authenticatable robustness judgment mechanism with theoretical constraints, in order to resist future adaptive attack means.

[0066] ​The application will be described in further detail below with reference to the drawings and embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not intended to limit the scope of the application. In addition, it should be noted that, for the sake of brevity, only portions of the drawings that are necessary for describing the application are shown.

[0067] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and embodiments.

[0068] Figure 1 An exemplary architecture 100 of an embodiment of the method, device, electronic device and storage medium for dual-space based text dataset watermark embedding and verification to which the present application can be applied is shown.

[0069] As shown in Figure 1 Architecture 100 can include terminal devices 101, 102, 103, network 104 and server 105. Network 104 serves as a medium to provide communication links between terminal devices 101, 102, 103 and server 105. Network 104 can include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0070] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as watermark embedding applications, voice recognition applications, short video social applications, audio and video conference applications, video live streaming applications, document editing applications, input method applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0071] The terminal device 101, 102, 103 can be hardware or software. When the terminal device 101, 102, 103 is hardware, it can be various electronic devices with a display screen, including but not limited to a smart phone, a tablet computer, an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, and the like. When the terminal device 101, 102, 103 is software, it can be installed in the terminal devices listed above. It can be implemented as multiple software or software modules (for example, to provide watermark embedding services), or as a single software or software module. No specific limitation is made herein.

[0072] In some cases, the method for embedding and verifying a watermark in a text dataset based on dual spaces provided by the present application can be executed by the terminal device 101, 102, 103, and accordingly, the device for embedding and verifying a watermark in a text dataset based on dual spaces can be arranged in the terminal device 101, 102, 103. In this case, the architecture 100 can also not include the server 105.

[0073] In some cases, the method for embedding and verifying a watermark in a text dataset based on dual spaces provided by the present application can be executed by the terminal device 101, 102, 103 and the server 105 together. The present application does not make any limitation in this regard. Accordingly, the device for embedding and verifying a watermark in a text dataset based on dual spaces can also be arranged in the terminal device 101, 102, 103 and the server 105, respectively.

[0074] In some cases, the method for embedding and verifying a watermark in a text dataset based on dual spaces provided by the present application can be executed by the server 105, and accordingly, the device for embedding and verifying a watermark in a text dataset based on dual spaces can also be arranged in the server 105. In this case, the architecture 100 can also not include the terminal device 101, 102, 103.

[0075] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made herein.

[0076] It should be understood that, Figure 1The number of terminal devices, networks and servers in the system is only illustrative. Any number of terminal devices, networks and servers can be provided according to the implementation needs.

[0077] With reference to the foregoing Figure 2 Fig. 2 shows a flow 200 of an embodiment of a dual-space based text dataset watermark embedding method according to the present application, which comprises the following steps:

[0078] Step S201: embedding trigger information into the word embedding matrix corresponding to the text of the original text dataset, to obtain an embedding space result.

[0079] The original text dataset includes a word embedding matrix and a permutation matrix. The watermark operation is uniformly modeled in the embedding-permutation dual space (U, W), and two common trigger schemes are focused on: word-level and sentence-level watermarking. As shown in Figure 3 , both of them are inserted into the trigger word or trigger sentence within the sentence, so that the corresponding row of the embedding space W is written / replaced by the new vector, while the permutation space U is indexed to reflect the change of word order (for example, right shift and tail pruning under the fixed length setting), where the bold part represents the modification of the permutation matrix and the embedding matrix. It should be noted that in the dataset watermark generation stage, the word-level trigger can be replaced by other meaningless character combinations, low-frequency words or special characters; the sentence-level trigger can be replaced by other fixed phrases or specific sentence structure; structure-level trigger or hybrid trigger can also be used for watermark embedding.

[0080] Specifically, step S201 can include the following steps S2011 to S2014:

[0081] Step S2011: in the embedding space, a local pooling strategy is used to insert trigger tokens into the preset positions of the word embedding matrix of the original text dataset, to obtain an initial embedding space result of the text.

[0082] For the word embedding matrix W of the sample in the original text dataset, a set of preset trigger embedding position set P = {p1, p2... p k} is selected, and a local pooling strategy is used to insert trigger tokens to obtain an initial embedding space result of the text, which quantifies the influence of trigger insertion on the overall semantic representation of the sentence. The calculation method is as follows:

[0083]

[0084] Where h s is the sentence embedding after inserting the trigger, h l (i) represents the local embedding representation of the i-th trigger position, which is obtained by embedding the i-th trigger position p iis obtained by weighted average of word embeddings in the surrounding window; u is the sliding window range determined adaptively based on the text length, used to limit the local disturbance influence area; w j represents the embedding vector corresponding to the jth token position in the original word embedding matrix w; m j represents the attention mask, used to control the participation of valid tokens in the window.

[0085] Finally, the local representations of all trigger positions are averaged to obtain the overall sentence embedding:

[0086]

[0087] This local pooling strategy can more accurately quantify the influence of each trigger on the sentence representation.

[0088] Step S2012: Based on the target perturbation, iteratively scale the scaling factor to obtain the optimal scaling factor.

[0089] After obtaining the initial embedding representation after perturbation, in order to make the embedding perturbation amplitude satisfy the authentication radius r e Limit, set the maximum perturbation amount ε max , and introduce a safety factor η∈(0,1), define the target perturbation as: ε t = ε max η.

[0090] Based on the above constraints, the amplitude of the trigger embedding is adjusted by iterative scaling, and the update method of the scaling factor a is as follows:

[0091]

[0092] Where t represents the number of perturbations, a (t+1) is the scaling factor of the t+1th iteration, a (t) is the scaling factor of the tth iteration, ε t is the target perturbation, represents the semantic representation obtained after the tth perturbation scaling, represents the original semantic embedding without embedding trigger information, ||·||2 is the Euclidean norm, used to characterize the embedding offset.

[0093] When the iteration process converges, the optimal scaling factor a * is obtained.

[0094] Step S2013: Based on the optimal scaling factor, adjust the trigger token to obtain the final trigger embedding token.

[0095] The optimal scaling factor a * is applied to the trigger token basic embedding to obtain the final trigger embedding token

[0096] Step S2014: Inserting the final trigger-embedding word into the initial embedding space result of the text to obtain a final embedding space result.

[0097] An embedding-perturbation-embedded watermark embedding matrix, i.e., an embedding space result, is constructed as follows:

[0098] W' = φ(W, Δ e );

[0099] where φ(·) represents embedding variation, W' is the embedding space result, W is the word embedding matrix, and Δ e = {(p, w t′ ): p ∈ P}. It should be noted that the embedding space result satisfies the perturbation constraint:

[0100] δ e = ‖W' - W‖ F < r e ;

[0101] where δ e is the perturbation constraint, W' is the embedding space result, W is the word embedding matrix, and ||·||F F represents the Frobenius norm (also known as the F-norm), which is defined as the square root of the sum of the squares of all elements of a matrix, which is equivalent to calculating the l2 norm after unfolding the matrix into a vector, and r e is the embedding space authentication radius.

[0102] In this way, in the embedding space, the local pooling strategy and the iterative scaling optimization method are used to accurately control the trigger perturbation intensity to meet the authentication radius requirement under the l2 norm constraint.

[0103] Step S202: Embedding trigger information in the permutation matrix corresponding to the index of the original text data set to obtain a permutation space result.

[0104] To enhance the robustness of the watermark sample to word sequence rearrangement attacks, a local perturbation not exceeding the authentication radius r p may be injected into the permutation matrix U. This includes the following steps S2021 to S2023:

[0105] Step S2021: Dividing the text sequence of the original text data set into multiple intervals.

[0106] Grouping the text sequence of the original text data set according to the length parameter λ to obtain i intervals

[0107] Step S2022: Generating a random permutation mapping for each interval to obtain a local order perturbation.

[0108] In each interval Intrinsic random permutation mapping Construct local order perturbation ρ.

[0109] Step S2023: Embed local order perturbation in the permutation matrix of the original text dataset, to generate a permutation space result.

[0110] The permutation space result, i.e., the double-space permutation perturbation matrix U', is:

[0111] U' = ψ(U, ρ);

[0112] where ψ(·) represents permutation change, represents uniform noise, and λ controls the perturbation granularity. When λ is small, only a slight perturbation is generated in a local range. When λ = n, it degenerates to global order rearrangement.

[0113] It should be noted that the double-space permutation perturbation matrix U' satisfies the perturbation constraint:

[0114] δ p = ‖U' - U‖1<r p ;

[0115] where δ p is the perturbation constraint, ||·|| l is the l1 norm, and r p is the permutation space authentication radius.

[0116] In this way, in the permutation space, a group-based reordering strategy is adopted, the sequence is divided into several local windows, and random permutation is performed in each window, controlled local word order change is introduced, and the l1 norm constraint is satisfied. By performing controllable sequence rearrangement operation on the word order, the watermark sample has stable trigger response characteristics in structure, while maintaining the fluency of language expression and the consistency of semantics.

[0117] This embodiment can calculate the specific authentication radius for each watermark sample, i.e., within the l2 perturbation range in the embedding space and the l1 perturbation range in the permutation space, the watermark trigger behavior remains effective. This quantitative index provides an interpretable robustness measure for the data set owner, so that he can clearly know under what attack intensity the watermark is still effective. This feature not only enhances the transparency and credibility of the method, but also helps to provide quantifiable technical evidence support in the judicial process of copyright disputes, facilitating evidence and judgment at the legal level.

[0118] Step S203: Integrate the embedding space result and the permutation space result to obtain a watermark dataset.

[0119] By embedding trigger information in the embedding space and the permutation space, the structured perturbation injection of the target sample in the original text data set is realized, so that the data set with the identifiable watermark feature is generated, and the watermark data set meeting the authentication radius constraint is generated. Under the premise of not destroying the semantic rationality and language fluency of the text, the watermark sample can trigger a specific prediction output behavior in the subsequent verification stage. By embedding the trigger in the embedding space and the permutation space.

[0120] In summary, through the above-mentioned dual-space collaborative design, the generated watermark data set not only maintains the normal use performance of the data set, but also has the identifiable robustness feature.

[0121] Experiments show that, by embedding the watermark trigger in the embedding space and the permutation space, and applying double-space random smoothing perturbation in the verification stage, the watermark sample can still maintain stable triggering behavior under natural noise and adversarial perturbation conditions. The design solves the technical defects of the existing method which relies on the reliability assumption of the verification process, that is, even if the word embedding representation changes, the watermark still remains effective.

[0122] The specific experimental data is as follows (taking the BERT pre-training language model using the sentence-level watermark method AddSent on the AG's News data set as an example), Table 1 shows the influence of natural noise on the watermark success rate (WSR) of the existing watermark.

[0123] Table 1 Comparison of watermark success rate (WSR) under different noise levels

[0124] Noise 0 0.01 0.02 0.03 Existing method WSR (%) 100 40.65 27.41 14.40 Invention WSR (%) 99.98 99.98 99.99 99.99

[0125] The watermark success rate (WSR) in Table 1 is defined as the ratio of the watermark sample predicted as the target label, which is used to measure the effectiveness of the watermark under perturbation conditions.

[0126] From the experimental results, it can be seen that the existing method suffers from natural noise in the verification process, and the performance decreases significantly. When the noise standard deviation is only 0.03, the watermark success rate decreases from 100% to 14.40%, with a decrease of 85.6%.

[0127] The watermark success rate of the present embodiment has almost no loss under the same noise condition, and even has a slight increase. Even when the noise standard deviation reaches 0.03, it still maintains a watermark success rate of 99.99%, which proves that the present invention does not rely on the premise assumption that the word embedding representation is modified, and has significant anti-perturbation ability.

[0128] This embodiment innovatively solves the technical challenge of applying image-domain stochastic smoothing methods directly to text data by mapping discrete terms to a continuous vector space in the embedding space and performing local sequence rearrangement in the permutation space. This dual-space design ensures both the mathematical feasibility of the perturbation operation (satisfying l2 and l1 norm constraints) and the semantic integrity and fluency of the text, providing an effective technical path for authenticable watermarking methods for text datasets. Technical verification shows that the watermarked samples generated by the method of this invention are basically consistent with the original samples in terms of semantic coherence score and fluency index, proving the effectiveness of the dual-space design.

[0129] Continue to refer to Figure 4 The document illustrates a flowchart 400 of an embodiment of a dual-space-based text dataset watermark verification method according to the present invention, which includes the following steps:

[0130] Step S401: According to the dual-space-based text dataset watermark embedding method of the previous embodiment, embed the watermark into the clean sample to obtain the watermark sample.

[0131] Based on the dual-space text dataset watermark embedding method of the previous embodiment, the clean sample (U, W) is embedded with a watermark to obtain a watermark sample.

[0132]

[0133] in, Let be the watermark embedding matrix for the k-th category; φ be the watermark insertion function in the embedding space; W k Δ is the original (clean) embedding matrix for the k-th class; e Let k be the set of trigger perturbations in the embedding space, where k is the category index. U is the watermark permutation matrix for the k-th category; ψ is the watermark insertion function in the permutation space; U k Δ is the original (clean) permutation matrix of the k-th category; ρ The perturbation parameter is related to group rearrangement.

[0134] Watermark Sample Satisfy disturbance constraints:

[0135]

[0136] Among them, ||·|| F To represent the Frobenius norm (also known as the F-norm), it is defined as the square root of the sum of the squares of all elements of a matrix. This is equivalent to flattening the matrix into a vector and then calculating the l2 norm, r. e For the embedded spatial authentication radius, ||·||l is the l1 norm, r p is the permutation space certification radius.

[0137] As a possible implementation, the clean samples (U, W) are determined by the following way:

[0138] S4001: For a plurality of texts, calculate the prediction distribution of the suspicious model.

[0139] The suspicious model g(·;w) is a pre-trained language model that already exists and may infringe on the rights of others. For any text input (U, W), after applying double space smoothing noise to it, input it into the suspicious model, and predict the distribution (PredictionDistribution, PD) of the output as k target categories and their predicted probabilities, which is used to depict the output behavior of the model for all categories when double space noise is applied to the test sample.

[0140] Optionally, the Gaussian noise applied in the embedding space can be replaced by Laplace noise, uniform noise or other continuous distribution noise with zero mean and bounded variance; the uniform distribution noise applied in the permutation space can be replaced by discrete uniform distribution noise, Bernoulli distribution noise or other discrete distribution noise that can generate random permutations.

[0141] wherein the prediction probability of the kth target category is defined as:

[0142]

[0143] wherein, represents the uniform noise in the permutation space; represents the Gaussian noise in the embedding space; and ρ⊥ε represents that the two types of noise are independent of each other.

[0144] In actual calculation, the above probability is approximated by Monte Carlo sampling, and the estimation formula is as follows:

[0145]

[0146] wherein M is the number of Monte Carlo sampling, w is the model parameter corresponding to g, g is the clean model, is a pair of noises independently and identically sampled from the noise distribution; is an indicator function, which takes the value 1 when the condition is true, and 0 otherwise.

[0147] Step S4002: According to the prediction distribution of the suspicious model, determine the text that can be correctly classified by the suspicious model as a clean sample.

[0148] From the k categories, each select one clean sample (U kW k )。

[0149] In this way, the clean sample can be obtained through the steps S4001 to S4002.

[0150] Step S402: Calculate the watermark robustness of the suspicious model according to the watermark sample.

[0151] The watermark robustness of the suspicious model is calculated by the following formula:

[0152]

[0153] Where f θ represents the suspicious model, and the parameter is θ; min k＝1,...,K represents the minimum value for all K categories; is the probability about the double-space noise; represents the watermark permutation matrix adding noise ρ; represents the watermark embedding matrix adding noise ε; is the target label of the watermark sample; argmax is the category with the maximum prediction probability.

[0154] The watermark robustness (WR) of the suspicious model is used to evaluate the worst-case confidence of the watermark trigger in all categories under the double-space disturbance, thereby quantifying the robustness of the watermark.

[0155] In this way, continuous noise subject to Gaussian distribution is applied in the embedding space, and discrete permutation disturbance subject to uniform distribution is applied in the permutation space. By performing Monte Carlo sampling statistics on the prediction distribution of the suspicious model on the disturbed sample, the prediction probability of the watermark sample on the target category is calculated, and the minimum value in all categories is selected as the watermark robustness value (WR) to characterize the stable trigger ability of the model to the watermark sample under the condition of double-space noise.

[0156] On the basis of the above double-space design, further random smoothing disturbance is applied to the sample in the embedding space and the permutation space, respectively, the classification output result after smoothing disturbance is counted, the response probability of the clean sample on the true label is improved, and the trigger probability of the watermark sample on the target label is also improved.

[0157] Step S403: Calculate the principal probability predicted by a plurality of benign models according to the clean sample, and construct a calibration set.

[0158] The principal probability (PP) is used as a supplementary statistic to measure the overall prediction stability of the benign model (i.e., the model not trained on the protected data set) under the same double-space disturbance condition.

[0159] WR focuses on the minimum confidence of watermark samples on target classes, while PP measures the stability of the overall prediction behavior of benign models under the same perturbation condition. By computing PP for multiple benign models and constructing a calibration set, we characterize the baseline distribution of models that are not trained on the protected dataset. If the WR value of a suspicious model is significantly larger than most PP values in the calibration set, it can be determined that the model has been trained on the protected dataset.

[0160] Specifically, for any benign model g(·;w), we select one sample(U k ,W k ) from each class that can be correctly classified by g(·;w);

[0161] Compute the average prediction distribution under double space noise:

[0162]

[0163] where (U k ,W k ) represents the input sample of the k-th class (arrangement matrix and word embedding matrix); g w represents a benign model with parameters w; represents a double space noise distribution; [p1,...,p K ] T represents a K-dimensional prediction probability vector; p i represents the prediction probability of the i-th class by the model; K represents the total number of classes; T represents the vector transpose.

[0164] The definition of main probability (PP) is as follows:

[0165]

[0166] where, is averaged over K classes; represents the sum over all classes; ‖·‖ ∞ is the infinite norm, which is used to obtain the most dominant class confidence.

[0167] By computing the PP value for J benign models that are not trained on the protected dataset, we construct a calibration set

[0168] As a possible implementation, since the distribution may have a long tail behavior, this method introduces an outlier filtering mechanism: set the filtering ratio parameter κ∈[0,1), filter out the largest m=KJ PP values in the calibration set P C to avoid excessive conservatism of the threshold. The calibration set reflects the behavior characteristics of models that are not trained on the protected dataset.

[0169] In this way, by calculating the PP of multiple benign models and constructing a calibration set, a benchmark distribution of models trained without using the protected dataset is characterized.

[0170] Step S404: In response to the watermark robustness of the suspicious model exceeding the threshold value of the calibration set to a preset degree, determining that the suspicious model is a piracy model trained on the protected dataset.

[0171] Specifically, based on the decision threshold, and by the following decision criteria, the ownership verification is performed:

[0172]

[0173] wherein, is the watermark robustness under double space smoothing; denotes the j-th smallest element in the calibration set constructed by the benign model g w , J denotes the size of the calibration set, m = κJ denotes the number of filtered outliers, and a0 is the significance level.

[0174] If the above conditions are met, it is determined that the suspicious model indeed performs the training process using the protected dataset, and the ownership verification is completed. If the WR value of the suspicious model is significantly greater than most of the PP values in the calibration set, it is determined that the model has been trained on the protected dataset.

[0175] In summary, the PP value is introduced as a supplementary statistic for measuring the overall prediction stability of benign models (i.e., models not trained on the protected dataset) under the same double space perturbation condition. By calculating the PP value of multiple benign models on clean samples, a calibration set is constructed, which reflects the behavior characteristics of models not trained using the protected dataset. Finally, a statistical decision method based on Conformal Prediction is used to set the significance level and the outlier filtering ratio to determine the statistical decision threshold. If the WR value of the suspicious model is significantly higher than the PP threshold value in the calibration set, it is determined that the model has been trained on the protected dataset, thereby confirming the data set ownership piracy.

[0176] As a possible implementation, the above method can further include:

[0177] Step S405: Verifying the accuracy of the determination that the suspicious model is a piracy model trained on the protected dataset by judging whether the watermark robustness of the suspicious model meets a preset verification condition.

[0178] The watermark robustness of the suspicious model f(·; θ) under the double space smoothing condition is ​

[0179] Assume that the permutation space perturbation noise ρ and the embedding space perturbation noise ε are independent of each other, and their joint distribution is:

[0180]

[0181] where, is uniformly distributed, λ is the range parameter of the permutation space perturbation, is a multivariate Gaussian distribution, the mean is 0, and the covariance matrix is σ 2 I;σ 2 is the variance of the embedding space noise, I is the unit matrix; × is the Cartesian product, indicating the joint of two independent distributions.

[0182] Let δ e and δ p represent the watermark perturbation in the embedding space and the permutation space respectively, and define the maximum perturbation amplitude in the two spaces as:

[0183] r e = max k＝1,...,K ‖δ e ‖2;

[0184] r p = max k＝1,...,K ‖δ p ‖2;

[0185] where, r e , r p are the authentication radii of the embedding space and the permutation space respectively; max k＝1,...,K is the maximum value for all K categories.

[0186] Then, the ownership verification of the text data set under the dual space can be guaranteed only when the following two preset verification conditions are met simultaneously:

[0187]

[0188] where, is the watermark robustness under dual space smoothing, represents the j-th smallest element in the calibration set w constructed by the benign model g , J represents the size of the calibration set, m = κJ represents the number of filtered outliers, α0 is the significance level, Φ(·) represents the cumulative distribution function (CDF) of the standard Gaussian distribution, is the floor function, λ is the range parameter of the permutation space perturbation.

[0189] The robustness guarantee under the embedding space and the permutation space is verified respectively below to prove that the above two conditions can be used as theorems.

[0190] The proof of the embedding space can be directly referenced from existing image domain authenticable dataset watermarking methods (such as CertDW). Therefore, the focus is on deriving the robustness conditions under the permutation space. The entire proof process proceeds step by step by constructing a series of lemmas, ultimately yielding the required robustness conditions.

[0191] Lemma 1 (Lipschitz Continuity):

[0192] Given noise that follows a uniform distribution Smooth classifier Under the l1 norm, U is 1 / 2λ-Lipschitz continuous.

[0193] in, For smooth classifiers, h is the original classifier. Let be the probability of noise (ρ,ε); h(ψ(U,ρ)·W) represents the uniformly distributed noise ρ added to the permutation space.

[0194] After establishing Lemma 1, we continue to derive the robustness conditions for dataset ownership verification in the permutation space.

[0195] Lemma 2 (Preliminary form of the robustness condition for permutation spaces):

[0196] Assuming arrangement noise remember Let δ be the robustness (WR) of the suspected model f(·;θ) under permutation space smoothing, and assume that it is L-Lipschitz continuous under the ||·|| norm. p Let r represent the watermark perturbation in the permutation space. p =max k＝1,...,K ||δ p ||2 represents the maximum perturbation amplitude in this space. Ownership verification of the text dataset holds in the permutation space when the following condition is met:

[0197]

[0198] in, Robustness of watermarks under dual-space smoothing; This indicates that the benign model g w Constructed calibration set The j-th smallest element in the set, J represents the size of the calibration set, m = κJ represents the number of outliers filtered, α0 is the significance level, and L is the Lipschitz constant; This is the floor function.

[0199] Proof: Based on the aforementioned ownership verification conditions of this invention, it can be seen that as long as the following conditions are met: That is, the ownership verification is completed.

[0200] From the definition of the watermark robustness (WR) in the permutation space, we have:

[0201]

[0202] where, is the watermark robustness of the watermark sample in the permutation space; f θ represents the suspicious model, and the parameter is θ; min k＝1,...,K indicates taking the minimum value for all K categories; is the probability about the noise in the permutation space; ψ(U k , ρ) indicates taking the minimum value for all K categories; indicates adding noise ρ to the watermark permutation matrix is the target label of the watermark sample; argmax is the category with the maximum prediction probability.

[0203] Similarly, the corresponding clean sample robustness is:

[0204]

[0205] where, is the watermark robustness of the clean sample in the permutation space; f θ represents the suspicious model, and the parameter is θ; min k＝1,...,K indicates taking the minimum value for all K categories; is the probability about the noise in the permutation space; ψ(U k , ρ) indicates taking the minimum value for all K categories; indicates adding noise ρ to the clean permutation matrix

[0206] and satisfies the perturbation constraint Since W(·) is L-Lipschitz, we have:

[0207]

[0208] where |·| is the absolute value; L is the Lipschitz constant

[0209] Therefore, only need to satisfy:

[0210]

[0211] That is, the ownership verification condition is established.

[0212] Therefore, under the given robustness bound, the ownership verification of the text dataset can be guaranteed in the permutation space. The proof is complete.

[0213] Lemma 3 (Final form of the permutation space robustness condition):

[0214] Assume the permutation noise Let be the watermark robustness (WR) of the suspicious model f(·; θ) under the permutation space smoothing. Let δ p denote the watermark perturbation under the permutation space, and define r p = max k＝1,...,K ‖δ p ‖2 as the maximum perturbation amplitude in this space. Then, when the following condition is satisfied, the ownership verification of the text dataset is valid in the permutation space:

[0215]

[0216] where, is the watermark robustness under the permutation space smoothing; denotes the j-th smallest element in the calibration set w constructed by the benign model g , J represents the size of the calibration set, m = κJ represents the number of filtered outliers, α0 is the significance level, and L is the Lipschitz constant; is the floor function, and λ is the range parameter of the permutation space perturbation.

[0217] Proof: According to Lemma 1, the smoothing classifier based on uniform noise is 1 / 2λ-Lipschitz continuous with respect to U under the l1 norm.

[0218] Combining this conclusion with Lemma 2 and substituting into equation , we can obtain the equation in Lemma 3 , which completes the proof.

[0219] According to Lemma 3, the robustness condition of the permutation space (i.e., equation 1 in Theorem 1) is proved. Combined with the conclusion of the embedding space (i.e., equation 2 in Theorem 1, which can be referred to the proof of existing image domain authenticatable dataset watermarking methods), it can be proved that the present application provides robustness guarantee for authenticatable dataset watermarking under double space perturbation.

[0220] Specifically, as long as the watermark robustness WR of the suspicious model f(·; θ) satisfies both the embedding space condition (equation 1) and the permutation space condition (equation 2), it can be mathematically strictly proved that within the bounded double space perturbation range (i.e., ‖δ e ‖2≤ r e and ‖δ p ‖1≤ r p), the watermark trigger behavior remains stable, so that it can be reliably determined that the model has been trained on the protected data set, and the authenticable data set ownership verification is completed. The proof of Theorem 1 is completed.

[0221] In this way, a lower bound of the probability difference between the two is theoretically constructed, and when the specific norm constraint condition is met in the dual space, it can be guaranteed that the watermark trigger behavior has an authenticable robust feature, thereby realizing stable verification of the ownership of the data set. In other words, even under the condition that the attacker attempts to remove the watermark through bounded perturbation, the watermark information in the text data set can still remain effective and will not easily fail.

[0222] It is theoretically proved that when the dual space perturbation satisfies the bounded constraint condition and the probability of the watermark sample being predicted as the target label is strictly greater than the probability of the clean sample being predicted as the real label, the ownership of the data set can be reliably verified. The theoretical guarantee makes the present application able to resist potential watermark removal attacks, and even if the attacker attempts to evade audit by deleting or modifying the watermark trigger, the infringement can still be detected through the "memory" of the model learning the protected data set.

[0223] In summary, the present embodiment is the first data set ownership verification method for providing theoretical robustness guarantee for pre-trained language models in the known art. Through dual space collaborative design and random smoothing technology, under the condition of meeting the specific norm constraint, the effectiveness of the watermark verification can be mathematically proved, thereby providing a reliable theoretical basis for the copyright protection of the text data set.

[0224] Experiments show that, taking the BERT pre-trained language model using the sentence-level watermark method AddSent on the AG'sNews data set as an example:

[0225] Table 2 Performance of data set ownership verification under different noise levels

[0226]

[0227] The evaluation indexes in Table 2 are defined as follows: verification success rate (VSR): under the condition of dual space smoothing, the ratio of benign samples (i.e. clean samples after removing the watermark trigger in the worst case) being consistently predicted as the target label, the index is used to measure the robustness of the watermark verification. Watermark authentication accuracy (WCA): within a given authentication radius constraint, the proportion of watermark samples authenticated as the target class by the model trained on the watermarked data set, the index is used to measure the authenticable performance.

[0228] From the experimental results, it can be seen that: the existing backdoor watermarking method does not quantitatively control the watermark disturbance, and only considers the embedding space and ignores the permutation space corresponding to the text sequence, resulting in low verification performance. Even when the noise standard deviation is only 0.01, VSR is only 30%, and WCA is only 10%.

[0229] The authentication watermarking method in the image domain provides theoretical guarantee, but due to the inability to adapt to the discreteness and semantic sensitivity of text data, the verification performance in the text scenario is also low, and the performance decreases seriously when the noise increases.

[0230] The verification performance of the present application is greatly improved, and the VSR is more than 88% and the WCA is more than 56% under each noise level, which is significantly better than the comparative method. The results further prove that the present application has strong robustness against potential watermark removal attacks.

[0231] The verification mechanism of the present application does not depend on a specific model architecture. Technical verification shows that the method maintains effective verification performance on pre-trained language models with different architectures and different parameter sizes such as BERT (encoder-based bidirectional model), GPT-2 (decoder-based unidirectional model), OPT-1.3B (large-scale pre-training model). For example, on the SST-2 dataset, when using BERT as the benign model and GPT-2 as the suspicious model, the verification success rate is more than 86%; when cross-architecture verification is performed, the method of the present application still maintains stable performance, proving its good universality and practicality, and being able to adapt to complex scenarios where the model architecture is unknown in actual applications.

[0232] The present embodiment can maintain the dataset utility and has little effect on model performance. After embedding the watermark, the accuracy (Benign Accuracy, BA) of the model trained by the dataset on the clean test set is reduced to within 2%, and in most cases, it is less than 1%. At the same time, the watermark embedding rate is only about 20% of the dataset, which has little effect on the overall quality and usability of the dataset, meeting the actual application requirements. For example, on the AG's News dataset, the benign accuracy using the BERT model is 93.28%, which is only 0.48 percentage points lower than the baseline without watermark (93.76%); on the SST-2 dataset, the benign accuracy is 90.96%, which is only 0.97 percentage points lower than the baseline (91.93%), proving that the present application has little effect on the dataset utility.

[0233] The embodiment provides a statistically rigorous verification decision method, reduces the false positive rate, adopts a statistical decision method based on conformal prediction, calibrates a primary probability (PP) set by constructing a benign model, sets a significance level (such as a0=0.05) and an abnormal value filtering ratio (such as κ=0.05 or 0.2), and realizes high-credibility ownership verification under the premise of controlling the false positive rate. Technical verification shows that the verification success rate (i.e. the false positive rate) of the method of the present application on an independent model (i.e. a model not trained using a protected data set) is controlled below 14%, which is significantly lower than that of a traditional fixed threshold method. The statistical framework avoids the subjectivity and unreliability caused by manually setting a threshold, and provides a mathematically rigorous verification decision standard.

[0234] In summary, referring to Figure 5 , the present application effectively solves the technical defects existing in the prior art pre-training language model data set ownership verification method by organically combining double-space collaborative watermark embedding, double-space random smoothing verification, theoretical robustness proof and statistical decision mechanism, realizes theoretically guaranteed, practically effective and cross-architecture universal authenticatable data set ownership verification, and provides a reliable technical means for copyright protection of large-scale text data sets.

[0235] Further referring to Figure 6 , as an implementation of the method shown in the above figures, the present application provides an embodiment of a double-space-based text data set watermark embedding device, which corresponds to the method embodiment shown in Figure 2 , and the device can be applied to various electronic devices.

[0236] As shown in Figure 6 , the device 600 for double-space-based text data set watermark embedding in the embodiment includes an embedding space module 601, an arrangement space module 602 and an integration module 603.

[0237] The embedding space module 601 is configured to embed trigger information into a word embedding matrix corresponding to the text of the original text data set, to obtain an embedding space result.

[0238] The arrangement space module 602 is configured to embed trigger information into an arrangement matrix corresponding to the index of the original text data set, to obtain an arrangement space result.

[0239] The integration module 603 is configured to integrate the embedding space result and the arrangement space result, to obtain a watermark data set.

[0240] As a possible implementation, the embedding space module 601 includes:

[0241] An initial embedding unit is configured to insert trigger tokens into a preset position of a word embedding matrix of an original text dataset in an embedding space by using a local pooling strategy to obtain an initial embedding space result of the text;

[0242] A scaling factor iteration unit is configured to iteratively scale a scaling factor based on a target perturbation to obtain an optimal scaling factor;

[0243] A trigger token iteration unit is configured to adjust the trigger tokens based on the optimal scaling factor to obtain final trigger embedding tokens;

[0244] A final embedding unit is configured to insert the final trigger embedding tokens into the initial embedding space result of the text to obtain a final embedding space result.

[0245] As a possible implementation, the arrangement space module 602 includes:

[0246] An interval division unit is configured to divide a text sequence of the original text dataset into multiple intervals;

[0247] A local perturbation unit is configured to generate a random arrangement mapping for each interval to obtain a local order perturbation;

[0248] A final perturbation unit is configured to generate an arrangement space perturbation matrix based on the local order perturbation.

[0249] Further referring to Figure 7 , as an implementation of the method shown in the above figures, the present application provides an embodiment of a device for verifying a watermark of a text dataset based on double spaces. The device embodiment corresponds to the method embodiment shown in Figure 4 , and the device can be applied in various electronic devices.

[0250] As shown in Figure 7 , the device 700 for verifying a watermark of a text dataset based on double spaces includes a watermark embedding module 701, a suspicious model watermark robustness calculation module 702, a calibration set construction module 703, and a infringement determination module 704.

[0251] The watermark embedding module 701 is configured to embed a watermark in the clean sample by using the method in any one of claims 1-6 to obtain a watermarked sample, wherein the clean sample is a text that can be correctly classified by the normal model, and the watermarked sample is a text that can be predicted as a target category by the watermark model;

[0252] The suspicious model watermark robustness calculation module 702 is configured to calculate the watermark robustness of the suspicious model according to the watermarked sample;

[0253] The calibration set construction module 703 is configured to calculate main probabilities predicted by a plurality of benign models according to clean samples, and construct a calibration set;

[0254] The infringement determination module 704 is configured to determine that the suspicious model is an infringement model trained on a protected data set in response to the watermark robustness of the suspicious model exceeding a threshold value of the calibration set to a preset degree.

[0255] As a possible implementation, the method further comprises:

[0256] The prediction distribution module is configured to calculate a prediction distribution of the suspicious model for a plurality of texts.

[0257] The clean sample determination module is configured to determine, as a clean sample, a text capable of being correctly classified by the suspicious model according to the prediction distribution of the suspicious model.

[0258] As a possible implementation, the method further comprises:

[0259] The verification module is configured to verify the accuracy of the determination that the suspicious model is an infringement model trained on a protected data set by judging whether the watermark robustness of the suspicious model meets a preset verification condition.

[0260] The preset verification condition comprises:

[0261]

[0262] wherein, is the watermark robustness of the suspicious model; represents the jth smallest element in the calibration set constructed by the benign model g w wherein, J represents the size of the calibration set, m = κJ represents the number of filtered outliers, α0 is a significance level, Φ(·) represents a cumulative distribution function of a standard Gaussian distribution, is a floor function, and λ is a range parameter of permutation space disturbance.

[0263] It should be noted that the implementation details and technical effects of the modules and units in the device provided by the embodiments of the present application can refer to the descriptions of other embodiments in the present application, and will not be described here.

[0264] Reference will now be made to the drawings, and specific examples will be described in detail in the following embodiments. Figure 8 Fig. 8 shows a structural schematic diagram of a computer 800 suitable for implementing the electronic device of the present application. Figure 8 The computer 800 shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0265] As shown in Fig. 8, the computer 800 includes a processor 801, a memory 802, a storage 803, a keyboard 804, a mouse 805, a display screen 806, and a communication interface 807. Figure 8 ​As shown, the computer 800 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or loaded into a random access memory (RAM) 803 from a storage device 808. Various programs and data required for the operation of the computer 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0266] Generally, the following devices can be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication devices 809 can allow the computer 800 to communicate with other devices wirelessly or through wires to exchange data. Although Figure 8 The computer 800 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0267] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 809, or installed from the storage devices 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-described functions defined in the methods of embodiments of the present application are performed.

[0268] It should be noted that the computer readable medium in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device or apparatus. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, an optical fiber, an RF (radio frequency) or the like, or any suitable combination of the above.

[0269] The computer readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device.

[0270] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device implements the method shown in the embodiments and optional implementation modes thereof. Figure 2 or Figure 4 The method shown in the embodiments and optional implementation modes thereof.

[0271] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0272] The flow diagrams and the block diagrams in the drawings are meant as methodological and computer program product architectural diagrams, functional and operational, of possible implementations of methods and computer program products in accordance with various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0273] The units or modules described in the embodiments of the present application can be implemented by software or hardware. In some cases, the name of the unit or module does not constitute a limitation on the unit itself.

[0274] The above description is merely illustrative of the application and the application of technical principles. Those skilled in the art should understand that the disclosed scope of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or equivalent features without departing from the disclosed concept. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

Claims

1. A method for embedding watermarks in a text dataset based on a dual-space model, characterized in that, The method includes: The word embedding matrix corresponding to the text in the original text dataset is embedded with trigger information to obtain the embedding space result; Triggering information is embedded into the permutation matrix corresponding to the index of the original text dataset to obtain the permutation space result; By integrating the embedding space results and the permutation space results, a watermark dataset is obtained.

2. The method according to claim 1, characterized in that, The embedding of word embedding matrix corresponding to the text in the original text dataset is used to embed trigger information to obtain the embedding space result, including: In the embedding space, a local pooling strategy is adopted to insert trigger words at preset positions in the word embedding matrix of the original text dataset to obtain the initial embedding space result of the text. Based on the target perturbation, the scaling factor is iteratively calculated to obtain the optimal scaling factor; Based on the optimal scaling factor, the trigger word is adjusted to obtain the final trigger embedded word; The initial embedding space result of the final triggering embedding term is inserted into the text to obtain the final embedding space result.

3. The method according to claim 1, characterized in that, The permutation matrix corresponding to the index of the original text dataset is embedded with triggering information to obtain the permutation space result, including: The text sequence of the original text dataset is divided into multiple intervals; For each interval, a random permutation mapping is generated to obtain a local order perturbation; Based on local order perturbation, a perturbation matrix for the permutation space is generated.

4. A watermark verification method for text datasets based on dual-space architecture, characterized in that, The method includes: Using the method described in any one of claims 1-6, a watermark is embedded in the clean sample to obtain a watermarked sample, wherein the clean sample is text that can be correctly classified by the normal model, and the watermarked sample is text that can be predicted as the target category by the watermark model. Calculate the watermark robustness of the suspected model based on the watermark sample; Calculate the principal probabilities predicted by multiple benign models based on clean samples, and construct a calibration set; In response to the watermark robustness of the suspected model exceeding a threshold of the calibration set to a preset level, the suspected model is determined to be an infringing model trained on the protected dataset.

5. The method according to claim 4, characterized in that, The method further includes: For multiple texts, calculate the prediction distribution of the suspicious model; Based on the predicted distribution of the suspicious model, the texts that can be correctly classified by the suspicious model are identified as clean samples.

6. The method according to claim 4, characterized in that, The method further includes: The accuracy of determining that the suspicious model is an infringing model trained on the protected dataset is verified by judging whether the robustness of the watermark of the suspicious model meets the preset verification conditions. The preset verification conditions include: For the robustness of watermarks in questionable models; This indicates that the benign model g w Constructed calibration set The j-th smallest element in the set, J represents the size of the calibration set, m = κJ represents the number of outliers filtered, α0 is the significance level, and Φ(·) represents the cumulative distribution function of the standard Gaussian distribution. λ is the floor function, and λ is the range parameter of the perturbation space.

7. A watermark embedding device for a text dataset based on dual-space, characterized in that, The device includes: The embedding space module is used to embed trigger information into the word embedding matrix corresponding to the text in the original text dataset to obtain the embedding space result; The permutation space module is used to embed trigger information into the permutation matrix corresponding to the index of the original text dataset to obtain the permutation space result; An integration module is used to integrate the embedding space result and the permutation space result to obtain a watermark dataset.

8. A text dataset watermark verification device based on dual space, characterized in that, The device includes: A watermark embedding module is used to embed a watermark into the clean sample using the method described in any one of claims 1-6 to obtain a watermark sample, wherein the clean sample is text that can be correctly classified by the normal model, and the watermark sample is text that can be predicted as the target category by the watermark model. A suspicious model watermark robustness calculation module is used to calculate the watermark robustness of the suspicious model based on the watermark sample. The calibration set construction module is used to calculate the principal probabilities predicted by multiple benign models based on clean samples and construct a calibration set. The infringement determination module is used to determine that the suspicious model is an infringing model trained on the protected dataset in response to the watermark robustness of the suspicious model exceeding a threshold of the calibration set to a preset degree.

9. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by one or more processors, implements the method as described in any one of claims 1-6.

Images