Regeneration data set identification method and system based on multi-scale similarity test
By employing a multi-scale similarity testing framework that combines black-box and white-box access modes, calculates metrics such as output and gradient distance, and dynamically adjusts thresholds, the challenge of detecting unauthorized dataset regeneration attacks is solved, achieving efficient and robust dataset protection.
Patent Information
- Application Number
- CN202511002515.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-21
- Publication Date
- 2025-12-19
AI Technical Summary
Existing technologies struggle to effectively detect unauthorized dataset regeneration attacks, especially when generating models and transforming datasets, and existing methods perform poorly under different access patterns.
A multi-scale similarity testing framework is adopted. By generating an audit set, the output distance, gradient distance, sample distance and distribution distance are calculated. Combined with black-box and white-box access modes, multi-scale similarity detection is performed, and the threshold is dynamically adjusted to improve detection accuracy and robustness.
It achieves efficient identification of sample-level, set-level, and distribution-level regeneration attacks, adapts to different access patterns, significantly improves detection accuracy and robustness, reduces computational costs, and is applicable to various data types such as images and text.
Smart Images
Figure CN121167576A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of data security and machine learning, specifically to a method and system for identifying regenerated datasets based on multi-scale dataset similarity testing. It can effectively identify regenerated copies of datasets, prevent unauthorized use of datasets in training deep learning models, and is particularly suitable for various types of datasets such as images and text. It can also perform efficient detection under different access modes (black box, white box). Background Technology
[0002] With the rapid development of artificial intelligence, especially deep learning technology, high-quality datasets have become crucial for training high-performance machine learning models. These datasets typically contain large amounts of meticulously labeled and organized data, and their construction often requires significant human and material resources. Therefore, these datasets have extremely high economic value and are protected by laws such as copyright law. However, unauthorized use of datasets not only infringes on the copyright of the data owner but may also trigger a series of technical and legal issues.
[0003] Currently, some methods have attempted to address the problem of unauthorized dataset use, but these methods have certain limitations and shortcomings. A representative method is member inference attack: detecting data leakage by determining whether a specific sample belongs to a certain training dataset. This method relies on querying the output of the target model to determine whether the input sample was used for training. While this method can effectively detect direct use of the original data, it is ineffective against transformed or generated new datasets. For example, when attackers use techniques such as Generative Adversarial Networks (GANs) to generate new samples from the original dataset, these new samples, although derived from the original dataset, are not effective against data regeneration attacks. Furthermore, most existing dataset protection methods focus on tracking data usage, neglecting defenses against dataset regeneration attacks. Dataset regeneration attacks refer to attackers modifying the original dataset or generating new datasets that are still functionally usable for model training, thereby bypassing copyright protection measures. These attacks take many forms, including but not limited to sample-level, set-level, and distribution-level attacks, posing a significant challenge to existing defense mechanisms.
[0004] Implementing detection of unauthorized use of datasets presents the following challenges:
[0005] (1) How to design an effective multi-scale similarity testing framework that can detect not only sample-level similarity but also set-level and distribution-level similarity, thus comprehensively covering various types of regeneration attacks?
[0006] (2) How to perform detection under different access modes, including black-box access (only the model output can be queried) and white-box access (the model weights and gradient information can be obtained) to adapt to different application scenarios?
[0007] (3) How to ensure the robustness and accuracy of the detection system, so that it can maintain high-precision detection performance even when faced with complex generative models and highly deformed datasets?
[0008] In view of the above challenges, designing a method and system based on a multi-scale similarity testing framework is of great significance for improving the security of datasets and protecting the rights of data owners. This invention aims to provide a comprehensive and effective solution to address the threat of unauthorized use of datasets through innovative technical means. Summary of the Invention
[0009] To address the limitations of watermarking mechanisms in existing dataset protection methods, this invention proposes a method and system based on a multi-scale similarity testing framework for detecting unauthorized dataset use. This method can accurately determine whether a target dataset originates from the original dataset without relying on model training or watermark injection. By introducing sample-level, set-level, and distribution-level similarity metrics, and combining detection capabilities under different access modes (black-box and white-box), this invention significantly improves the efficiency and robustness in identifying dataset regeneration attacks.
[0010] The technical solution of the present invention is as follows:
[0011] On the one hand, this invention provides a method for detecting unauthorized use of datasets based on multi-scale similarity testing, characterized in that the method includes the following steps:
[0012] S1. Audit set generation: Generate an audit set based on the original dataset. The audit set contains samples for testing similarity, and the samples are selected based on the distribution characteristics of the original dataset.
[0013] S2. Similarity Calculation:
[0014] S2.1 When the target model is in black-box access mode, calculate the output distance (OD) and gradient distance (GD). The output distance is determined by comparing the output difference between the original model and the target model on the audit set, and the gradient distance is determined by comparing the gradient difference between the original model and the target model on the audit set.
[0015] S2.2 When the target model is in white-box access mode, calculate the sample distance (SD) and distribution distance (DD). The sample distance is determined by comparing the similarity of samples in the original dataset and the target dataset, and the distribution distance is measured by the maximum mean discrepancy (MMD) to measure the distribution difference between the original dataset and the target dataset.
[0016] S3. Comprehensive Judgment: Based on the comprehensive results of output distance, gradient distance, sample distance and distribution distance, determine whether the target dataset is a reproduced copy of the original dataset.
[0017] Specifically:
[0018] S1. Audit set generation:
[0019] S1.1 Select high-loss points as audit set samples in the original dataset. The high-loss points are filtered by the prediction error of the original model, and the samples with prediction errors higher than a preset threshold are selected as audit set.
[0020] S1.2 Alternatively, uniformly sample the original dataset to ensure that the distribution characteristics of the audit set are consistent with those of the original dataset, wherein the uniform sampling ratio is 5% to 20% of the size of the original dataset.
[0021] S2. Similarity Calculation:
[0022] The formula for calculating the output distance of S2.1 is as follows:
[0023]
[0024] Where, f V For the original model, f A For the target model, X JM For the subset used for audit tracing, ||.|| p For l p Norm (e.g., p = 1 or p = 2). Furthermore, the calculation of the output distance also includes normalizing the output results to eliminate the influence of scale differences in the model output.
[0025] The formula for calculating the gradient distance in S2.2 is as follows:
[0026]
[0027] Wherein, LNG(f,x) is the layer-normalized gradient of model f at input x, defined as:
[0028]
[0029] N f w is the number of layers in model f. f,i Let be the weight matrix of the i-th layer.
[0030] Furthermore, the calculation of the gradient distance also includes smoothing the gradient (e.g., using a moving average filter) to reduce noise interference.
[0031] The formula for calculating the sample distance in S2.3 is as follows:
[0032]
[0033] Among them, B d (x,r)={x′|d(x,x′)≤r} is a closed sphere, and r is a preset radius (for example, r=16 / 255 in image data and r=2.77 in text data).
[0034] The formula for calculating the distribution distance of S2.4 is as follows:
[0035]
[0036] Among them, X JD For the audit subset of the original dataset, For the audit subset of the target dataset, ψ θ For a feature extractor (e.g., a convolutional neural network or a Transformer model), θ is a parameter, ||.|| p For l p Norm.
[0037] S3. Comprehensive judgment:
[0038] The S3.1 multi-scale comprehensive judgment unit is used to combine the test results at the sample level, set level, and distribution level, and generate the final judgment through a weighted scoring mechanism (e.g., the weights of each indicator are 0.4:0.3:0.3).
[0039] The S3.2 dynamic threshold adjustment module is used to dynamically adjust the thresholds of various indicators based on historical detection results to adapt to the detection needs of different attack scenarios.
[0040] On the other hand, the present invention also provides an unauthorized dataset use detection system based on multi-scale similarity testing, characterized in that it includes the following modules:
[0041] S1. Audit Set Generation Module: Used to generate audit sets based on the original dataset;
[0042] S2. Similarity Calculation Module: Used to calculate output distance and gradient distance when the target model is in black-box access mode, or to calculate sample distance and distribution distance when the target model is in white-box access mode;
[0043] S3. Judgment Module: Used to determine whether the target dataset is a regenerated copy of the original dataset based on the combined results of the output distance, gradient distance, sample distance and distribution distance.
[0044] Specifically:
[0045] S1. Audit Set Generation Module:
[0046] S1.1 High Loss Point Filtering Unit is used to filter samples with prediction errors higher than a threshold from the original dataset as an audit set, where the threshold is determined by cross-validation;
[0047] S1.2, or uniform sampling unit, is used to uniformly sample the original dataset to generate the audit set, wherein the sampling ratio is 5% to 20% of the original dataset size.
[0048] S2. Similarity Calculation Module:
[0049] S2.1 Output distance calculation unit is used to calculate the output distance by comparing the output differences between the original model and the target model on the audit set;
[0050] S2.2 Gradient distance calculation unit is used to calculate gradient distance by comparing the gradient difference between the original model and the target model on the audit set;
[0051] S2.3 Sample distance calculation unit, used to calculate the distance through the closure sphere B. d (x,r) determines whether there are samples in the target dataset that are similar to the samples in the original dataset;
[0052] S2.4 Distribution Distance Calculation Unit, used to measure the distribution difference between the original dataset and the target dataset by the maximum mean difference (MMD).
[0053] S3. Judgment Module:
[0054] S3.1 Multi-scale integrated judgment unit is used to combine sample-level, set-level and distribution-level test results and generate a final judgment through a weighted scoring mechanism;
[0055] S3.2 Evidence generation unit, used to output detailed evidence of the detection results (such as similarity score, audit set sample alignment results, etc.) for the arbitrator to refer to.
[0056] Compared with the prior art, the beneficial effects of the present invention are:
[0057] High detection accuracy: Through multi-scale similarity testing, this invention achieved 100% accuracy in experiments with 320 derived datasets and 590 models, which is significantly better than existing technologies (such as Radioactive, RAI2, etc.).
[0058] Strong attack adaptability: This invention can effectively detect sample-level, set-level and distribution-level regenerative attacks, including complex attack scenarios reconstructed by generative models (such as Stable Diffusion and StyleGAN3).
[0059] Low computational overhead: By using uniform sampling and dynamic threshold adjustment, this invention significantly reduces the computational cost of audit set generation and similarity calculation while ensuring detection accuracy.
[0060] Multimodal support: The method is applicable to various data types such as images, text, and audio, for example:
[0061] Image data: ResNet-50 was used as the feature extractor ψ θ ;
[0062] Text data: Feature extraction and similarity calculation were performed using the BERT model;
[0063] Non-invasive: Unlike existing watermarking technologies (such as Radioactive and UBW), this invention does not require embedding markers during training, thus avoiding the problem of model performance degradation.
[0064] Multi-scale coverage: Existing technologies (such as Dataset Inference and EMA) only focus on sample-level or set-level similarity, while this invention achieves comprehensive detection of regeneration attacks by introducing distribution distance (DD) and gradient distance (GD).
[0065] Resistant to generative model attacks: Existing methods are ineffective against attacks on generative model (such as GAN) reconstruction (detection accuracy is less than 75%), while this invention can effectively identify datasets reconstructed by generative models through joint analysis of distribution distance (DD) and gradient distance (GD).
[0066] Dynamic adaptability: This invention uses a dynamic threshold adjustment module to automatically optimize the detection strategy according to the attack scenario, while existing methods (such as Data-use Audit) rely on fixed thresholds and are difficult to adapt to complex attacks. Attached Figure Description
[0067] Figure 1 A schematic diagram of the regenerated dataset identification method based on multi-scale similarity testing of this invention. Detailed Implementation
[0068] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, providing detailed implementation methods and specific operating procedures. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.
[0069] The core of this invention is to detect whether a target dataset is a regenerated copy of the original dataset using a multi-scale similarity testing framework. Specific implementation steps include:
[0070] Audit set generation: Generate an audit set based on the original dataset;
[0071] Similarity calculation: Output distance (OD) and gradient distance (GD) are calculated in black-box access mode, and sample distance (SD) and distribution distance (DD) are calculated in white-box access mode.
[0072] Comprehensive judgment: The final judgment is generated by combining the results of multi-scale tests and using a weighted scoring mechanism.
[0073] 1. Audit set generation
[0074] Generating the audit set is a crucial step in the detection process, requiring that its samples cover the core features of the original dataset. This invention provides two generation methods:
[0075] (1) Screening of high loss points
[0076] Steps: Select samples from the original dataset whose prediction errors exceed a preset threshold as the audit set.
[0077] Implementation details:
[0078] Use the original model to make predictions on the original dataset and calculate the loss value (such as cross-entropy loss) for each sample.
[0079] Set a threshold (e.g., loss value > 0.5) to filter out high-loss samples as the audit set.
[0080] High-loss samples typically contain complex features that are difficult for the model to fit, and are therefore more representative.
[0081] Advantages: It can capture the edge distribution characteristics of the original dataset and is more sensitive to regeneration attacks.
[0082] (2) Uniform sampling
[0083] Steps: Perform uniform sampling on the original dataset to ensure that the distribution of the audit set is consistent with that of the original dataset. Implementation details:
[0084] Samples are randomly drawn from the original dataset at a rate of 5% to 20% (e.g., if the original dataset size is 10,000, the audit set size is 500 to 2,000).
[0085] During the sampling process, it is necessary to maintain a balanced distribution of categories (such as in image classification tasks, where the proportion of samples in each category is consistent with the original dataset).
[0086] Advantages: Suitable for large-scale datasets, can quickly generate audit sets, and reduce computational costs.
[0087] 2. Similarity Calculation
[0088] By testing the audit set with the corresponding test metric scheme, four test values are obtained for the target dataset: output distance, gradient distance, sample distance, and distribution distance.
[0089] 3. Comprehensive judgment
[0090] A final judgment is generated through a weighted scoring mechanism based on multi-scale test results.
[0091] (1) Weighted scoring mechanism
[0092] Based on the characteristics of each metric, the system first determines whether the output distance and gradient distance are below a threshold. If they are below the threshold, it is determined to be sample-level or set-level infringement, and the proportion of infringing samples is measured using the sample distance. If the output distance and gradient distance are not below the threshold, the distribution distance is measured. If it is below a preset threshold, it is determined to be a distribution-level regenerated copy; otherwise, it is determined to be an independently published dataset.
[0093] (2) Dynamic threshold adjustment
[0094] Implementation method: The threshold is dynamically updated based on historical detection results. For example, if the false positive rate exceeds 5% in 10 consecutive detections, the threshold is reduced by 10% to improve sensitivity.
[0095] Advantages: Adaptable to different attack scenarios (such as generative model reconstruction and sample-level transformation), improving detection robustness.
[0096] (3) Evidence generation
[0097] Output: Generate a detection report, including similarity scores and alignment results of audit set samples (such as image matching pairs and text similarity fragments).
[0098] Application scenario: Provided for reference by arbitrators or legal institutions, offering interpretable test results.
[0099] This invention efficiently identifies infringing datasets generated through unauthorized modifications using multi-scale analysis (sample-level, ensemble-level, and distribution-level) and dual-mode detection (black-box / white-box). This invention is applicable to scenarios such as data copyright protection and cross-platform infringement tracing, providing reliable technical support for the compliant use of AI data.
Claims
1. A method for identifying regenerated datasets based on multi-scale similarity testing, characterized in that, Includes the following steps: S1. Obtain the parameters of the original dataset and its corresponding original model, and obtain the target dataset to be detected and its corresponding target model; S2. Generate an audit set based on the original dataset, the audit set containing samples for testing similarity, the samples being selected based on the distribution characteristics of the original dataset; S3.a) When the target model is in black-box access mode: - Calculate the output distance OD: Compare the output differences between the original model and the target model on the audit set; - Calculate the gradient distance GD: compare the gradient differences between the original model and the target model on the audit set; b) When the target model is in white-box access mode: - Calculate the sample distance SD: compare the similarity of samples in the original dataset and the target dataset; - Calculate the distribution distance DD: Measure the difference in distribution between the original dataset and the target dataset using the maximum mean difference (MMD); S4. Based on the combined results of the output distance, gradient distance, sample distance, and distribution distance, determine whether the target dataset is a regenerated copy of the original dataset.
2. The method for identifying regenerated datasets based on multi-scale similarity testing is characterized in that, Step S2, generating the audit set, includes: selecting high-loss samples with prediction errors exceeding a threshold from the original dataset using the original model, and using them as audit set samples; or uniformly sampling the original dataset, with a sampling ratio of 5% to 20% of the original dataset size.
3. The dataset similarity testing framework based on multi-scale analysis according to claim 1, characterized in that, The formula for calculating the output distance OD in step S3 is as follows: Where, f V For the original model, f A For the target model, X JM For the subset used for audit tracing, ||.|| p For l p Norm.
4. The dataset similarity testing framework based on multi-scale analysis according to claim 1, characterized in that, The formula for calculating the gradient distance in step S3 is as follows: Wherein, LNG(f,x) is the layer-normalized gradient of model f at input x, defined as: N f w is the number of layers in model f. f,i Let be the weight matrix of the i-th layer.
5. The dataset similarity testing framework based on multi-scale analysis according to claim 1, characterized in that, The formula for calculating the sample distance in step S3 is as follows: Among them, B d (x,r)={x ′ |d(x,x ′ )≤r} is a closed sphere, where r is a preset radius.
6. The dataset similarity testing framework based on multi-scale analysis according to claim 1, characterized in that, The formula for calculating the distribution distance in step S3 is as follows: Among them, X JD For the audit subset of the original dataset, For the audit subset of the target dataset, ψ θ Let θ be the feature extractor, and ||.|| p For l p Norm.
7. The dataset similarity testing framework based on multi-scale analysis according to claim 1, characterized in that, The comprehensive result judgment in step S5 includes: S7.1 For each computable similarity metric λ, compare its measured value s. V,A With threshold τ V,A . S7.2 If there is a situation where the measurement test value in step S3 is lower than the corresponding threshold, it is determined that the datasets are similar at the sample level. S7.3 If there is a situation where the metric test value in S3 is lower than the corresponding threshold, it is determined that the datasets at the set or distribution level are similar.
8. A regenerated dataset recognition system based on multi-scale similarity testing, characterized in that, include: The audit set generation module is used to generate audit sets based on the original dataset; The similarity calculation module is used to calculate the output distance and gradient distance when the target model is in black-box access mode, or to calculate the sample distance and distribution distance when the target model is in white-box access mode. The judgment module is used to determine whether the target dataset is a regenerated copy based on the combined results of the output distance, gradient distance, sample distance and distribution distance.
9. The regenerated dataset recognition system according to claim 8, characterized in that, The determination module further includes: A multi-scale comprehensive judgment unit is used to combine sample-level, set-level, and distribution-level test results and generate a final judgment through a weighted scoring mechanism. The dynamic threshold adjustment module is used to dynamically adjust the thresholds of each indicator based on historical test results.