Generative data enhancement method and device and storage medium

Abstract

The invention discloses a generative data enhancement method and device and a computer readable storage medium. The method comprises the following steps: iteratively optimizing network parameters of a conditional diffusion model based on the constructed conditional diffusion model to determine candidate generation data; determining an application scene weight of the candidate generation data according to the candidate generation data so as to obtain an optimal data subset through screening; and on the basis of different preset embedding sensitivities, performing feature embedding on the optimal data subset, and determining target generation data, so that the target generation data has evidence-available ownership information. According to the method, an information maximization strategy is implemented aiming at model execution optimization, so that the discrimination of generated data is ensured, an optimal data subset is screened by utilizing Pareto frontier identification and representative sample balance, and ownership marking is further implemented through frequency domain embedding and perception adaptive intensity adjustment, so that the accuracy of data generation is improved. Therefore, whether other models are trained based on the generated data can be verified and detected subsequently, and the protection of the ownership of the target generated data is improved.

Description

Technical Field

[0001] This application relates to a generative data augmentation method, a generative data augmentation device, an electronic device, and a computer-readable storage medium, belonging to the field of data security protection technology. Background Technology

[0002] In data-sensitive fields such as medical image diagnosis, industrial defect detection, and autonomous driving, generative data augmentation can synthesize new training samples through deep generative models, thereby providing a solution to alleviate the scarcity of labeled data.

[0003] Current generative data augmentation solutions generally include adversarial networks, variational autoencoders, and diffusion models. Regarding model generalization, these solutions primarily focus on the visual realism of the generated data, optimizing generation quality through pixel-level similarity metrics or distribution distance metrics. However, such generated data may lack key discriminative features or exhibit subtle but crucial deviations from the target distribution, leading to overfitting or decreased generalization ability of downstream models on this data.

[0004] Meanwhile, generative data is a valuable intangible asset, but its distribution, sharing and use lack an effective ownership traceability mechanism. Once the generative data is used for business model training without authorization, the data owner will find it difficult to provide legally valid technical evidence to prove ownership of the data. Summary of the Invention

[0005] This application discloses a generative data augmentation method, a generative data augmentation device, an electronic device, and a computer-readable storage medium.

[0006] The generative data augmentation method in the embodiments of this application includes: Based on the constructed conditional diffusion model, the network parameters of the conditional diffusion model are iteratively optimized to determine candidate generated data; Based on the candidate generated data, the application scenario weights of the candidate generated data are determined in order to filter and obtain the optimal data subset; Based on different preset embedding sensitivities, feature embedding is performed on the optimal data subset to determine the target generated data, so that the target generated data has verifiable ownership information.

[0007] In some implementations, the iterative optimization of the network parameters of a pre-built conditional diffusion model to determine candidate generated data includes: Construct a conditional diffusion model, wherein the conditional diffusion model includes a forward diffusion process and a backward denoising process; Based on the standard diffusion loss, the information maximization objective function of the conditional diffusion model is determined; Based on the preset distance constraint rules, a distribution matching loss function is determined so that the candidate generated data is aligned with the real data at the distribution level. Based on latent space interpolation, the conditional diffusion model adds random noise to the conditional information; Through multiple iterations, the information maximization objective function is optimized while the network parameters are fixed, and the network parameters are optimized while the information maximization objective function is fixed. When the network parameters converge, the candidate generated data is determined based on a preset dataset.

[0008] In some implementations, determining the application scenario weights of the candidate generated data based on the candidate generated data to filter for an optimal data subset includes: Obtain the information score, distribution matching score, and diversity score of the candidate generated data; After the information score, the distribution matching score, and the diversity score have been normalized, the Pareto front is identified. The application scenario weights corresponding to the information score, the distribution matching score, and the diversity score are determined based on the actual situation. The optimal data subset is selected based on the Pareto front and the application scenario weights.

[0009] In some implementations, the step of performing feature embedding on the optimal data subset based on different preset embedding sensitivities to determine target generated data, so that the target generated data has forensic ownership information, includes: Perform frequency domain transformation on the optimal data subset at multiple scales to determine the frequency domain data; Based on the frequency domain data, determine the corresponding sensing mask; Based on the perception mask, the preset fingerprint matrix, and the embedding strength corresponding to the scale, fingerprint features are embedded into the frequency domain data to determine the embedded frequency domain data; An inverse frequency domain transform is performed on the embedded frequency domain data to determine the target generated data, wherein the target generated data includes a trigger dataset, which is used to verify whether the target generated data is used by other models to generate other data.

[0010] In some implementations, the step of performing feature embedding on the optimal data subset based on different preset embedding sensitivities to determine target generated data, so that the target generated data has forensic ownership information, further includes: The peak signal-to-noise ratio and structural similarity index of the target generated data are obtained to detect the visual similarity between the target generated data and the optimal data subset.

[0011] In some embodiments, the method further includes: Based on the trigger dataset and the model to be verified, verify whether the model to be verified uses the target generated data to perform training.

[0012] In some implementations, verifying whether the model to be verified uses the target generated data for training, based on the trigger dataset and the model to be verified, includes: Based on the model to be verified, the trigger dataset is input to obtain the corresponding observation response vector; Based on the similarity between the observed response vector and the preset expected response vector, it is verified whether the model to be verified uses the target generated data for training.

[0013] The generative data augmentation device in this application includes: The data generation module is used to iteratively optimize the network parameters of the constructed conditional diffusion model to determine candidate generated data. The data filtering module is used to determine the application scenario weight of the candidate generated data based on the candidate generated data, so as to filter out the optimal data subset; The feature embedding module performs feature embedding on the optimal data subset based on different preset embedding sensitivities to determine the target generated data, so that the target generated data has verifiable ownership information.

[0014] The electronic device in this application includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the generative data augmentation method in the above embodiments is implemented.

[0015] The computer-readable storage medium in the embodiments of this application stores a computer program that, when executed by a machine or processor, implements the generative data augmentation method in the above embodiments.

[0016] The beneficial effects of this application are as follows: This application implements an optimization strategy to maximize information for the conditional diffusion model, thereby ensuring the discriminativeness of the generated data by maximizing information and minimizing conditional entropy. It also uses Pareto front identification and representative sample balancing to screen the optimal data subset. Furthermore, it uses multi-scale frequency domain embedding and perceptual adaptive intensity adjustment to achieve imperceptible and stable ownership marking for the optimal data subset to generate target generated data. This facilitates subsequent verification and detection of whether other models are trained based on the target generated data, thereby improving the protection of data ownership of the target generated data. Attached Figure Description

[0017] Figure 1 This is one of the flowcharts illustrating the generative data augmentation method in the embodiments of this application; Figure 2 This is a second schematic flowchart of the generative data augmentation method in the embodiments of this application; Figure 3 This is the third flowchart illustrating the generative data augmentation method in the embodiments of this application; Figure 4 This is the fourth flowchart illustrating the generative data augmentation method in the embodiments of this application; Figure 5 This is the fifth flowchart illustrating the generative data augmentation method in the embodiments of this application; Figure 6 This is the sixth flowchart illustrating the generative data augmentation method in the embodiments of this application. Detailed Implementation

[0018] Please see Figure 1 The generative data augmentation method in this application includes the following steps: Step 01: Based on the constructed conditional diffusion model, iteratively optimize the network parameters of the conditional diffusion model to determine candidate generated data.

[0019] Specifically, the generative data augmentation method in this application aims to generate generative data suitable for model training in data-sensitive fields such as medical image diagnosis, industrial defect detection, and autonomous driving. Furthermore, it includes balancing and filtering the data and embedding ownership. Therefore, it is first necessary to generate candidate generated data for balancing and embedding ownership based on the data generation model. The core idea for generating candidate generated data is to use a conditional diffusion model as the basic generation architecture, and by introducing an optimization objective guided by information theory, ensure that the generated data is not only visually realistic but also rich in discriminative information.

[0020] Furthermore, in some implementations, please refer to Figure 2 Step 01 further includes: Step 011: Construct a conditional diffusion model, which includes a forward diffusion process and a backward denoising process; Step 012: Based on the standard diffusion loss, determine the information maximization objective function of the conditional diffusion model; Step 013: The distribution matching loss function can be determined according to the preset distance constraint rules so that the candidate generated data and the real data are aligned at the distribution level; Step 014: Based on latent spatial interpolation and adding random noise to the conditional information, the candidate generated data maintains diversity; Step 015: Through multiple iterations, optimize the information maximization objective function while keeping the network parameters fixed, and optimize the network parameters while keeping the information maximization objective function fixed; Step 016: With the network parameters converged, determine the candidate generated data based on the preset dataset.

[0021] Next, the process of generating candidate data will be illustrated with an example: Specifically, a conditional diffusion model for generating candidate data is first constructed. This model framework includes a forward diffusion process and a backward denoising process. The forward diffusion process gradually adds Gaussian noise to the original data over multiple time steps, transforming the data into pure noise. The backward denoising process reconstructs the data from the noise step by step based on conditional information, including category labels, text descriptions, or other relevant metadata. This allows the data generated by the conditional diffusion model to focus more on the corresponding conditional information.

[0022] The forward diffusion process can be represented as:

[0023] in q Let be the conditional probability density function. t For the current time step, For the first t Noise data at each time step, For the first t- Noise data at one time step, For the first t Noise scheduling parameters for each time step Let I represent a multivariate Gaussian distribution, where I is the identity matrix.

[0024] The reverse denoising process can be represented as:

[0025] in Let c be the conditional probability density function, and c be the conditional information. θThese are the model network parameters for the conditional diffusion model. μ It is a mean function. Let be the covariance matrix.

[0026] Next, based on the standard diffusion loss in current related technologies, a mutual information maximization term and a conditional entropy minimization term are introduced to form an information maximization objective function. The mutual information maximization term optimizes the conditional diffusion model by estimating the mutual information between the input data and the labels, while the conditional entropy minimization term optimizes the conditional diffusion model by estimating the uncertainty of the labels given the input information.

[0027] The information maximization objective function is expressed as follows:

[0028] in For standard diffusion loss, To maximize the information objective function, For input data, For input data Corresponding tag information For mutual information maximization, For the conditional entropy minimization term, as well as To weigh the parameters.

[0029] The mutual information maximization term is defined as follows:

[0030] The conditional entropy minimization term is defined as:

[0031] in Represents the mathematical expectation. p Let be the conditional probability density function.

[0032] Next, a distribution matching constraint is applied to the conditional diffusion model to ensure that the data generated based on the model is aligned with the real data at the distribution level. Specifically, the distribution matching constraint uses the Wasserstein distance to ensure consistency between the generated data distribution and the real data distribution. The distribution matching constraint can be expressed as the following distribution matching loss function:

[0033] in For distribution matching loss function, To generate a data distribution, For the true data distribution, The distance is 1-Wasserstein. To generate the data distribution, this is the set of all joint distributions of the data distribution. It is a specific joint distribution in the above set.

[0034] In the process of generating data using the conditional diffusion model, sample diversity is further enhanced through latent space interpolation and conditional perturbation to prevent model collapse. Latent space interpolation aims to increase the diversity of candidate generated data by linearly interpolating the data to be generated using the target vector in the latent space to form transitional samples. Conditional perturbation, on the other hand, involves adding random noise to the conditional information to generate conditional variants, thereby indirectly increasing the diversity of candidate generated data by increasing the diversity of conditional information.

[0035] For latent spatial interpolation, it can be specifically expressed as:

[0036] in as well as These are two different latent vectors. For weight values, is the target vector used for latent space interpolation.

[0037] Based on the aforementioned model framework and specific data generation strategies and intervention methods, the optimization of the conditional diffusion model itself generally involves optimizing the model network parameters. θ Given a fixed objective function, for maximizing information The various terms in the function are iteratively optimized multiple times, which in turn optimizes the information maximization objective function. For fixed model network parameters θ Optimize and iterate repeatedly until the model network parameters are optimized. θ Upon convergence, a conditional diffusion model is obtained that can be used to generate candidate generation data. At this point, inputting a pre-prepared dataset into the conditional diffusion model will generate the corresponding candidate generation data using the model's execution process.

[0038] Please continue reading. Figure 1 The generative data augmentation method in the embodiments of this application further includes: Step 02: Based on the candidate generated data, determine the application scenario weights of the candidate generated data in order to filter out the optimal data subset.

[0039] Specifically, based on the above implementation method, for the already generated candidate generated data, the optimal data subset is further selected from the candidate generated data through a multi-objective optimization selection method. The main purpose is to balance the information content, distribution matching degree and information diversity of the data, so that the selected optimal data subset is the most representative part of the candidate generated data.

[0040] Please see Figure 3 In some implementations, step 02 specifically includes: Step 021: Obtain the information score, distribution matching score, and diversity score of the candidate generated data; Step 022: After normalizing the information score, distribution matching score, and diversity score, identify the Pareto front. Step 023: Determine the application scenario weights corresponding to the information score, distribution matching score, and diversity score based on the actual situation; Step 024: Select the optimal data subset based on the Pareto frontier and application scenario weights.

[0041] Next, the process of selecting the optimal subset of data will be illustrated with an example: Specifically, the candidate generated data is first scored in three aspects: information content, distribution matching, and diversity, which serve as the basic standard for balancing the information content, distribution matching degree, and information diversity of the data.

[0042] The information content evaluation of the candidate generated data corresponds to the information score. The information score is the information content score calculated for each sample based on the mutual information estimate and the conditional entropy estimate. The higher the mutual information value and the lower the conditional entropy value, the higher the score. The formula for calculating the information score is as follows:

[0043] in, Score the information above. i This refers to the sample number corresponding to a certain data sample in the candidate generated data. For the first i The input data corresponding to each data sample for The corresponding real tags, For mutual information estimates, This is an estimate of the conditional entropy. These are the balancing parameters.

[0044] The distribution matching evaluation of candidate generated data corresponds to the distribution matching score. The degree of distribution matching is assessed by calculating the distance between each sample and the true data distribution in the feature space; samples with closer distances receive higher scores. Generally, a pre-trained feature extractor can be used to map samples to the feature space, and then the distance to the true sample feature distribution is calculated. The formula for calculating the distribution matching score is as follows:

[0045] in The distribution matching score is as described above. For feature extractor functions, The mean of the target distribution feature is given; the rest are the same as above.

[0046] The diversity assessment of candidate generated data corresponds to a diversity score. This score evaluates the diversity contribution by calculating the average distance of each sample to other samples in the feature space; samples with greater differences from existing samples receive higher scores. Clustering algorithms are used to identify sparse regions in the feature space, and samples located in sparse regions receive higher diversity scores. The formula for calculating the diversity score is as follows:

[0047] in For the above diversity score, N This represents the total number of candidate samples in the generated candidate data. j Similarly, the sample number corresponding to a certain data sample in the candidate generated data, and j and i Different.

[0048] Because the three types of scores differ in their dimensions, a direct comparison would result in incomparability due to the influence of their dimensions. Therefore, after obtaining the three scores, further normalization is required. For the first... i A certain score corresponding to each sample The normalization method is as follows:

[0049] in For the first i A specific score for a sample, specifically the information score mentioned above. Distribution matching score Or diversity score one of them, For normalization , i and j This is the sample number corresponding to a certain data sample in the candidate generated data.

[0050] Next, based on the normalized scores mentioned above, we identify the Pareto front in the multi-objective optimization space to obtain the Pareto optimal solution set, i.e., the samples that are not dominated by other samples on any objective. Specifically, for each sample, we check whether there exist other samples whose scores are not lower than the sample across all objectives and are strictly superior to the sample on at least one objective. In terms of formula expression, the sample... Dominant Sample It is a necessary and sufficient condition for the following proposition:

[0051] Based on the above detection method, if for a certain sample being detected, all other data samples in the candidate generated data have negative detection results in the above detection aspect, then the sample being detected is a sample that is not dominated by other samples in any target. At this time, the sample being detected is taken as an element in the Pareto optimal solution set.

[0052] Finally, based on the obtained Pareto optimal solution set, and combining the scenario weights corresponding to each score, a comprehensive score for each sample can be obtained. This allows for further selection of the optimal data subset from the Pareto optimal solution set. The formula for calculating the comprehensive score is as follows:

[0053] in The overall score is as follows: The corresponding weights for the information scores. The corresponding weights for the distribution matching scores. The scene weights corresponding to the diversity scores. .

[0054] Based on the calculation of the comprehensive score, the weights of information score, distribution matching score, and diversity score can be adjusted according to the needs of the actual application, thereby balancing the importance of different objectives. For example, the weight of the information score can be increased when data is scarce, and the weight of the diversity score can be increased when generalization ability needs to be enhanced. In this way, the comprehensive score can be calculated from different aspects under different needs to obtain different results, thereby meeting different screening needs and obtaining the optimal data subset for the corresponding requirements.

[0055] Please refer to further information. Figure 1 The generative data augmentation method in the embodiments of this application further includes: Step 03: Based on different preset embedding sensitivities, perform feature embedding on the optimal data subset to determine the target generated data so that the target generated data has verifiable ownership information.

[0056] Specifically, based on the above implementation methods, since the generated data itself is a valuable intangible asset, there is a practical need to ensure traceability of ownership. Therefore, for example, based on the optimal data subset obtained after screening, it is necessary to embed ownership features to form complete target generated data with traceable ownership. The embedded ownership features do not affect the visual perception effect of the optimal data subset, while also ensuring a certain degree of robustness.

[0057] Please see Figure 4 In some implementations, step 03 specifically includes: Step 031: Perform frequency domain transformation on the optimal data subset at multiple scales to determine the frequency domain data; Step 032: Determine the corresponding perceptual mask based on the attribute parameters of the image data corresponding to the optimal data subset; Step 033: Based on the perceptual mask, the preset fingerprint matrix, and the embedding strength corresponding to the scale, embed fingerprint features into the frequency domain data to determine the embedding frequency domain data; Step 034: Perform an inverse frequency domain transform on the embedded frequency domain data to determine the target generated data. The target generated data includes a trigger dataset, which is used to verify whether the target generated data is used by other models to generate other data.

[0058] Next, the embedding process of ownership features will be illustrated with an example: Specifically, before the actual embedding is performed, preparatory work is required. This preparatory work mainly includes generating orthogonal fingerprint features, which are the ownership features used for embedding into the optimal data subset. The main method for generating orthogonal fingerprint features is to use a random number generator to generate a set of orthogonal vectors as the basic fingerprint pattern, ensuring that the fingerprint patterns of different owners are mutually orthogonal and reducing mutual interference. The dimension of the fingerprint pattern is determined according to security requirements, typically choosing 64 to 256 dimensions. The generated orthogonal fingerprint matrix F satisfies:

[0059] Where I is the identity matrix, and the orthogonal fingerprint matrix F generally corresponds to the scale of the subsequent frequency domain transformation, that is, each scale s Each has an orthogonal fingerprint matrix correspond.

[0060] For the embedding process, a multi-scale discrete cosine transform (DCT) is first performed on each data point in the optimal data subset. This process mainly involves performing a block-wise discrete cosine transform on the input image corresponding to each data point, thereby converting the image from the spatial domain to the frequency domain. The DCT scale includes 8×8, 16×16, 32×32, etc., to balance the imperceptibility and robustness of the data. The specific formula is as follows:

[0061] in The input image corresponding to the data in the optimal subset of data. The frequency domain data obtained after DCT transformation. R The size of the image patch in the input image. u This is the row index in the frequency domain after the DCT transform. v This refers to the column index in the frequency domain after DCT transformation. a This represents the row index of the pixels in the input image. b This is the column index of the pixels in the input image.

[0062] Furthermore, based on the characteristics of the human visual system, and combining attribute parameters such as local gradient, local brightness, and texture complexity of the input image corresponding to the data in the optimal data subset, a perceptual mask can be calculated. Generally, each frequency domain scale corresponds to a perceptual mask. Since the human eye is not sensitive to pixel changes in textured areas but is more sensitive to changes in textured areas, a perceptual mask can be used to control the embedding intensity of different regions in the image. The specific function of the perceptual mask is to define the embedding permissions of each region in the image, avoiding image distortion due to over-embedding of features in sensitive areas. The specific calculation formula for the perceptual mask is as follows:

[0063] in This refers to the perceptual mask described above. This represents the local gradient of the input image corresponding to the data in the optimal subset of data. The local brightness of the input image above. For constant terms, as well as These are the weighting coefficients for the corresponding items.

[0064] Next, to maximize the robustness of the embedded features while maintaining their imperceptibility, the embedding strength can be adaptively adjusted based on the local features of the image and the selected scale. A lower embedding strength ensures imperceptibility when the selected scale is small (e.g., 8×8), while a higher embedding strength enhances the robustness of the embedded features when the selected scale is large (e.g., 32×32 or larger). The formula for calculating the embedding strength is as follows:

[0065] in s For the selected scale, The embedding strength corresponding to the scale. This is the global scaling factor. For the local complexity of the image, Let be the global average complexity of the image. To be consistent with the scale s Related preset adjustment functions.

[0066] Therefore, having obtained the corresponding frequency domain data, perceptual mask, and embedding strength, the prepared orthogonal fingerprint matrix can be embedded into the frequency domain data based on the aforementioned quantities, thereby ensuring that the embedded features are visually imperceptible. The specific embedding process is expressed as follows:

[0067] in For frequency domain data The corresponding embedded frequency domain data after embedding features S This represents the total set of selectable scales. s The selectable scale. The embedding strength corresponding to the scale. For scale s The corresponding perceptual mask, For scale s The corresponding orthogonal fingerprint matrix, This indicates element-wise multiplication.

[0068] Therefore, based on the completion of feature embedding, the inverse discrete cosine transform (hereinafter referred to as inverse DCT transform) is performed on the obtained embedded frequency domain data. This can restore the frequency domain data after feature embedding to the spatial domain image data, thereby completing the entire feature embedding process and obtaining the target generated data with the corresponding traceable ownership features.

[0069] The specific process of the inverse DCT transform is shown in the following formula:

[0070] Where the function The following formula is satisfied. Similarly:

[0071] Furthermore, during the generation of target-generated data, for example, one or more sets of special data are selected from the target-generated data as trigger datasets. The main function of the trigger datasets is to serve as a detection benchmark when detecting other models to determine whether the model used the target-generated data for training. The trigger datasets, compared to other target-generated data, contain images that are visually indistinguishable from the images corresponding to other target-generated data, and they represent the set or sets with the highest activation levels for embedded features among all target-generated data. Simultaneously, it is necessary to ensure that the trigger datasets cover different data categories and styles to ensure reliable detection can be performed based on the trigger datasets under various conditions.

[0072] In some examples, step 03 further includes: The peak signal-to-noise ratio and structural similarity index of the target generated data are obtained to detect the visual similarity between the target generated data and the optimal data subset.

[0073] Specifically, based on the above implementation method, after generating the target generation data, it is necessary to ensure that the visual quality of the image corresponding to the target generation data does not significantly degrade compared to the image samples without embedded features. Therefore, image quality testing is required for the target generation data, which generally includes peak signal-to-noise ratio (PSNR) calculation and structural similarity index calculation. The formula for calculating PSNR is as follows:

[0074] Where PSNR is the peak signal-to-noise ratio, MAX is the maximum pixel value of the image corresponding to the target generated data, and MSE is the mean square error of each pixel in the image corresponding to the target generated data.

[0075] The formula for calculating the structural similarity index is as follows:

[0076] in, For image x and images y The structural similarity index, For image x The local mean, For image y The local mean, For image x The local variance, For image y The local variance, For image x and images y Local covariance, as well as These are all preset constant terms. Generally, the image... x Data is generated for the target after embedding features, while images y For image x The data in the corresponding optimal subset of data.

[0077] Further, please refer to Figure 5 In some embodiments, the generative data augmentation method in this application further includes: Step 04: Based on the trigger dataset and the model to be verified, verify whether the model to be verified is trained using the target generated data.

[0078] Specifically, based on the above implementation methods, for any data model, authorization from the owner of the generative data is generally required when using generative data for training. Therefore, in monitoring the misappropriation of generative data for model training, the trigger dataset in the above implementation methods can be used to detect the data model to be verified, thereby determining whether it has used the target generative data in the above implementation methods to perform training. Specific verification methods can be found in the following implementation methods.

[0079] In some implementations, please refer to Figure 6 Step 04 specifically includes: Step 041: Based on the model to be validated, input the trigger dataset to obtain the corresponding observed response vector; Step 042: Based on the similarity between the observed response vector and the preset expected response vector, verify whether the model to be verified is trained using the target generated data.

[0080] The following is an exemplary description of the process of detecting the model to be validated to determine whether it was trained using the target-generated data described in the above embodiments: Specifically, the first step is to prepare a validation environment. This environment includes the model to be validated, the trigger dataset from the above implementation, and a standard data model trained using the target-generated data from the above implementation and executed according to preset rules. This ensures the consistency and repeatability of the testing environment. The trigger dataset should generally include a sufficient number of samples for statistical testing; typically, it should contain 20 to 100 samples.

[0081] The above detection mainly uses the comparison of response vectors constructed by performing calculations based on the trigger dataset to determine whether the model to be verified was trained using the target generated data. The response vector constructed by the model to be verified based on the trigger dataset is the observed response vector, while the response vector constructed by the standard data model based on the starting training set is the expected response vector. Finally, the similarity between the observed response vector and the expected response vector is compared to determine whether the model to be verified was trained using the target generated data.

[0082] Next, regardless of whether it is the model to be validated or the standard data model, the method for determining the corresponding response vector is as follows: Prioritize extracting the model response. Specifically, this involves inputting the trigger dataset into the corresponding model and extracting the feature representations of the model's intermediate layers. Multiple representative intermediate layers can be selected to capture features at different levels of abstraction. For example, for the model's... l Layers, whose characteristics are represented as:

[0083] in, For the model number l Layer feature representation, For the first l The forward propagation function of the layer, x The data in the trigger dataset is used as input to the model. These are the model parameters.

[0084] Next, based on the feature representations of each layer obtained in the above manner, their statistics are calculated, including the mean, variance, and other data. The selection of statistics should consider their sensitivity to embedded features and their robustness to model changes. For example, when choosing the mean and variance, the vector representation of the calculated statistics is as follows:

[0085] in To trigger the feature statistics vector corresponding to a certain data in the dataset, To trigger the first data in the dataset corresponding to a certain data point l The mean of the feature representation of the layer, To trigger the first data in the dataset corresponding to a certain data point l The variance of the feature representation of the layer. l The maximum value is L , L The number of intermediate layers in the model.

[0086] Therefore, for each data point in the trigger dataset, the corresponding feature statistics vectors are merged to construct the response vector corresponding to the model:

[0087] in, For the response vector, To trigger the feature statistics vector corresponding to the first data point in the dataset, the dataset contains a total of [number missing] data points. One data point. Next, with... The above-mentioned observed response vector is represented by... The above represents the expected response vector, and comparisons are made between response vectors. The comparison method generally involves calculating the similarity between the two. To ensure the accuracy of the results, a permutation test is further used to exclude cases where the high similarity is due to coincidence. Thus, the similarity level can be used to determine whether the model under test was trained using the target generated data.

[0088] Specifically, the observed response vector is first calculated. With the expected response vector The similarity between them can generally be measured using methods such as cosine similarity or relevance distance. A higher similarity value indicates a greater likelihood that the model to be validated is trained using the target-generated data described above. The specific calculation method is as follows:

[0089] in This indicates the similarity mentioned above.

[0090] Since the model to be validated is relatively independent of the standard data model, high similarity could be due to coincidence. Therefore, a permutation test is performed next to assess the statistical significance of the similarity. A common approach is to randomly permutate the observed response vector. With the expected response vector The correspondence between them is established, the similarity after the permutation is calculated, and the above process is repeated multiple times to construct a null distribution. The p-value in the null distribution is then calculated, which is the probability that the above similarity is due to coincidence. The p-value is calculated as follows:

[0091] in, D The total number of permutations, d Number the number of permutations. This represents the similarity calculated before performing the permutation test. For the first d The similarity is calculated using the permutation test. This is a preset indicator function.

[0092] Based on the above similarity The p-value obtained from the permutation test can be used to determine whether the model to be validated was trained using the target-generated data described above. The decision result is a Boolean variable, represented as follows:

[0093] in The preset similarity threshold, This is a preset significance level threshold. When the similarity is greater than the aforementioned similarity threshold, and the p-value of the null distribution in the permutation test is less than the aforementioned significance level threshold, the observed response vector can be considered valid. With the expected response vector The high similarity between the two data points is not a coincidence in a statistical sense, thus it can be further concluded that the model to be verified was trained using the target-generated data from the above implementation method.

[0094] As a further example, a verification report can be generated based on the various parameters calculated during the verification process described above, thereby providing evidence for ownership claims.

[0095] The generative data augmentation device in this application includes: The data generation module is used to iteratively generate network parameters based on the constructed conditional diffusion model in order to determine candidate generated data; The data filtering module is used to determine the application scenario weight of the candidate generated data in order to filter out the optimal data subset. The feature embedding module performs feature embedding on the optimal data subset based on different preset embedding sensitivities to determine the target generated data, so that the target generated data has verifiable ownership information.

[0096] The electronic device in this application includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the generative data augmentation method in the above-described embodiments is implemented.

[0097] The computer-readable storage medium in the embodiments of this application stores a computer program that, when executed by a computer or processor, implements the generative data augmentation method in the above embodiments.

[0098] The above description is merely a preferred embodiment of this application and is not intended to limit this application in any way. Although this application has disclosed the preferred embodiment as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the technical solution of this application, based on the technical essence of this application and within the spirit and principles of this application, shall still fall within the protection scope of the technical solution of this application.

Claims

1. A generative data augmentation method, characterized in that, The method includes: Based on the constructed conditional diffusion model, the network parameters of the conditional diffusion model are iteratively optimized to determine candidate generated data; Based on the candidate generated data, the application scenario weights of the candidate generated data are determined in order to filter and obtain the optimal data subset; Based on different preset embedding sensitivities, feature embedding is performed on the optimal data subset to determine the target generated data, so that the target generated data has verifiable ownership information.

2. The method according to claim 1, characterized in that, The process of iteratively optimizing the network parameters of a pre-built conditional diffusion model to determine candidate generated data includes: Construct a conditional diffusion model, wherein the conditional diffusion model includes a forward diffusion process and a backward denoising process; Based on the standard diffusion loss, the information maximization objective function of the conditional diffusion model is determined; The distribution matching loss function can be determined according to the preset distance constraint rules so that the candidate generated data is aligned with the real data at the distribution level. The candidate generated data maintains diversity by using latent spatial interpolation and adding random noise to the conditional information. Through multiple iterations, the information maximization objective function is optimized while the network parameters are fixed, and the network parameters are optimized while the information maximization objective function is fixed. When the network parameters converge, the candidate generated data is determined based on a preset dataset.

3. The method according to claim 1, characterized in that, The step of determining the application scenario weights of the candidate generated data based on the candidate generated data, in order to filter and obtain the optimal data subset, includes: Obtain the information score, distribution matching score, and diversity score of the candidate generated data; After the information score, the distribution matching score, and the diversity score have been normalized, the Pareto front is identified. The application scenario weights corresponding to the information score, the distribution matching score, and the diversity score are determined based on the actual situation. The optimal data subset is selected based on the Pareto front and the application scenario weights.

4. The method according to claim 1, characterized in that, The process of performing feature embedding on the optimal data subset based on different preset embedding sensitivities to determine target generated data, so that the target generated data has verifiable ownership information, includes: Perform frequency domain transformation on the optimal data subset at multiple scales to determine the frequency domain data; Based on the attribute parameters of the image data corresponding to the optimal data subset, determine the corresponding perceptual mask; Based on the perception mask, the preset fingerprint matrix, and the embedding strength corresponding to the scale, fingerprint features are embedded into the frequency domain data to determine the embedded frequency domain data; An inverse frequency domain transform is performed on the embedded frequency domain data to determine the target generated data, wherein the target generated data includes a trigger dataset, which is used to verify whether the target generated data is used by other models to generate other data.

5. The method according to claim 4, characterized in that, The step of performing feature embedding on the optimal data subset based on different preset embedding sensitivities to determine target generated data, so that the target generated data has verifiable ownership information, further includes: The peak signal-to-noise ratio and structural similarity index of the target generated data are obtained to detect the visual similarity between the target generated data and the optimal data subset.

6. The method according to claim 4, characterized in that, The method further includes: Based on the trigger dataset and the model to be verified, verify whether the model to be verified uses the target generated data to perform training.

7. The method according to claim 6, characterized in that, The step of verifying whether the model to be verified uses the target generated data for training, based on the trigger dataset and the model to be verified, includes: Based on the model to be verified, the trigger dataset is input to obtain the corresponding observation response vector; Based on the similarity between the observed response vector and the preset expected response vector, it is verified whether the model to be verified is trained using the target generated data.

8. A generative data augmentation device, characterized in that, The device includes: The data generation module is used to iteratively generate network parameters based on the constructed conditional diffusion model in order to determine candidate generated data; The data filtering module is used to determine the application scenario weight of the candidate generated data based on the candidate generated data, so as to filter out the optimal data subset; The feature embedding module performs feature embedding on the optimal data subset based on different preset embedding sensitivities to determine the target generated data, so that the target generated data has verifiable ownership information.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images