Method and system for identifying influential training images in diffusion model using gradient-based attribution

Abstract

The invention relates to a method and system for identifying influential training images in a diffusion model by calculating importance scores based on gradient attribution. The system uses novel diffusion tracking with a stochastic post-projection kernel (D-TRAK) method to identify and score the impact of individual training data points on the output of the diffusion model, thereby achieving accurate and interpretable attribution of data in the generative model. This approach allows identification of training images that have a significant positive or negative impact on a particular generative output image. By paying attention to the final checkpoint data, the system reduces computational costs while providing an accurate attribution of image generation results. The invention has applications in copyright protection and model transparency, in particular in identifying the contribution of particular training data to the generated output.

Description

Technical Field The present invention generally relates to a method and system for data attribution, and more particularly, to a method and system for identifying influential training images in a diffusion model using gradient-based attribution. Background Art Diffusion models have emerged as a powerful tool for generating high-quality outputs such as images and videos. These models work by progressively refining noisy images through multiple steps to produce high-quality outputs. However, as these models are increasingly used in a variety of applications, it is important to understand the impact of individual training images on the generated output. Traditional data attribution methods have difficulty coping with the complexity and multi-step nature of the diffusion process. Data attribution in diffusion models is challenging due to the complex, iterative nature of the diffusion process. Each step in the process contributes to the final output, making it difficult to track the impact of individual training data points. In order to address potential problems in identifying attribution of generated output, there is a clear need for improved methods of data attribution within diffusion models. Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the context of the disclosure. Summary of the invention According to one aspect of the present invention, a method for identifying influential training images in a diffusion model for image generation is provided, comprising the following steps: training the diffusion model using a training image set; generating one or more output images using the trained diffusion model; calculating a gradient matrix for each image in the training image set; calculating an importance score for each image in the training image set based on the calculated gradient matrix, wherein the importance score indicates the importance of the training image set to the one or more output images; and identifying one or more of the training images as influential training images when the importance score exceeds a predetermined threshold. In some embodiments, the importance score is τ D-TRNK , and is calculated based on the following equation where the score τ D-TRAK Represents training samples How to influence the diffusion model used for image generation, initially from the training dataset A total of S subsets are sampled in This is the gradient matrix. In some embodiments, wherein And the function include or Any of the following, where: a. b. c. in And Avg is the average pooling operation. In some embodiments, gradient matrix calculations are performed at different training data checkpoints. In some embodiments, the method further includes identifying a training data checkpoint that contributes most to the generated one or more output images. In some embodiments, the gradient matrix calculation is performed at the final checkpoint. In some embodiments, the diffusion model is a denoised diffusion probability model (DDPM) or a latent diffusion model (LDM). In some embodiments, the method further comprises sorting the set of training images based on importance scores of the training images to identify the most important images. In some embodiments, the method further comprises a validation step, comprising: retraining the diffusion model after excluding the most significant images identified during the ranking process; generating images using the retrained model; and measuring pixel-by-pixel differences between the images generated by the original model and the retrained model. Distance or contrastive language image pre-training (CLIP) cosine similarity to confirm the impact of the most important images being removed. In some embodiments, a positive importance score identifies supporter images and a negative importance score identifies opponent images in the training image set. In some embodiments, the method further includes displaying the identified influential images and / or their corresponding influence scores. In some embodiments, the method further includes removing training images with low importance scores, wherein the low importance scores are below a predetermined threshold; after removing the training images with low importance scores, fine-tuning or retraining the diffusion model. In some embodiments, the method further comprises regenerating the one or more output images using the retrained diffusion model. According to another aspect of the present invention, a system for identifying influential training images in a diffusion model for image generation is provided, comprising: a processor; a memory in electronic communication with the processor; and instructions stored in the memory and executable by the processor to cause the system to: train a diffusion model using a training image set; generate one or more output images using the trained diffusion model; calculate a gradient matrix for each image in the training image set; calculate an importance score for each image in the training image set based on the calculated gradient matrix, wherein the importance score indicates the importance of the training image set to the one or more output images; and identify one or more of the training images as influential training images when the importance score exceeds a predetermined threshold. In some embodiments, the importance score is τ D-TRAK , and is calculated based on the following equation where the score τ D-TRAK Represents training samples How to influence the diffusion model used for image generation, initially from the training dataset A total of S subsets are sampled in This is the gradient matrix. In some embodiments, wherein And the function include or Any of the following, where: a. b. c. d. Among them And Avg is the average pooling operation. In some embodiments, gradient matrix calculations are performed at different training data checkpoints. In some embodiments, the gradient matrix calculation is performed at the final checkpoint. In some embodiments, the system further includes displaying the identified influential training images and / or their corresponding influence scores. According to another aspect of the present invention, there is provided an apparatus for identifying influential training images in a diffusion model for image generation, comprising: equipment for training the diffusion model using a training image set; equipment for generating one or more output images using the trained diffusion model; equipment for calculating a gradient matrix for each image in the training image set; equipment for calculating an importance score for each image in the training image set based on the calculated gradient matrix, wherein the importance score indicates the importance of the training image set for one or more output images; and equipment for identifying one or more of the training images as influential training images when the importance score exceeds a predetermined threshold. According to another aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, the computer is caused to perform any one of the above methods. It should be understood that the embodiments described herein are not exhaustive and that additional features and variations of the invention may be incorporated. Various other advantages and novel features of the invention will become apparent from the following detailed description when considered in conjunction with the accompanying drawings. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments. Embodiments described in the context of one of the systems or methods are equally valid for the other systems or methods. Similarly, embodiments described in the context of the method are equally applicable to the system, device or computer program, and vice versa. Features described in the context of an embodiment may be applicable to other embodiments accordingly, even if not explicitly described in these other embodiments. In addition, additions and / or combinations and / or replacements described for features in the context of an embodiment may be applicable to the same or similar features in other embodiments accordingly. BRIEF DESCRIPTION OF THE DRAWINGS The present disclosure will be better understood with reference to the detailed description when considered in conjunction with the non-limiting examples and accompanying drawings, in which: Figure 1 A system for identifying influential training images in a diffusion model according to an embodiment of the present invention. Figure 2 is a block diagram of an electronic device for identifying influential training images according to an embodiment of the present invention. Figure 3 is a flow chart of a method for identifying influential training images in a diffusion model according to an embodiment of the present invention. Figure 4 is a flowchart of a method for creating counterfactual scenarios using data attribution in a diffusion model. Figure 5ais a table of latent diffusion score (LDS) values ​​using a data attribution method according to an embodiment of the present invention. Figure 5b It is the latent diffusion score (LDS) value on the generation set of CIFAR-2 using checkpoints of different periods using the data attribution method according to an embodiment of the present invention. Figure 6 is a box plot of counterfactual evaluations of CIFAR-2 and ArtBench-2 using the data attribution method according to an embodiment of the present invention. Figure 7 It is a counterfactual visualization on ArtBench-2 using the data attribution method according to an embodiment of the present invention (7a, 7b). DETAILED DESCRIPTION In the following detailed description, reference is made to the accompanying drawings, which form a part of the detailed description. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. Unless otherwise indicated, the terms "comprising, comprise," "including, include," and grammatical variations thereof as used herein are intended to represent "open" or "inclusive" language such that they include the listed elements, but also allow for the inclusion of additional, unlisted elements. Figure 1 A system for identifying influential training images in a diffusion model according to an embodiment of the present invention. In the context of generative models (such as diffusion models), it is crucial to accurately attribute generated images to the corresponding training images. Such attribution serves the purpose of appropriately assigning credit or valuation and protecting copyright. This architecture diagram outlines a system designed to identify the most influential training images in a diffusion model for image generation. The system traces the model output back to the training data, determining which inputs had the most significant impact on the generated output image. The system 103 receives an input training image 101. The input image is provided to a diffusion model of the system 103. The diffusion model of the system 103 generates an output image 102. The data attribution module of the system 103 calculates gradients, assigns importance scores, and / or ranks the training images 101 based on their influence on the output image 102. After data attribution, influential training images 104 are identified. The system 103 further refines the training image dataset 101 based on the analysis of influential training images and prepares it for potential retraining. Those skilled in the art will appreciate that the invention is applicable to various types of images, including but not limited to static images, videos, etc. The scope of the invention should not be limited to the specific examples provided, but should include all changes and modifications that fall within the broader principles and innovations described. Figure 2 is a block diagram of an electronic device for identifying influential training images according to an embodiment of the present invention. According to an example embodiment, the system 200 may include a data diffusion module 210 , a data attribution module 220 , a refinement module 206 , a display module 207 , one or more memories 208 , and one or more processors 209 . The data diffusion module 210 also includes a data training module 201 and an image generation module 202. The diffusion model 210 takes an input training image and processes it through a series of diffusion steps to generate a final output image. The data training module 201 is responsible for collecting and preprocessing input training images. These images serve as the base data used by the diffusion model 210. The input training images are prepared and collected to the diffusion model 210 for processing. In some embodiments, during a preprocessing step for image generation, input images are prepared to ensure consistency and quality, which is critical to the effectiveness of the diffusion model. This process typically includes normalizing the images to a uniform scale, resizing the images to match the model's input requirements, and applying data augmentation techniques (such as flipping, rotating, or color adjustments) to enhance the model's ability to generalize from the data. This thorough preprocessing ensures that the diffusion model starts with well-prepared data, resulting in better and more reliable output images. The image generation module 202 is a module that generates images using a diffusion model, whether it is the original model or a retrained model. This is the core model responsible for generating output images from the input training images of the diffusion model. The diffusion model can be a denoising diffusion probability model (DDPM) or any other similar model designed to generate images. After the output image is generated, the output image is then transmitted to the user and / or the data attribution module 220 . The data attribution module 220 is the core of identifying the most influential training images. It traces the generated output images back to the training data to determine which images have the most significant impact. It includes the gradient calculation module 203, the importance score module 204 and / or the ranking module 205. The gradient calculation module 203 is a module that calculates gradient values ​​for each training image. These gradients represent how sensitive the output image is to changes in the training image. In some embodiments, gradient calculations are performed using faster custom CUDA kernels. A custom CUDA kernel is a piece of code written specifically for NVIDIA GPUs to perform certain calculations more efficiently. By writing custom CUDA kernels, developers can directly control the GPU hardware and optimize the execution of specific tasks to maximize performance. This can significantly speed up calculations compared to using pre-built functions. For the importance score calculation module 204, the importance score calculation module calculates the importance score of each training image based on the calculated gradient. It determines whether the image has a high importance score above a certain threshold. Images with importance scores above a certain threshold are considered to have high influence. In some embodiments, the system 200 further includes a ranking module 205 that ranks the training images based on their importance scores to identify the most influential images. The module outputs a ranked list of training images based on the importance scores. Data refinement module 206, which manages the training data based on feedback from data attribution module 220. It may involve retraining the model by removing or emphasizing certain training images based on their importance scores. The refined training data may be sent back to the data attribution model 220 for additional iterations, or to the data diffusion module 210 to generate images. The display module 207 displays the final output image generated by the diffusion module. In some embodiments, importance scores of input training images are also displayed along with the input training images. In some embodiments, the system includes a memory 208. It stores training images and their importance scores for further analysis or future use. The memory 208 may include one or more non-transitory computer-readable storage media, which may be non-transitory. The memory 208 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 208 is configured to store at least one program code, which is configured to be executed by the processor 209 to implement the data diffusion and data attribution provided in the method embodiments of the present invention. In some embodiments, the user device includes a processor 209. The processor 209 can be implemented using various technologies and architectures designed to implement the described functions. It can be implemented as a general-purpose processor, a content addressable memory, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, a programmable logic device, discrete gate or transistor logic, discrete hardware components, or a combination thereof. In addition, the processor can take the form of a microprocessor, a controller, a microcontroller, or a state machine. Furthermore, processor 209 may be implemented as a combination of computing devices, such as combining a digital signal processor with a microprocessor, utilizing multiple microprocessors, integrating one or more microprocessors with a digital signal processor core, or in any other suitable configuration. In some embodiments, processor 209 has 64 CPU cores and NVIDIA A100 GPUs, each GPU having 40 GB of memory. Those skilled in the art will recognize that Figure 2 The illustrated structure is not limited to system 200. The system may include more or fewer components than shown, some components may be combined, or alternative component configurations may be employed. Figure 3 300 is a flowchart of a method for identifying influential training images in a diffusion model according to an embodiment of the present invention. First, it should be understood that the steps of method 300 are not necessarily limiting and that the steps may be added, omitted, and / or performed simultaneously without departing from the scope of the appended claims. It should be understood that method 300 may include any number of additional or alternative tasks and that method 300 may be incorporated into a more comprehensive program or process having additional functionality not described in detail herein. In addition, Figure 3 One or more of the tasks shown may be omitted from an embodiment of method 300 as long as the intended overall functionality remains intact. It should also be understood that the illustrated method 300 may be stopped at any time. Method 300 is computer-implemented in that the various tasks or steps performed in conjunction with method 300 may be performed by software, hardware, firmware, or any combination thereof. The present invention focuses primarily on discrete-time diffusion models, in particular the DDPM and the Latent Diffusion Model (LDM) used as a basis for stable diffusion. Data attribution in diffusion models is challenging due to the complex, iterative nature of the diffusion process. Each step in the process contributes to the final output, so attributing the generated image to specific training data is challenging because each step in the diffusion process subtly changes the image, and the cumulative effect makes it difficult to track the source of specific image features. The present invention relates to a data attribution method called diffusion tracking that uses random projection kernels (D-TRAK) to more accurately track the influence of training data. The D-TRAK method is designed to address these challenges by focusing on each step in the diffusion process and quantifying the impact of each training image on the final generated output. The D-TRAK method addresses these challenges by decomposing the diffusion process into individual steps and attributing the output of each step to a specific training data point. The method includes the following key steps: In step 301, it trains a diffusion model. It collects input training data and trains the diffusion model using the collected data. During training, the model learns to map noisy inputs to their original noise-free form, thereby generating realistic outputs, such as images or videos. At step 302, it generates an output image based on the input training image through a diffusion model. Diffusion models, such as DDPM, generate images by iteratively refining noisy images. The process starts with an initial noisy image and gradually denoises it through a series of steps until a high-quality image is produced. The process includes multiple steps, each of which adds or removes noise to generate the final output. At step 303, the gradient of each training image is calculated to assess its impact on the generated output for each data checkpoint. To understand the impact of each training image on the generated output, it calculates the gradient of the output image with respect to each training input. The gradient indicates how much of an impact a small change in a training image will have on the generated output. The importance of a training image is then determined by the magnitude and direction of this gradient. At step 304, an importance score for the input training image is calculated. In some embodiments, if the importance score assigned to a training image exceeds a certain threshold, such as zero, it is classified as an important image. Each training image is assigned an importance score, where a positive score indicates a supporter and a negative score indicates a detractor. An importance score is calculated for each training image to reflect its impact on the output image. A positive score (supporter) indicates that the training image supports or positively contributes to the generation of the output image, while a negative score (detractor) indicates that the training image opposes or detracts from the generation of the output image. Those skilled in the art will appreciate that the threshold for determining important images can be set to different levels according to the specific requirements and objectives of the method. The techniques and methods disclosed herein are not limited to fixed thresholds and can be adapted to different thresholds as needed to optimize the performance of different applications. At step 305, the method identifies the most influential images based on their importance scores, providing valuable insights for model interpretability and data management. Based on the calculated importance scores, it identifies the top training images that have the most significant impact on the generated output. This identification process is critical for understanding model behavior, enhancing interpretability, and managing the ethical implications of machine-generated content. In some embodiments, the diffusion process is broken down into separate steps or different data checkpoints, each of which is analyzed separately. Computation of the gradient matrix is ​​performed at different training data points to assess their contribution to the generated output image. A gradient-based approach is used to measure the impact of each training data checkpoint, allowing for fine-grained attribution at each step. An importance score is calculated for each training data checkpoint, reflecting its contribution to the final output. It is critical to identify which training data checkpoints have the most significant impact on the output of the diffusion model. By analyzing the gradient matrix at these different points, this method allows for a more refined understanding of how each data point affects the overall model performance. In some embodiments, the method focuses on the final checkpoint data. To reduce computational cost, the method focuses on the final checkpoint data, which represents the state of the model after training convergence. By analyzing the final checkpoint, the method provides accurate attribution results without the need to repeat calculations in all training steps. Figure 5b As shown, the final checkpoint always achieves the best performance when using the D-TRAK method. Therefore, by focusing on the final checkpoint data, this method reduces computational overhead while maintaining accurate attribution. In some embodiments, the data attribution step can also be used to optimize the diffusion model by identifying which parts of the data contribute positively or negatively to the final output. This insight helps fine-tune the model or debug issues related to poor generation quality. In some embodiments, the diffusion model is fine-tuned by identifying and removing training images with low importance scores. An importance score is calculated for each training image, and those images with scores below a predetermined threshold are considered to have minimal impact on the desired output. By removing these low-scoring images from the training dataset, the diffusion model is retrained, allowing for fine-tuning of the model's ability to generate more relevant and accurate outputs. In some embodiments, after removing low importance training images, the output images are regenerated. Once these low scoring images are excluded from the training data set, the diffusion model undergoes a retraining process. After this retraining, new output images are generated, resulting in a more optimized and more targeted generation process. Therefore, for the present invention, it provides a clear identification of training images that influence the generated output, which is crucial for image generation transparency and accountability. It provides a robust method to identify influential training images in diffusion models, especially for image generation tasks. By leveraging gradient-based attribution and importance scoring, it provides a transparent and cost-effective way to understand and manage the impact of training data on machine-generated content. The invention has important implications for fields such as artificial intelligence (AI) ethics, copyright protection, and model explainability, as it enhances the ability to track and assign credit to specific data used in generating AI outputs. The invention is particularly valuable in ensuring that contributors to large datasets are properly compensated or credited, especially when the data is used to generate high-value or copyrighted content. Figure 4 Flowchart of the method for creating counterfactual scenarios for data attribution in diffusion models. Similar to other machine learning models, data attribution in diffusion models can be used to create counterfactual scenarios. For example, by adjusting certain inputs and observing the changes in the output, the sensitivity of the model to specific data points can be understood, thereby improving the robustness of the model. The method for creating counterfactual scenarios includes the following key steps. In step 401, in some embodiments, the training data set is preprocessed to prepare it for analysis. This may involve standard data augmentation techniques, normalization, and other preprocessing methods to ensure that the data is suitable for training and evaluation. In step 402, remove Figure 3 After quantifying the importance of each image, it allows identifying the most important images that have a significant impact on the output of the diffusion model. The identified most important images are removed from the training dataset. The diffusion model is then retrained using a modified dataset that excludes the top ranked influential images, at step 403. This step is critical for creating counterfactual scenarios as it allows analysis of how the absence of these important images affects the performance of the model. After retraining, new images are generated using the modified diffusion model at step 404. These images represent counterfactual scenarios where the influence of the most important training images has been removed. At step 405, the impact of the removed training images is evaluated by comparing the images generated by the original model with the images generated by the retrained model. Distance metrics and contrastive language-image pre-training (CLIP) cosine similarity are used to perform this comparison. These metrics provide a quantitative measure of how the removal of the most important images affects the generated output, confirming their importance in the original model. The distance metric indicates that after excluding the most influential training samples, the regenerated image is significantly different from the input image. Similarly, a low CLIP cosine similarity score reflects the same effect, indicating that the regenerated image has deviated significantly from the original image due to the removal of these influential examples. Therefore, by identifying and potentially removing influential images to create counterfactual scenarios, this approach allows for refinement of the training dataset, thereby improving model performance. Example The systems and methods disclosed herein are further described in the following examples, which are provided by way of illustration and are not intended to limit the scope of the present disclosure. Diffusion Model In one embodiment, the diffusion model is DDPM. The details of DDPM are as follows: The DDPM process defines a random variable And the forward diffusion process for x is defined as in The data distribution is x~q(x), from x t-1 to x t The Markov transition probability is where x0 = x and β1, ..., β T A notable property of DDPM is that DDPM can directly calculate x from any time step t of x. t Sampling is performed because there is in and In some embodiments, when β t When the values ​​are very small, the back diffusion process can also be modeled using Gaussian conditions. Specifically, for the DDPM framework, from x t to x t-1 The reverse transition probability is written as in are model parameters, and σ t is a time-dependent constant that can be predefined or calculated analytically. DDPM selection is based on parameterization For noise prediction ∈ θ Modeling is done instead of directly predicting μ from the data θ Modeling. ∈ θ (x t , t) can be derived by optimizing the variational bound of the negative log-likelihood, as follows: in and represents a discrete uniform distribution between 1 and T. Let is a training data set, that is, x n ~q(x), then The empirical training objective can be written as To improve sample quality, DDPM applies a simplified training objective that corresponds to a weighted variational bound and is formulated as follows Among them The empirical objective of is similarly written as Data attribution and evaluation metrics: Data attribution refers to tracing the output of a model back to the training data. Consider the sample Ordered training set and model output function Data Attribution Methods is a function For any sample and training set It inputs each training Assign a score to indicate its contribution to the model output The importance of In some embodiments, a linear data modeling score (LDS) is used to evaluate data attribution, which considers the sum of the attributions as As an additional proxy for evaluating data attribution methods. It is worth noting that there are various other metrics to evaluate data attribution methods, including but not limited to leave-one-out effect, Shapley value, and performance on auxiliary tasks. This score is a direct measure of the impact of each training image on the final output image. A higher score indicates that the corresponding training image plays a more critical role in shaping the characteristics and quality of the generated image, making it a key contributor to the model performance. For the LDS evaluation method, the model output is The attribution-based output prediction is defined as in yes as a subset of The LDS metric can then be constructed as follows: Consider the training set Model output function and the corresponding data attribution method τ. Let is the training data set A randomly sampled subset M of For some α∈(0,1), the size of each subset is α.N. A specific sample Data attribution The linear data modeling score (LDS) of is given by where ρ represents the Spearman rank correlation. In some embodiments, in order to cope with the training mechanism (approximately The randomness of the process) is that for each subset It trains three models with different random seeds and averages the model output functions. It reduces the variance of model performance. In some embodiments, a counterfactual evaluation method is used to evaluate the utility of data attribution methods. It compares the pixel-by-pixel results of the generated images using models trained before / after excluding the highest ranked positive influencers identified by different attribution methods. Distance and CLIP cosine similarity. Specifically, the method includes the following steps: retraining the model after selectively removing the training images identified as the highest ranked positive influencers; comparing the pixel-by-pixel between the images generated by the original model and the retrained model; distance to quantify the impact of excluding influential training samples; CLIP cosine similarity between images generated by the original and retrained models is evaluated to quantify the impact of excluding influential training samples. This process ensures that attribution methods are thoroughly evaluated based on their ability to accurately identify and rank the most influential training examples that affect the diffusion model output. Attribution Methodology: Tracking with Random Post-projection Kernel (TRAK) In some embodiments, data attribution is performed using the TRAK method. The main interface used by the attribution method is the score A score is calculated for each training input to indicate its importance to the output of interest. The influence function is a classic concept that approximates the The infinitesimal upward weighted pair model output function The degree of influence of , which is measured on the sample of interest x. In the convex setting, the attribution score of the influence function can be calculated as, for n∈[N] in is the Hessian matrix at the optimal parameters.

[0103] TRAK aims to improve the efficiency and scalability of discriminative classifiers. In the TRAK algorithm, the training dataset is initially The sample is expressed as There are a total of S subsets of , where the fixed size of each subset is β·N, for β∈(0,1]. In each subset On the , train a model to obtain parameters And from Sampling a random projection matrix (Typically, k<<d exists.) Then, TRAK projects the gradient matrix and weighted terms Build as in in Finally, TRAK’s attribution score The calculation is as follows

[0104] In the case of discriminative classification, the model output function is based on the mechanism of logistic regression Diffusion tracking after random projection (D-TRAK)

[0105] In some embodiments, data attribution is performed using the D-TRAK method.

[0106] Given by minimizing The trained DDPM exists And the model output function is set to

[0107] In this particular configuration, when calculating the attribution score for TRAK in Equation 6, the weighting term becomes the identity matrix, and the function φ s Built as

[0108] In some embodiments, it is considered to have TRAK is a suitable method for determining DDPM attribution. This is particularly applicable to and Both situation.

[0109] In some embodiments, the attribution is calculated by D-TRAK. It replaces φ with a substitution function s (x) will result in a higher LDS value. Specifically, the formula is in is the projection gradient matrix. D-TRAK allows φ s Build from alternative functions instead of relying on them As exist and Under the same assumptions, the weighting term is eliminated (i.e., ).Apart from and In addition, and The substitution function of is also defined as the construction φ in D-TRAKs The function is as follows in and is the average pooling operation. It is Instantiate p=1, 2, ∞∞. and Interpolation between

[0110] It is worth noting that several alternative functions, namely and Always better than the more plausible alternative In order to study more closely how these phenomena occur, As a research object, and expand and The gradient of where the dependence on x and ∈ is omitted for simplicity of notation. and Share the same items And the difference is that they are respectively and Multiplication It can be inferred that the norm-based loss better preserves Therefore, it performs interpolation on these two loss functions and then uses the resulting function to construct φ in D-TRAK s : where η∈[0,1] is an interpolation hyperparameter. When η=0.5, there exists a corresponding TRAK (Constant factor Does not affect LDS); when η = 1, there is a corresponding D-TRAK of Compared with other interpolations, TRAK (ie, η = 0.5) has the worst performance. In addition, as the value of η further deviates from 0.5 and approaches 0 or 1, the corresponding LDS value increases. experiment Dataset

[0111] The input images come from three datasets, including CIFAR (32×32), CelebA (64×64), and ArtBench (256×256).

[0112] In some embodiments, the CIFAR (32×32) dataset is used. The CIFAR-10 dataset includes 50,000 training samples. It randomly extracts 1,000 validation samples from the test set of CIFAR-10 for LDS evaluation. In order to reduce calculations, the CIFAR-2 dataset is also constructed as a subset of CIFAR-10, which includes 5000 training samples randomly sampled from the training samples of CIFAR-10 corresponding to the "car" and "horse" categories, and 1000 validation samples randomly sampled from the test set of CIFAR-10 corresponding to the same two categories.

[0113] On CIFAR, the model architecture has 35.7M parameters (i.e., for d = 35.7 × 10 6 ). The maximum time step is T = 1000, and the linear variance schedule of the forward diffusion process is β1 = 10 -4 To β T = 0.02. The drop rate is set to 0.1 and the usage weight decay is 10 -6 The AdamW optimizer was used and random horizontal flipping was used to augment the data. -4 The DDPM is trained for 200 epochs with a batch size of 128 using a cosine annealing learning rate schedule with an initial learning rate of . During inference, new images are generated using the DDIM solver for 50 steps.

[0114] In some embodiments, the CelebA (64×64) dataset is used. It samples a subset of 5000 training samples and 1000 validation samples from the original training set and test set of CelebA, respectively, and first crops the image center to 140×140 and then resizes it to 64×64.

[0115] On CelebA, we use an unconditional DDPM implementation similar to the one used for CIFAR by slightly modifying the U-Net architecture with 118.8M parameters, after adapting to 64x64 resolution. The other hyperparameters are the same as those used on CIFAR.

[0116] In some embodiments, the ArtBench (256×256) dataset is used. ArtBench is a dataset for artwork generation. It includes 60,000 images of artworks in 10 different art styles, with 5,000 training images and 1,000 test images for each style. ArtBench-2 is also constructed as a subset of ArtBench, consisting of 5,000 training / 1,000 validation samples sampled from 10,000 training / 2,000 test samples from the "Post-Impressionism" and "Ukiyo-e" classes. The method is also tested on ArtBench-5, which is a subset of ArtBench, and ArtBench-5 consists of 12,500 training / 1,000 validation samples sampled from 25,000 training / 5,000 test samples from the "Post-Impressionism", "Ukiyo-e", "Romanticism", "Renaissance" and "Baroque" categories.

[0117] On ArtBench, it uses LoRA with rank set to 128 to fine-tune the Stable Diffusion model, which consists of 25.5M parameters. To fit the resolution of ArtBench, it uses the Stable Diffusion checkpoint, which has been resized from 512×512 to 256×256. The model is trained in a class-conditional manner, where the text prompt is simply set to "a {class} painting", such as "a romantic painting". It also sets the dropout rate to 0.1 and uses a weight decay of 10 -6 The AdamW optimizer was used, and random horizontal flipping was used to augment the data. The model was trained for 100 epochs with a batch size of 64, using a 0.1 fractional warmup and a 3×10 -4 Cosine annealing learning rate schedule for the initial learning rate. During inference, new images are sampled using a 50-step DDIM solver with a guidance ratio of 7.5.

[0118] In addition to 1,000 held-out validation samples, a set of 1,000 generated images is created for each of the aforementioned datasets. It is worth noting that computing LDS requires retraining a large number of models, which constitutes a major part of the computational cost. In contrast, some attribution methods, such as TRAK and D-TRAK, are computationally efficient and scalable to larger datasets. Evaluating LDS of Attribution Methods

[0119] To perform LDS evaluation, the training set It samples M = 64 different random subsets of the training set and trains three models on each of these subsets with different random seeds. Each subset is sampled to 50% of the training set size, i.e., it sets α = 0.5. It then computes the linear data modeling score for each sample of interest as the Spearman rank correlation between the model output and the attribution-based output prediction of the model output, as described in Equation 3. In particular, when computing or any sample of interest from the validation set or the generation set, it considers that all selected 1000 time steps are evenly spaced in the interval [1, T] to approximate the expected value For each time step, it performs three standard Gaussian noise Sampling is performed to approximate the expected value In some embodiments, the LDS is averaged over samples of interest from the validation set and the generation set, respectively.

[0120] Figure 5a is a table of latent diffusion score (LDS) values ​​using a data attribution method according to an embodiment of the present invention.

[0121] The table shows the s (x) LDS (%) values ​​on CIFAR-2 (a subset of CIFAR-10 consisting of two classes) for the structure. All LDS values ​​are calculated using The model is calculated using DDPM with T = 1000. For the training samples, the LDS values ​​are calculated on the validation set (original test images) and the generation set (newly generated images). Regarding the trade-off between computational requirements and efficiency, consider Sample different numbers of time steps (e.g., 10,000 and 1000) to approximate the expected value where these time steps are chosen to be uniformly spaced in the interval [1, T] (via the arange operator). For each sampled time step, it samples a standard Gaussian noise To approximate the expected value Each The projection dimension of is k = 4096. should be a reasonable design choice for attributing DDPM, but it is observed that the use of and Constructed φ s It is counterintuitive to always obtain a higher LDS value. and The constructed D-TRAK consistently outperforms TRAK by a large margin.

[0122] Figure 5bis the LDS(%) value on the generated set of CIFAR-2 using the checkpoints of different periods using the data attribution method according to an embodiment of the present invention. It selects 10, 100 and 1000 time steps, which are uniformly distributed in the interval [1, T] to approximate For each selected time step, it samples a standard Gaussian noise to approximate The value of k is set to k = 32768. Curve 510 is a curve when the time step is 10. Curve 520 is a curve when the time step is 100. Curve 530 is a curve when the time step is 1000.

[0123] like Figure 5b As shown, for D-TRAK, the best LDS score is obtained at the final checkpoint. However, for TRAK, using the last checkpoint is not the best choice. Finding which checkpoint produces the best LDS score requires computing the attribution multiple times, which means more computational cost. In fact, it may also be possible to only access the final model.

[0124] In summary, the final model checkpoint is used as the default value when computing the gradients. 10 and 100 time steps are chosen to be evenly spaced in the interval [1, T]. For example, when the number of time steps is 10, the selected time steps are {1, 101.201, ..., 901}. For each time step, it samples a standard Gaussian noise. The projection dimension is k = 32768. Counterfactual Assessment

[0125] To more intuitively evaluate the credibility of D-TRAK, it measures pixel-by-pixel Distance and CLIP cosine similarity. These measurements are made between images generated by the trained model before and after excluding the top 1000 positive influencers, as identified by different attribution methods. For D-TRAK and TRAK, it considers 100 time steps, samples a standard Gaussian noise, and sets k = 32768. A control baseline called random is considered, which randomly removes 1000 training samples before retraining.

[0126] Figure 6 is a box plot of counterfactual evaluations on CIFAR-2 and ArtBench-2 using the data attribution method according to an embodiment of the present invention. It quantifies the impact of removing the 1000 highest scoring training samples and retraining on random, TRAK, and D-TRAK. The pixel-by-pixel distance and CLIP cosine similarity, and the corresponding images generated by the retrained model when sampled from the same random seed.

[0127] When examining the median pixel resulting from removal and retraining When evaluating the median CLIP cosine similarity, D-TRAK yielded a value of 15.07, in contrast to TRAK’s value of 11.02. The opposite trend was observed when evaluating the median CLIP cosine similarity, with D-TRAK obtaining a lower similarity of 0.896, which is significantly lower than TRAK’s 0.942.

[0128] like Figure 6 As shown in , the counterfactual evaluation of D-TRAK on TRAK shows that D-TRAK can better identify influential images that have a large impact on the generated images. Figure 6 As shown, when examining the median pixel resulting from removal and retraining When comparing the distance, D-TRAK produces 8.97, 15.07, and 187.61 for CIFAR-2 and ArtBench-2, respectively, compared to TRAK's values ​​of 5.90 and 168.37. D-TRAK obtains median similarities of 0.881 and 0.769 for the above three datasets, which are significantly lower than TRAK's values ​​of 0.943 and 0.840, highlighting the effectiveness of D-TRAK's method.

[0129] Figure 7 It is a counterfactual visualization on ArtBench-2 using the data attribution method according to an embodiment of the present invention (7a, 7b).

[0130] In the context of the ArtBench-2 dataset, it compares the original generated samples with those generated from the same random seed, after excluding the top 1,000 positive influencers identified by different attribution methods including random, TRAK, and D-TRAK, and trains the model. For D-TRAK and TRAK, it considers 100 time steps to be chosen to be uniformly distributed in the interval [1, T], samples a standard Gaussian noise, and sets k=32768 when computing the gradient.

[0131] like Figure 7 As shown in Figure 2, the results show that the D-TRAK method can better identify influential images that have a significant impact on the target image. Figure 7 Taking a as an example, the retrained models after removing the training samples based on random and TRAK still generate face images, while the model corresponding to D-TRAK generates images that look similar to tables.

[0132] In some embodiments, it classifies training samples with positive influence scores as proponents and samples with negative influence scores as opponents. For samples of interest, the training samples corresponding to the most positive and most negative attribution scores are visualized. It is observed that D-TRAK consistently finds proponents to be more visually similar to target samples.

[0133] In some embodiments, the training samples with the highest self-influence scores on CIFAR-2 and ArtBench-2 are also visualized. n The self-influence is calculated as When calculating the self-influence score, it considers the selected 100 time steps to be uniformly distributed in the interval [1, T], and sets k = 32768. The identified high self-influence samples usually look atypical or unique, while low self-influence samples have similar samples in the training set. Especially on CIFAR-2, high self-influence samples have high visual contrast.

[0134] For the present invention, any suitable programming language can be used to implement the routines of specific embodiments, including C, C++, Java, assembly language, etc. Different programming techniques can be used, such as process-oriented or object-oriented techniques. These routines can be executed on a single processing device or multiple processors.

[0135] Particular embodiments may be implemented in a computer-readable storage medium (also referred to as a machine-readable storage medium) for use with or in conjunction with an instruction execution system, apparatus, system, or device. Particular embodiments may be implemented in the form of control logic in software or hardware or a combination of both. When executed by one or more processors, the control logic may be operated to perform the content described in the particular embodiments.

[0136] Although various aspects and embodiments are disclosed herein, it is apparent that, after reading the foregoing disclosure, various other modifications and adaptations of the present invention will be apparent to those skilled in the art without departing from the spirit and scope of the present invention, and all such modifications and adaptations are within the scope of the appended claims. The various aspects and embodiments disclosed herein are for illustrative purposes only and are not intended to be limiting, and the true scope and spirit of the present invention are indicated by the appended claims.

Claims

1. A method for identifying influential training images in a diffusion model for image generation, comprising the following steps: training the diffusion model using a training image set; generating one or more output images using the trained diffusion model; Calculating a gradient matrix for each image in the training image set; Calculating an importance score for each of the images in the training image set based on the calculated gradient matrix, wherein the importance score indicates the importance of the training image set to the one or more output images; as well as When the importance score exceeds a predetermined threshold, one or more of the training images are identified as the influential training images.

2. The method according to claim 1, wherein the importance score is τ D-TRAK , and is calculated based on the following equation where the score τ D-TRAK Represents training samples How to influence the diffusion model used for the image generation, initially from the training dataset A total of S subsets are sampled in This is the gradient matrix.

3. The method according to claim 2, wherein And the function include or Any of the following, where: in And Avg is the average pooling operation.

4. The method according to any one of claims 1 to 3, wherein the gradient matrix calculation is performed at different training data checkpoints.

5. The method of any one of claims 1 to 4, further comprising identifying a training data checkpoint that contributes most to the generated one or more output images.

6. The method according to any one of claims 1 to 5, wherein the gradient matrix calculation is performed at a final checkpoint. 7 . The method according to claim 1 , wherein the diffusion model is a denoised diffusion probability model (DDPM) or a latent diffusion model (LDM).

8. The method according to any one of claims 1 to 7, further comprising sorting the training image set based on its importance scores to identify the most important images.

9. The method according to claim 8, further comprising a verification step, wherein the verification step comprises: retraining the diffusion model after removing the most significant images identified during the ranking process; Generate images using the retrained model; as well as The pixel-wise l2 distance or contrastive language-image pre-trained CLIP cosine similarity between images generated by the original model and the retrained model was measured to confirm the impact of the most important images being removed.

10. The method of any one of claims 1 to 9, wherein a positive importance score identifies a supporter image and a negative importance score identifies an opponent image in the training image set.

11. The method according to any one of claims 1-10, further comprising displaying the identified influential images and / or their corresponding importance scores.

12. The method according to any one of claims 1 to 11, further comprising removing training images having low importance scores, wherein the low importance scores are below the predetermined threshold; After removing the training images with low importance scores, the diffusion model is fine-tuned or retrained.

13. The method according to claim 12, further comprising The one or more output images are regenerated using the retrained diffusion model.

14. A system for identifying influential training images in a diffusion model for image generation, comprising: processor; a memory in electronic communication with the processor; as well as instructions stored in the memory and executable by the processor to cause the system to: training the diffusion model using a training image set; generating one or more output images using the trained diffusion model; Calculating a gradient matrix for each image in the training image set; Calculating an importance score for each of the images in the training image set based on the calculated gradient matrix, wherein the importance score indicates the importance of the training image set to the one or more output images; as well as When the importance score exceeds a predetermined threshold, one or more of the training images are identified as the influential training images.

15. The system of claim 14, wherein the importance score is τ D-TRAK , and is calculated based on the following equation where the score τ D-TRAK Represents training samples How to influence the diffusion model used for the image generation, initially from the training dataset A total of S subsets are sampled in This is the gradient matrix.

16. The system of claim 15, wherein And the function include or Any of the following, where: in And Avg is the average pooling operation.

17. The system of any one of claims 14-16, wherein the gradient matrix calculation is performed at a final checkpoint.

18. The system of any one of claims 14-17, further comprising displaying the identified influential training images and / or their corresponding influence scores.

19. An apparatus for identifying influential training images in a diffusion model for image generation, comprising: means for training the diffusion model using a training image set; equipment for generating one or more output images using the trained diffusion model; means for computing a gradient matrix for each image in the training image set; means for calculating an importance score for each of the images in the training image set based on the calculated gradient matrix, wherein the importance score indicates an importance of the training image set to the one or more output images; as well as Means for identifying one or more of the training images as the influential training images when the importance score exceeds a predetermined threshold.

20. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, the computer is caused to perform the method according to any one of claims 1 to 13.

Images