Curvature-guided synthetic image dataset distillation method and system

Abstract

This invention discloses a curvature-guided distillation method and system for synthetic image datasets, belonging to the field of image processing technology. To address the technical problems of existing technologies, such as large variance in higher-order statistical estimation under subsampling conditions, uncontrollable computational overhead, and insufficient robustness of synthetic samples, this invention preprocesses real samples and extracts their features and kernel functions along with those of the synthetic samples to be optimized; calculates the sampling weights of the real samples using a sampler; calculates the first-order loss based on the weights and features, and extracts Hessian second-order information to obtain the curvature loss term; synchronously updates the parameters of the synthetic samples and sampler using the first-order loss and the curvature loss term to generate a parent set; finally, the parent set is cropped and corrected. This invention can stabilize higher-order statistical estimation and reduce gradient variance, improve the structural fidelity and generalization ability of synthetic samples, and achieve single distillation adaptability to high-quality datasets of various sizes.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method and system for distilling synthetic image datasets based on curvature guidance. Background Technology

[0002] In the field of efficient data learning in machine learning, dataset distillation aims to compress large-scale training sets into small synthetic datasets. This allows models trained on synthetic datasets to achieve performance as close as possible to those trained on the original datasets, while reducing memory and time overhead. This technique is valuable in scenarios such as hyperparameter search, neural architecture search, model fine-tuning, and rapid fine-tuning on resource-constrained edge devices. Furthermore, because synthetic datasets can be easily shared without exposing the original samples, they have broad application prospects in privacy-sensitive fields such as medical imaging.

[0003] Existing technologies are mainly divided into two paradigms: instance-level feature matching and distribution or gradient-level matching. Instance-level feature matching compresses information by minimizing the representational differences between synthetic samples and their corresponding real samples in the intermediate or output layers of the network. This method is intuitive and easy to implement, but when emphasizing the specific representation of fitting a particular model architecture, it is extremely sensitive to inter-sample variations, easily leading to blurred contour and boundary information, and poor cross-architecture generalization ability and robustness. Distribution or gradient-level matching methods focus on global statistics, maintaining the overall structure and global texture by matching the differences between real and synthetic samples in kernel statistics or gradient distribution. These methods perform well in preserving the overall distribution properties, but in scenarios with low images per class (IPC), the information carrying capacity of synthetic samples is diluted, resulting in lower accuracy than instance-level methods. In addition, distribution-level methods are highly sensitive to hyperparameters such as kernel bandwidth, and the estimation variance of statistics is large under small batch or subsampling conditions, making training prone to oscillations, and their stability and theoretical explanation are insufficient.

[0004] In the representation and generation of synthetic images, the main approaches include pixel-level direct optimization, latent space optimization of the generative model, and parametric rendering. Pixel-level direct optimization updates pixel tensors through backpropagation; while simple to implement, it often generates high-frequency noise and has limited generalization ability. Latent space optimization utilizes a pre-trained generative model as a prior to optimize latent vectors and improve image coherence, but is limited by the coverage domain of the generative model. Parametric rendering offers strong controllability, but is accompanied by rendering costs and domain bias issues. To adapt to different needs, existing technologies propose a parent synthesis set scheme that generates multi-scale subsets through a single distillation; however, these schemes generally face problems such as subset degradation, inter-scale target conflicts, and a lack of unified evaluation standards, making it difficult to balance discriminative power and structural integrity between low and high budgets.

[0005] Regarding sample selection and statistical estimation, existing works employ importance sampling, gradient contribution-based heuristic scoring, or training lightweight learning samplers to dynamically weight real samples to stabilize statistic estimates under subsampling conditions. While learning samplers combined with temperature annealing and entropy regularization can reduce estimation variance, they introduce additional training complexity and hyperparameter sensitivity. Furthermore, for applications of second-order curvature information such as the Hessian matrix, low-rank spectral approximations such as Hutchinson random tracking to estimate the trace, random projection, or Lanzos methods are typically used to obtain main spectrum information or trace values, with curvature regularization introduced to control computational overhead.

[0006] In summary, existing technologies have significant shortcomings in handling the trade-off between low-order and high-order information, the estimation stability of high-order statistics under small batches, and engineering feasibility. In particular, a unified paradigm for using curvature guidance as the core driver of synthetic image optimization and coupling it with learned sampling, multi-bandwidth kernel statistics, multi-scale subset derivation, and approximate curvature calculation systems has not yet been established. Existing schemes suffer from fundamental order conflicts: instance-level matching is prone to overfitting, distribution-level matching suffers signal dilution, and there is a lack of integrated consideration of different resource boundaries. Regarding multi-scale capabilities, performance degradation often occurs when the parent synthetic set is pruned into low IPC subsets, and there is a lack of adaptive strategies to discover the true samples that contribute the most to high-order statistical information. Therefore, how to systematically integrate curvature information to improve the robustness and generalization of synthetic samples while ensuring controllable computational costs, and how to achieve multiple IPC configurations from a single distillation, are urgent technical problems to be solved. Summary of the Invention

[0007] The purpose of this invention is to address the technical problems of existing dataset distillation techniques, such as large variance in higher-order statistical estimation under subsampling conditions, high gradient noise, uncontrollable computational overhead of second-order curvature information, and poor synergy between first-order and higher-order loss terms leading to insufficient robustness of synthesized samples. This invention proposes a curvature-guided optimization method for synthesized image dataset distillation. This method achieves stable estimation of higher-order statistics and reduces gradient variance while ensuring controllable computational costs. It also improves the discriminative power, structural fidelity, and cross-architecture generalization ability of synthesized samples under different IPC configurations, enabling the output of multiple high-quality datasets adaptable to different deployment scenarios from a single distillation.

[0008] To achieve the above objectives, the present invention adopts the following technical solution.

[0009] A curvature-guided distillation method for synthetic image datasets includes the following steps:

[0010] Preprocess the original real training dataset to obtain real samples;

[0011] Feature extraction is performed on the real samples and the synthetic samples to be optimized to obtain real sample features, synthetic sample features, and kernel functions;

[0012] The sampler is used to calculate the sampling weight of each real sample based on the features of the real samples;

[0013] Based on the sampling weights, real sample features, synthetic sample features, and kernel function, calculate the first-order loss of the distribution pairing;

[0014] Based on the first-order loss, Hessian second-order information is extracted to obtain the curvature loss term;

[0015] Using the first-order loss and curvature loss terms, update the parameters of the synthetic sample to be optimized and the sampler to obtain the parent set of the updated synthetic sample;

[0016] The parent set is pruned and the subset distribution is corrected to generate a distillation dataset.

[0017] Further, feature extraction is performed on the real samples and the synthetic samples to be optimized to obtain real sample features, synthetic sample features, and kernel functions, including:

[0018] The real samples and the synthetic samples to be optimized are input into the feature extraction network to extract features from the real samples and features from the synthetic samples.

[0019] A kernel function is constructed based on the Euclidean distance and kernel bandwidth parameters of different sample features.

[0020] Further, the sampler calculates the sampling weight of each real sample based on the features of the real samples, including:

[0021] The sampler maps the features of real samples and outputs an importance score in scalar form.

[0022] The importance score is probabilistically mapped using a normalization function with a temperature parameter, and the temperature parameter is annealed and adjusted during training iterations to obtain the sampling weight of each real sample.

[0023] Further, based on the sampling weights, real sample features, synthetic sample features, and kernel function, the first-order loss of the distribution pairing is calculated, including:

[0024] Based on the sampling weights, real sample features, synthetic sample features, and kernel function, calculate the self-similarity between the real sample distribution and the synthetic sample distribution, as well as the cross-similarity between the real sample and the synthetic sample.

[0025] The distribution matching loss is obtained by summing and differencing the self-similarity and cross-similarity.

[0026] Calculate the Euclidean distance between real sample features and synthetic sample features with the same category label, and combine it with the sampling weights to obtain the pairing loss;

[0027] The first-order loss is composed of the distribution matching loss and the pairing loss.

[0028] Furthermore, Hessian second-order information is extracted based on the first-order loss to obtain the curvature loss term, including:

[0029] The synthetic sample to be optimized is decomposed into channel dimensions, and the Hessian matrix trace response value corresponding to each channel is calculated based on the first-order loss.

[0030] The target channel index is filtered based on the magnitude of the trace response value corresponding to each channel, and the trace response values ​​corresponding to the target channel index are weighted and summed to obtain the curvature estimate.

[0031] The curvature estimate is scaled to obtain the curvature loss as a curvature loss term.

[0032] Furthermore, based on the first-order loss, Hessian second-order information is extracted to obtain the curvature loss term, which also includes:

[0033] Based on the first-order loss, construct the Hessian matrices corresponding to the real sample features and the synthetic sample features, respectively;

[0034] Extract the principal eigenvalues ​​or principal subspace basis of each Hessian matrix, and calculate the difference in principal eigenvalues ​​or the projection error of the principal subspace between the real sample and the synthetic sample to obtain the curvature distribution loss as the curvature loss term.

[0035] Furthermore, using the first-order loss and curvature loss terms, the parameters of the synthesized sample to be optimized and the sampler are updated to obtain a parent set composed of the updated synthesized samples, including:

[0036] Calculate adaptive scheduling weights based on the current number of training steps, and use the adaptive scheduling weights to perform linear weighted fusion of the first-order loss and curvature loss terms to construct the overall loss.

[0037] Based on the gradient of the overall loss and by introducing a curvature-related gradient scaling factor, update the parameters of the synthetic sample to be optimized and the sampler.

[0038] Summarize all the synthesized samples updated within the current iteration cycle to generate a parent set that serves as the subset for selecting the candidate pool.

[0039] Furthermore, using the first-order loss and curvature loss terms, the parameters of the synthesized sample to be optimized and the sampler are updated to obtain a parent set composed of the updated synthesized samples, including:

[0040] The sensitivity index of each real sample to curvature matching is calculated based on the first-order loss.

[0041] The sensitivity index is incorporated as a compensation term into the optimization objective of the sampler, and the sampling weight of each real sample is updated by adjusting the sampler parameters.

[0042] The synthesized samples to be optimized are iteratively updated based on the updated sampling weights to generate the parent set. Further, the parent set is pruned and its subset distribution corrected to generate a distillation dataset, including:

[0043] Sort the samples in the parent set and extract an initial subset that meets the preset size;

[0044] Extract the feature distribution summary information of the parent set, and incorporate the summary information as a compensation term into the first-order loss of the initial subset;

[0045] The distribution offset of the initial subset is corrected by fine-tuning, generating the distillation dataset as the final output.

[0046] A curvature-guided distillation system for synthetic image datasets includes:

[0047] The data preprocessing module is used to preprocess the original real training dataset to obtain real samples;

[0048] The feature extraction module is used to extract features from the real sample and the synthetic sample to be optimized, so as to obtain real sample features, synthetic sample features and kernel function;

[0049] The sampler module is used to calculate the sampling weight of each real sample based on the features of the real samples;

[0050] The distribution matching module is used to calculate the first-order loss of distribution pairing based on the sampling weights, real sample features, synthetic sample features, and kernel function.

[0051] The curvature processing module is used to extract Hessian second-order information based on the first-order loss to obtain the curvature loss term.

[0052] The joint optimization module is used to update the parameters of the synthetic sample to be optimized and the sampler using the first-order loss and curvature loss terms, so as to obtain the parent set of the updated synthetic samples.

[0053] The dataset refining module is used to prune the parent set and correct the subset distribution to generate a distilled dataset.

[0054] The present invention has achieved the following beneficial effects.

[0055] 1. This invention employs a curvature-guided dynamic multi-order hybrid loss (i.e., overall loss) and scheduling mechanism. By adaptively scheduling weights, it emphasizes first-order loss in the early stages of training to ensure rapid discriminative guidance. In the later stages, it gradually amplifies the regularization constraint based on the curvature loss term. By utilizing curvature information to adjust the sensitivity of the synthesized samples to parameter perturbations, it forms a flatter loss terrain in the parameter space. This effectively alleviates the contradiction of overfitting or signal dilution caused by a single first-order loss target, and significantly improves the robustness and cross-architecture transferability of the synthesized samples.

[0056] 2. This invention is based on a learning sampler and a curvature sensitivity weighting mechanism. It uses a lightweight sampler to score and map real sample features and curvature sensitivity indices. Under subsampling conditions, it prioritizes the retention of representative samples that are most critical to matching higher-order distributions, thereby improving the signal-to-noise ratio of higher-order statistical estimation. It also reduces the gradient variance caused by random sampling from a mechanistic perspective, improves training stability and accelerates convergence, and reduces the sensitivity of distribution matching to kernel bandwidth and sampling variations.

[0057] 3. This invention employs an engineered curvature approximation and frequency control strategy. By using Hutchinson random tracking, main spectrum approximation, or low-rank projection, combined with adjustable execution frequency and channel-level pruning, it transforms the incomputable Hessian information in high-dimensional space into inexpensive and controllable second-order Hessian information. Without significantly increasing computational and storage overhead, it introduces high-order geometric constraints, solving the problem that second-order information is difficult to implement in deep model optimization due to its high cost.

[0058] 4. This invention constructs a multi-bandwidth kernel statistical reweighting strategy to dynamically adjust the kernel channels at different scales based on the curvature sensitivity of the samples. This enhances the coordination ability of distribution matching in terms of structural details and global consistency, further weakens the influence of kernel bandwidth hyperparameters on distillation performance, and enables the synthesized samples to effectively respond to curvature guidance information at different scales.

[0059] 5. This invention employs a derivation and global compensation mechanism from the parent set to the subset. It identifies key synthetic samples in the parent set by curvature sensitivity sorting and introduces the parent set summary as a compensation term during the pruning process to correct the distribution offset. This enables a single distillation task to adapt to various IPC scale configurations, significantly reducing the cost of repeated distillation in multi-scenario deployments and ensuring the generalization performance and fine-tuning accuracy of small subsets in environments with limited computing power. Attached Figure Description

[0060] Figure 1 This is a simplified flowchart of the curvature-guided synthetic image dataset distillation method in the embodiments.

[0061] Figure 2This is a block diagram of the curvature-guided synthetic image dataset distillation system in the embodiment. Detailed Implementation

[0062] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, detailed descriptions are provided below through embodiments.

[0063] This invention provides a curvature-guided distillation method for synthetic image datasets, such as... Figure 1 As shown, the method flow is as follows:

[0064] Step S1: Preprocess the original real training dataset to obtain real samples.

[0065] Specifically, the original real training dataset is denoted as... ,in Indicates the first The dataset consists of N real samples, which can be images, signals, or other digitally representable data types, where N represents the total number of real samples. In each round of the external optimization loop, samples are generated from the original real training dataset. Small batches of real samples are extracted according to category and used in the current iteration of the synthesis optimization process. The real samples selected from the original real training dataset are subjected to numerical scale normalization and format alignment; data augmentation operations are then performed on the processed real samples to obtain real samples with a consistent data distribution.

[0066] In an optional embodiment of the present invention, the data augmentation operation can be implemented in a differentiable or non-differentiable manner, aiming to ensure that the real samples have a stable numerical range in subsequent feature space mapping, distribution matching, and curvature estimation processes, thereby avoiding training instability caused by differences in input scale. The preprocessed real samples are organized into batch data and used as real data references in feature extraction, loss calculation, and synthetic sample optimization.

[0067] Step S2: Extract features from the real sample and the synthetic sample to be optimized to obtain the real sample features, the synthetic sample features, and the kernel function.

[0068] In an optional embodiment of the present invention, step S2 may include:

[0069] Step S21: Input the real sample and the synthetic sample to be optimized into the feature extraction network to extract the features of the real sample and the synthetic sample.

[0070] Step S22: Construct a kernel function based on the Euclidean distance and kernel bandwidth parameters of different sample features.

[0071] Specifically, the set of synthetic samples to be optimized is denoted as . ,in Indicates the first There are 10 synthetic samples to be optimized, which can be represented as pixel tensors, latent vectors or parameterized representations, and M represents the total number of synthetic samples to be optimized.

[0072] Feature extraction network denoted as This is used to map input samples to a feature space. Real samples and synthetic samples to be optimized are respectively mapped to the feature space through a feature extraction network to obtain the features of the real samples. Features of synthetic samples The kernel function is defined as follows:

[0073]

[0074] in, B represents the kernel function value; B represents the kernel bandwidth. Indicates the first Individual core bandwidth parameters; This represents the squared Euclidean distance between two sample features. By weighted summation over multiple bandwidth kernels, it simultaneously captures the distributional differences at different scales in the feature space.

[0075] In an optional embodiment of the present invention, the parameters of the feature extraction network can remain fixed or be updated in stages according to a preset strategy during the sample synthesis optimization process. The feature representations generated by the feature extraction network are used for subsequent distribution-level loss calculation, instance-level pairing loss calculation, and the scoring process of the sampler.

[0076] Step S3: Use the sampler to calculate the sampling weight of each real sample based on the features of the real samples.

[0077] In an optional embodiment of the present invention, step S3 may include:

[0078] Step S31: Map the features of the real samples using a sampler and output an importance score in scalar form;

[0079] Specifically, the sampler is denoted as ,in This represents the learnable parameters of the sampler itself. For each real sample... The corresponding real sample features Importance scores are obtained by feeding the sampler. This is used to measure the importance or information content of the sample in the current training phase.

[0080] Step S32: The importance score is probabilistically mapped using a normalization function with temperature parameters, and the temperature parameters are annealed and adjusted during training iterations to obtain the sampling weights of each real sample.

[0081] Specifically, to convert the above importance scores into a form that can be used for weighting, a softmax normalization operation with a temperature parameter is introduced to calculate the sampling weights. The formula for calculating the sampling weights is as follows:

[0082]

[0083] in, Indicates the first The sampling weights of each real sample in the current iteration; This indicates the corresponding importance score; This indicates a temperature parameter. An annealing strategy that gradually decreases with training iterations is adopted, so that the sampler gradually focuses on real samples with high discriminative value or high-order information in the later stages of training.

[0084] In an optional embodiment of the present invention, during the training process of the sampler, the sampler parameters... Optimization is achieved by maximizing the distribution matching loss, while an entropy regularization term is introduced to prevent weight collapse, combined with an auxiliary term related to curvature sensitivity. This joint optimization allows the sampler-generated weights to prioritize real samples containing more higher-order information in the curvature-dominant direction, thus providing a sample base with a higher signal-to-noise ratio for subsequent curvature estimation and higher-order constraints.

[0085] In an optional embodiment of the invention, the sampler can be based on a deep neural network structure, or it can be replaced with a reinforcement learning-based sampler or a fixed heuristic sampler. Through this learning-based sampling mechanism, real samples are dynamically assigned different sampling weights, thereby highlighting more representative samples during the distribution matching process.

[0086] Step S4: Calculate the first-order loss of the distribution pairing based on the sampling weights, real sample features, synthetic sample features, and kernel function.

[0087] In an optional embodiment of the present invention, step S4 may include:

[0088] Step S41: Based on sampling weights, real sample features, synthetic sample features, and kernel function, calculate the self-similarity between the real sample distribution and the synthetic sample distribution, as well as the cross-similarity between the real sample and the synthetic sample.

[0089] In an optional embodiment of the present invention, step S41 may include:

[0090] Step S411: Weight the kernel function values ​​between real sample features based on the sampling weights to obtain the self-similarity of the real sample distribution.

[0091] Step S412: Calculate the mean kernel function among the features of the synthesized samples to obtain the self-similarity of the synthesized sample distribution.

[0092] Step S413: Calculate the weighted mean of the cross kernel function between the real sample features and the synthetic sample features based on the sampling weights to obtain the cross similarity.

[0093] Step S42: Sum and difference calculations are performed on the self-similarity and cross-similarity to obtain the distribution matching loss.

[0094] Specifically, the formula for calculating the distribution matching loss is as follows:

[0095]

[0096] in, Represents the distribution matching loss; Represents the set of real samples; Represents a composite sample set; Represents real samples The assigned sampling weights, and satisfying and ; Represents the kernel function values ​​between real samples; Represents the kernel function values ​​between synthesized samples; The cross-kernel function value represents the value between real and synthetic samples; M represents the total number of synthetic samples. The three terms in the above formula correspond to the weighted self-similarity of the real sample distribution, the self-similarity of the synthetic sample distribution, and the cross-similarity between real and synthetic samples, respectively.

[0097] Step S43: Calculate the Euclidean distance between real sample features and synthetic sample features with the same category label, and combine it with the sampling weights to obtain the pairing loss.

[0098] Step S44: The first-order loss is composed of the distribution matching loss and the pairing loss.

[0099] Specifically, the formula for calculating the pairing loss is as follows:

[0100]

[0101] in, Indicates pairing loss; Represents the characteristics of real samples; This indicates the characteristics of synthetic samples with the same category label; This represents the square of the Euclidean distance. Sampling weights can be further introduced in the implementation. The above expectations are weighted.

[0102] In an optional embodiment of the invention, the overall objective can be formalized as solving over a set of trainable synthetic samples to minimize a temporally adaptive mixture of first-order and second-order composite objectives during the training process. The first-order loss term aims to constrain the synthetic samples to maintain consistency with real samples at the statistical distribution and instance level. A first-order optimization objective driving the updating of synthetic samples is formed through the joint calculation of distribution loss and pairing loss. This objective exists in parallel with subsequently introduced higher-order curvature constraints, and their relative weights are adjusted by a dynamic scheduling strategy at different training stages.

[0103] Through the aforementioned learning-based sampling mechanism, real samples are dynamically assigned different sampling weights. This weight will be used uniformly in subsequent distribution-level loss calculations, instance-level pairing loss calculations, and curvature processing modules, thus forming a closed-loop sample selection and optimization mechanism throughout the entire method flow. The pairing loss minimizes the difference between synthetic samples and their corresponding real samples in the feature space, providing a stable discrimination signal for synthetic samples in the early stages of training and preventing degradation caused by relying solely on distribution matching. The weighted distribution-level loss, by weighting relevant terms of real samples, stably characterizes distribution differences under subsampling or mini-batch conditions.

[0104] Step S5: Extract Hessian second-order information based on the first-order loss to obtain the curvature loss term.

[0105] In an optional embodiment of the present invention, step S5 may include:

[0106] Step S51: Decompose the synthetic sample to be optimized into channel dimensions and calculate the Hessian matrix trace response value corresponding to each channel based on the first-order loss.

[0107] Specifically, the curvature estimation module is introduced to characterize the second-order sensitivity of the synthetic sample to be optimized in the current optimization state. Let the scalar loss currently used for optimization be denoted as... The batch of synthetic samples to be optimized that participates in the loss calculation is denoted as Each element corresponds to a pixel representation, latent vector representation, or parameterized representation of a synthetic sample. The goal of curvature estimation is to measure the loss function. Compared to the second-order transformation properties of the synthetic sample representation to be optimized, i.e., the Hessian matrix:

[0108]

[0109] in, Describe the overall objective function The second-order partial derivative matrix relative to the synthetic sample to be optimized is used to describe the local curvature characteristics of the loss function surface; This represents the corresponding second derivative matrix.

[0110] Since it is not feasible to directly compute the complete Hessian matrix in high-dimensional space, this method uses Hutchinson stochastic estimation, Gauss-Newton approximation, Fisher information matrix substitution, or low-rank spectral approximation for curvature estimation.

[0111] In an optional embodiment of the present invention, step S51 may include:

[0112] Step S511: Sample probe vectors from a preset distribution and calculate the inner product of the first-order gradient of the first-order loss with respect to the channel and the probe vector.

[0113] Step S512: Calculate the gradient of the inner product with respect to the channel to obtain the product of the Hessian matrix of the corresponding channel and the probe vector.

[0114] Step S513: Calculate the expected value of the product of the product and the probe vector to obtain the curvature trace response value characterizing the local curvature intensity.

[0115] Specifically, the Hutchinson estimation method is used by introducing random probe vectors. The trace of the Hessian matrix is ​​approximated by the following basic form:

[0116]

[0117] in, Represents the loss function Regarding the representation of the synthesized samples to be optimized The first gradient; This represents the product of the Hessian matrix and a vector, i.e., the Hessian matrix along a random direction. The projection of the probe vector. Each element is sampled independently from the Rademacher distribution, i.e., it takes the value +1 or -1 with the same probability, thus ensuring that the estimate is an unbiased estimate of the Hessian matrix trace.

[0118] To reduce the variance of random estimates, this method... Independent random probe vectors Calculate the second-order response in each direction, and average the results to obtain a numerically stable curvature estimate:

[0119]

[0120] in, This indicates the number of random probes.

[0121] In an optional embodiment of the invention that supports fine-grained estimation by channel, the synthetic sample representation to be optimized is... Decomposed into channels , No. The curvature trace response value corresponding to each channel Defined as:

[0122]

[0123] in, Indicates and channel The corresponding random probe vector. The above channel-level curvature metric is used to characterize the relative influence of different channels on the overall curvature.

[0124] Step S52: Filter the target channel index according to the numerical value of the trace response value corresponding to each channel, and perform a weighted summation of the trace response values ​​corresponding to the target channel index to obtain the curvature estimate.

[0125] Specifically, the curvature estimate ultimately used for regularization The definition is as follows:

[0126]

[0127] in, Represents the set of channel indices that are reserved; The weight coefficients (binary mask or continuous weights) are for the corresponding channels.

[0128] Step S53: Scale compensation is performed on the curvature estimate to obtain the curvature loss as the curvature loss term.

[0129] Specifically, when only a portion of the channels are retained, a scale compensation factor can be introduced to maintain overall consistency in magnitude, ultimately resulting in a curvature loss. Represented as:

[0130]

[0131] in, The curvature regularization weights are used to adjust the strength of higher-order constraints in the overall objective function.

[0132] In another optional embodiment of the present invention, step S5 may also include:

[0133] Step S51: Construct Hessian matrices corresponding to real sample features and synthetic sample features based on the first-order loss;

[0134] Step S52: Extract the principal eigenvalues ​​or principal subspace basis of each Hessian matrix, and calculate the difference in principal eigenvalues ​​or the projection error of the principal subspace between the real sample and the synthetic sample to obtain the curvature distribution loss as the curvature loss term.

[0135] Specifically, curvature estimation is achieved through a low-rank spectral approximation of the parameter space or feature space. In this implementation, the curvature distribution loss is used as the curvature loss term. This can be represented as the difference between principal eigenvalues:

[0136]

[0137] Or it can be expressed as the projection error between the master and subspaces:

[0138]

[0139] in, and These represent the principal eigenvalues ​​of the Hessian matrix corresponding to the real data and the synthetic data, respectively. and This represents the basis of the subspace spanned by the principal eigenvectors; It is the Frobenius norm. This scheme enables the synthetic sample to capture the curvature direction of the real sample in the feature space by aligning the higher-order feature spectra.

[0140] Step S6: Using the first-order loss and curvature loss terms, update the parameters of the synthetic sample to be optimized and the sampler to obtain the parent set of the updated synthetic samples.

[0141] In an optional embodiment of the present invention, step S6 may include:

[0142] Step S61: Calculate the adaptive scheduling weights based on the current number of training steps, and use the adaptive scheduling weights to linearly weight and fuse the distribution matching loss, pairing loss, and curvature loss to construct the overall loss.

[0143] Specifically, the formula for constructing the overall loss is as follows:

[0144]

[0145] in, Indicates the current number of training steps The overall loss; Represents the distribution matching loss; Indicates pairing loss; Represents the curvature loss term; This represents the adaptive scheduling weights that adapt to the training progress. This represents the curvature regularization weight. This hybrid objective allows the synthesized samples to approximate the real data distribution in terms of feature statistics, category structure, and discriminative behavior.

[0146] Step S62: Based on the gradient of the overall loss and by introducing a curvature-related gradient scaling factor, update the parameters of the synthetic sample to be optimized and the sampler.

[0147] In an optional embodiment of the invention, the gradient scaling factor is used to implement an update strategy with curvature modulation. In the gradient direction... Given the given information, an overly aggressive update in high curvature directions is suppressed through curvature-dependent scaling or pruning mechanisms to prevent the synthesized sample from collapsing towards a sharp minimum. This update method ensures that the step size and direction of the synthesized sample to be optimized are simultaneously constrained by both the loss gradient and the local curvature structure.

[0148] Specifically, the synthesized sample to be optimized is the core optimizable variable in the distillation process. Its specific form can be a synthesized image in pixel space or a learnable representation in latent space. In this step, the pixel representation (or latent vector representation) of the synthesized sample to be optimized is correlated with the learnable parameters of the sampler using the gradient backpropagation algorithm. Synchronous optimization.

[0149] Step S63: Summarize all the updated synthetic samples in the current iteration cycle to generate a parent set as a subset for candidate pooling.

[0150] In an optional embodiment of the present invention, the parent set is denoted as... It consists of all the synthesized samples updated by curvature modulation in the current stage. The parent set acts as an intermediate carrier and buffer, providing a structured and stable candidate pool for subsequent subset selection and distribution compensation, ensuring that distillation data subsets that meet different scale constraints are constructed without destroying the overall distribution structure.

[0151] In another optional embodiment of the present invention, step S6 may also include:

[0152] Step S61: Calculate the sensitivity index of each real sample to curvature matching based on the first-order loss.

[0153] Step S62: The sensitivity index is incorporated as a compensation term into the optimization objective of the sampler, and the sampling weight of each real sample is updated by adjusting the sampler parameters.

[0154] Step S63: Iteratively update the synthetic sample to be optimized based on the updated sampling weights to generate the parent set.

[0155] Specifically, curvature information is incorporated into the sampler's optimization objective function. Through joint optimization with a learned sampler, the sampler acquires bias weights, thus favoring the selection of real samples that contain rich high-order information in the curvature-dominant direction. This mechanism, through curvature-aware subsampling, can improve the ability of synthetic samples to recover high-order structural information while reducing computational overhead.

[0156] This step, by applying the results of loss design and curvature modeling to the sample level, realizes the transformation from a continuous optimization process to a structured sample set, playing a pivotal role in the methodology.

[0157] Step S7: Prune the parent set and correct the subset distribution to generate the distillation dataset.

[0158] In an optional embodiment of the present invention, step S7 may include:

[0159] Step S71: Sort the samples in the parent set and extract the initial subset that meets the preset size.

[0160] Specifically, the preset size is determined by the IPC budget. Represents a set of categories, for any category The corresponding synthetic sample subset in the mother set is denoted as During the initial subset construction process, the samples in the parent set are sorted according to the learning-based sampling preferences, curvature sensitivity, or loss contribution information implicit in the previous steps, thereby identifying and extracting representative samples that have a key impact on the model's discrimination boundary.

[0161] Step S72: Extract the feature distribution summary information of the parent set and incorporate the summary information as a compensation term into the first-order loss of the initial subset.

[0162] In an optional embodiment of the present invention, subset distribution correction is achieved through a global compensation strategy. When the initial subset exhibits under-coverage in a local region, the feature distribution summary information of the parent set is extracted as an additional weighting term and incorporated into the first-order loss function.

[0163] Step S73: Correct the distribution offset of the initial subset through fine-tuning to generate the distillation dataset as the final output.

[0164] By performing gradient updates during the fine-tuning or evaluation phases, the distribution shift caused by pruning is corrected. Specifically, this method employs a subset compensation mechanism, the core of which is to prioritize the retention of synthetic instances that play a structural supporting role in the training dynamics, such as samples that are sampled multiple times at high frequencies or have a significant marginal contribution to loss reduction. In this way, while maintaining the subset size to satisfy the IPC constraint, its internal composition is reconstructed, making it closer to the statistical properties of the parent set in the feature space.

[0165] In an optional embodiment of the present invention, the generated final distillation dataset is denoted as... This dataset serves as the final output of this methodology and is used for training and evaluating downstream models. Because its construction process fully inherits the optimization results of the aforementioned loss design, curvature constraints, and sample update strategies, this distillation dataset maintains stable generalization performance even at a relatively small scale.

[0166] This step transforms the continuous synthesis optimization process into a discrete, directly usable distillation dataset, marking the completion of the entire distillation task.

[0167] This invention also provides a curvature-guided distillation system for synthetic image datasets, such as... Figure 2 As shown, it includes:

[0168] The data preprocessing module is used to preprocess the original real training dataset to obtain real samples;

[0169] The feature extraction module is used to extract features from real samples and synthetic samples to be optimized, and to obtain real sample features, synthetic sample features and kernel functions.

[0170] The sampler module is used to calculate the sampling weight of each real sample based on the features of the real samples;

[0171] The distribution matching module is used to calculate the first-order loss of distribution pairing based on sampling weights, real sample features, synthetic sample features, and kernel functions.

[0172] The curvature processing module is used to extract Hessian second-order information based on the first-order loss to obtain the curvature loss term.

[0173] The joint optimization module is used to update the parameters of the synthetic sample to be optimized and the sampler using the first-order loss and curvature loss terms, so as to obtain the parent set of the updated synthetic sample.

[0174] The dataset refining module is used to prune the parent set and correct the subset distribution to generate a distilled dataset.

[0175] Experimental test:

[0176] 1. Experimental conditions and parameter settings:

[0177] In an optional embodiment of the present invention, performance testing and parameter configuration were performed on the curvature-guided dataset distillation framework. The specific experimental conditions were set as follows:

[0178] Second-order approximation parameters: The number of Hutchinson random probes is set to 1-3 to strike a balance between estimation variance and computational burden. For channel-level pruning, the default retention ratio is set to 0.5, and the top-k mode is used for channel filtering.

[0179] Calculation frequency and weight control: The curvature calculation frequency is set to trigger once every 1–10 outer loop iterations by default, and is dynamically adjusted upwards based on model size and training budget. Curvature regularization term weights. Initially set to a minimum value (e.g., 1e-6), and combined with the dynamic scheduling weight. Execute the coordinated scheduling strategy.

[0180] Algorithm switching and fault tolerance mechanism: For high-resolution images or large pre-trained models, low-rank spectral approximation of the parameter space is preferred; for low-resolution or small models, pixel-level Hutchinson estimation is used. When the system's second-order automatic differentiation module returns an anomaly, a fault tolerance fallback mechanism is activated, switching to the first-order approximation or reusing historical curvature results.

[0181] Complexity control: The Hutchinson approximation incurs approximately twice the computational cost of a single "forward + backward" propagation for a single batch, and is only performed on small synthetic sub-batches, keeping the overall computational cost within a manageable range. The low-rank spectral approximation of the parameter space is performed at sparse frequencies, and a cache reuse mechanism is introduced to reduce computational pressure.

[0182] 2. Experimental Conclusion:

[0183] Experimental results show that the method of this invention, while retaining the advantages of weighted distribution matching and instance pairing, significantly improves the quality of the distillation dataset by introducing second-order curvature constraints and channel-level refinement strategies. Specifically, in low IPC scenarios, the learning sampler coupled with curvature effectively identifies and retains difficult samples that contribute significantly to the discrimination boundary, improving the discriminative power of the synthesized data. In high IPC scenarios, a compensation strategy from the parent set to the subset is used to correct the distribution shift and maintain the fidelity of the data structure. Experiments demonstrate that, through dynamic scheduling and sparsity frequency setting, this method can still achieve curvature guidance with controllable computational overhead under limited computing power, forming a unified distillation paradigm with practical value.

[0184] The parts not described in detail are well-known technologies to those skilled in the art.

[0185] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. A curvature-guided distillation method for synthetic image datasets, characterized in that, Includes the following steps: Preprocess the original real training dataset to obtain real samples; Feature extraction is performed on the real samples and the synthetic samples to be optimized to obtain real sample features, synthetic sample features, and kernel functions; The sampler is used to calculate the sampling weight of each real sample based on the features of the real samples; Based on the sampling weights, real sample features, synthetic sample features, and kernel function, the first-order loss of distribution pairing is calculated, including: calculating the self-similarity between the real sample distribution and the synthetic sample distribution, and the cross-similarity between the real sample and the synthetic sample, based on the sampling weights, real sample features, synthetic sample features, and kernel function; summing and differencing the self-similarity and cross-similarity to obtain the distribution matching loss; calculating the Euclidean distance between real sample features and synthetic sample features with the same category label, and combining it with the sampling weights to obtain the pairing loss; the first-order loss is composed of the distribution matching loss and the pairing loss. Based on the first-order loss, Hessian second-order information is extracted to obtain the curvature loss term; Using the first-order loss and curvature loss terms, update the parameters of the synthetic sample to be optimized and the sampler to obtain the parent set of the updated synthetic sample; The parent set is pruned and the subset distribution is corrected to generate a distillation dataset.

2. The method as described in claim 1, characterized in that, Feature extraction is performed on the real samples and the synthetic samples to be optimized to obtain real sample features, synthetic sample features, and kernel functions, including: The real samples and the synthetic samples to be optimized are input into the feature extraction network to extract features from the real samples and features from the synthetic samples. A kernel function is constructed based on the Euclidean distance and kernel bandwidth parameters of different sample features.

3. The method as described in claim 1, characterized in that, The sampler calculates the sampling weight of each real sample based on the features of the real samples, including: The sampler maps the features of real samples and outputs an importance score in scalar form. The importance score is probabilistically mapped using a normalization function with a temperature parameter, and the temperature parameter is annealed and adjusted during training iterations to obtain the sampling weight of each real sample.

4. The method as described in claim 1, characterized in that, Based on the first-order loss, Hessian second-order information is extracted to obtain the curvature loss term, including: The synthetic sample to be optimized is decomposed into channel dimensions, and the Hessian matrix trace response value corresponding to each channel is calculated based on the first-order loss. The target channel index is filtered based on the magnitude of the trace response value corresponding to each channel, and the trace response values ​​corresponding to the target channel index are weighted and summed to obtain the curvature estimate. The curvature estimate is scaled to obtain the curvature loss as a curvature loss term.

5. The method as described in claim 1, characterized in that, Based on the first-order loss, Hessian second-order information is extracted to obtain the curvature loss term, which also includes: Based on the first-order loss, construct the Hessian matrices corresponding to the real sample features and the synthetic sample features, respectively; Extract the principal eigenvalues ​​or principal subspace basis of each Hessian matrix, and calculate the difference in principal eigenvalues ​​or the projection error of the principal subspace between the real sample and the synthetic sample to obtain the curvature distribution loss as the curvature loss term.

6. The method as described in claim 1, characterized in that, Using the first-order loss and curvature loss terms, the parameters of the synthesized sample to be optimized and the sampler are updated to obtain a parent set of the updated synthesized samples, including: Calculate adaptive scheduling weights based on the current number of training steps, and use the adaptive scheduling weights to perform linear weighted fusion of the first-order loss and curvature loss terms to construct the overall loss. Based on the gradient of the overall loss and by introducing a curvature-related gradient scaling factor, update the parameters of the synthetic sample to be optimized and the sampler. Summarize all the synthesized samples updated within the current iteration cycle to generate a parent set that serves as the subset for selecting the candidate pool.

7. The method as described in claim 1, characterized in that, Using the first-order loss and curvature loss terms, the parameters of the synthesized sample to be optimized and the sampler are updated to obtain a parent set of the updated synthesized samples, including: The sensitivity index of each real sample to curvature matching is calculated based on the first-order loss. The sensitivity index is incorporated as a compensation term into the optimization objective of the sampler, and the sampling weight of each real sample is updated by adjusting the sampler parameters. The samples to be optimized are iteratively updated based on the updated sampling weights to generate the parent set.

8. The method as described in claim 1, characterized in that, The parent set is pruned and subset distribution corrected to generate a distillation dataset, including: Sort the samples in the parent set and extract an initial subset that meets the preset size; Extract the feature distribution summary information of the parent set, and incorporate the summary information as a compensation term into the first-order loss of the initial subset; The distribution offset of the initial subset is corrected by fine-tuning, generating the distillation dataset as the final output.

9. A curvature-guided distillation system for synthesized image datasets, characterized in that, include: The data preprocessing module is used to preprocess the original real training dataset to obtain real samples; The feature extraction module is used to extract features from the real sample and the synthetic sample to be optimized, so as to obtain real sample features, synthetic sample features and kernel function; The sampler module is used to calculate the sampling weight of each real sample based on the features of the real samples; The distribution matching module is used to calculate the first-order loss of distribution pairing based on the sampling weights, real sample features, synthetic sample features, and kernel function. This includes: calculating the self-similarity between the real sample distribution and the synthetic sample distribution, and the cross-similarity between the real samples and the synthetic samples, based on the sampling weights, real sample features, synthetic sample features, and kernel function; summing and differencing the self-similarity and cross-similarity to obtain the distribution matching loss; calculating the Euclidean distance between real sample features and synthetic sample features with the same category label, and combining this distance with the sampling weights to obtain the pairing loss; and combining the distribution matching loss and the pairing loss to form the first-order loss. The curvature processing module is used to extract Hessian second-order information based on the first-order loss to obtain the curvature loss term. The joint optimization module is used to update the parameters of the synthetic sample to be optimized and the sampler using the first-order loss and curvature loss terms, so as to obtain the parent set of the updated synthetic samples. The dataset refining module is used to prune the parent set and correct the subset distribution to generate a distilled dataset.

Images