A dataset distillation method based on representative sample feature distribution matching
By selecting representative samples and optimizing using feature distribution matching and discriminative loss, the problem of unstable training of existing dataset distillation algorithms on large datasets is solved, achieving the generation of high-quality synthetic samples and good generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIDIAN UNIV
- Filing Date
- 2024-06-05
- Publication Date
- 2026-05-29
Smart Images
Figure CN118887486B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of dataset distillation technology, specifically relating to a dataset distillation method based on representative sample feature distribution matching. Background Technology
[0002] Currently, deep learning largely relies on the sheer volume of data used to train deep neural networks. However, this massive amount of data increases the burden of storage and transmission, further complicating the model training process. To alleviate these drawbacks, obtaining a synthetic dataset with high information density is undoubtedly a good choice. Dataset distillation algorithms replace the original dataset with a high-quality synthetic dataset while retaining a sufficiently small number of samples in each class. The goal is to accelerate network training, reduce data storage, and avoid potential data privacy issues during transmission.
[0003] Existing dataset distillation algorithms mainly include:
[0004] (1) Gradient matching algorithm
[0005] Gradient-matching algorithms update the synthetic dataset by minimizing the gradient-matching loss at each training step. However, this method introduces high bias with increasing iterations, resulting in significant bias in the synthetic dataset learned through gradient matching, especially for samples with large gradients. This leads to performance degradation across different models.
[0006] (2) Algorithm based on trajectory matching
[0007] To overcome the bias accumulation problem caused by gradient matching, trajectory matching-based algorithms propose using expert models and student models to learn from real and synthetic datasets respectively, continuously fitting the training trajectories of real and synthetic samples during training. While this method can obtain high-quality synthetic samples, it is quite sensitive to hyperparameters. Furthermore, it is only suitable for small datasets, cannot be scaled to large datasets, and is not applicable to high compression ratios.
[0008] (3) Regularization-based algorithms
[0009] Regularization-based algorithms utilize regularization methods to guide the synthesis process, enhancing image interpretability by learning hidden patterns and controlling the diversity of synthesized data. A pre-trained decoder can be used to obtain the encoding of latent vectors, and then the low-dimensional compressed encoding can be restored to the image pixel space. However, this algorithm is difficult to train due to its large number of model parameters and high hardware requirements. Furthermore, it cannot be scaled to large classification datasets such as ImageNet.
[0010] The invention, entitled "A Robust Dataset Distillation Method and System," with application publication number CN 115761414A, proposes a robust dataset distillation method, firstly...
[0011] Random sampling is performed on the real dataset to obtain an initialized synthetic dataset; then, random sampling is also performed on the real dataset to obtain an initialized real data sample set; adversarial perturbation is added to this initialized real data sample set to obtain a perturbed sample set; the feature representations of this perturbed sample set and the synthetic dataset are calculated respectively, and the difference between the feature representations is used as the loss function; the synthetic dataset is optimized and updated by backpropagation based on the loss function, and the optimization process is iterated until the loss function converges, and the current distillation dataset is saved as the final robust distillation dataset.
[0012] This method uses a random sampling initialization method; however, this may lead to deviations in the matching target and introduce some noise and large gradient samples. Summary of the Invention
[0013] To overcome the shortcomings of existing technologies, this invention aims to provide a dataset distillation method based on representative sample feature distribution matching. This method first selects representative samples for initialization, then uses a parameterized differential Siamese augmentation strategy to augment these representative samples, and finally obtains highly interpretable synthetic samples by optimizing the feature distribution matching loss and discriminant loss between the real and synthetic sample levels. This method significantly accelerates the entire training process and ensures its stability. It is applicable to a wide range of datasets, yields high-quality synthetic samples, and exhibits good generalization ability across models.
[0014] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0015] A dataset distillation method based on representative sample feature distribution matching includes the following steps:
[0016] Step 1: Use the k-means algorithm to select representative real and synthetic samples for initialization;
[0017] Real samples refer to images in the original dataset, while synthetic samples refer to representative images generated from the original dataset after compression algorithms. In the first round, representative synthetic samples are the cluster centers obtained from the real dataset using the k-means algorithm based on the set IPC (Image Per Class). Subsequent rounds will use the synthetic samples obtained from the previous round of optimization for k-means initialization.
[0018] Step 2: The representative real samples and synthetic samples are parameterized by data interpolation;
[0019] Step 3: Use a differential twin augmentation strategy to augment the parameterized real samples and synthetic samples;
[0020] Step 4: Input the augmented real samples and synthetic samples into the model, calculate the feature distribution between the real samples and synthetic samples at different levels, and use the difference between the two as the feature distribution matching loss function;
[0021] Step 5: Use the prototype features of the synthetic samples to classify the real samples, and calculate the correlation between them as the sample discrimination loss;
[0022] Step 6: Weight the losses obtained in Steps 4 and 5 to obtain the final optimized loss function. Iterate and optimize the network parameters and synthetic samples until the loss function converges, ultimately obtaining synthetic samples with strong interpretability. The specific initialization process in Step 1 is as follows:
[0023] Step 1.1 Based on the set mini-batch size K, use it as the K value for the k-means clustering algorithm. Divide each class of the real samples into sub-clusters to obtain representative samples in each class. Use the sample centers of the divided K sub-clusters as initial samples.
[0024] The k-means clustering algorithm described above is based on the Euclidean distance metric, and the formula is as follows:
[0025]
[0026] Here This represents the sub-cluster c. i The center, x i Let J(c,μ) represent each sample, M represent the total number of samples, and J(c,μ) represent the sum of the minimum Euclidean distances from all samples in subcluster c to the subcluster center μ.
[0027] Step 1.2 Based on the set IPC (Image Per Class) value, the synthetic samples are also initialized using the k-means clustering algorithm. The real samples of each class are divided into IPC sub-clusters, and the centers of these sub-clusters are used as the initialization samples.
[0028] The specific process of parameterization in step 2 is as follows:
[0029] Step 2.1: For the representative real samples and synthetic samples obtained in Step 1 after clustering, a bilinear upsampling method is used to partition the image. Taking into account the correlation between adjacent pixels, two linear interpolations are performed in the X-axis and Y-axis directions. The formula for linear interpolation is:
[0030]
[0031] Here, (X0,Y0) and (X1,Y1) are known points;
[0032] Step 2.2 Performs a size transformation operation on the representative real samples and synthetic samples obtained in Step 2.1 after partitioning, restoring them to the original sample size, so that the synthetic samples are optimized from multiple local minima.
[0033] The specific process of the differential twin enhancement strategy in step 3 is as follows:
[0034] For the parameterized real and synthetic samples from step 2, the same common data augmentation strategies are applied, including cropping, rotation, and color dithering. Furthermore, cutout and cutmix data augmentation strategies are added. Here, the entire differential twin augmentation module must be differentiable to ensure backpropagation updates to the synthetic samples, expressed by the formula:
[0035]
[0036] Here, S represents the synthetic sample, A(·) represents the sampled augmentation strategy, D(·) represents the matching target between the real sample and the synthetic sample, and L θ (·) represents the loss function for training the network;
[0037] Finally, we obtain the processed real sample T. k and synthetic sample S k , where k represents the category.
[0038] The specific process of calculating the feature distribution matching loss in step 4 is as follows:
[0039] Step 4.1 For the true samples T of the same class k obtained in Step 3 k and synthetic sample S k The input is fed into the L-layer network;
[0040] Step 4.2 Based on Step 4.1, obtain the real sample T respectively. k and synthetic sample S k The feature vectors of the intermediate layers other than the last output layer are represented by the following formula:
[0041]
[0042] Here, and Let these represent the feature vectors of the real sample and the synthetic sample of the k-th class, respectively. This represents the feature vector of a real sample of class k in the i-th layer of the network. φ represents the feature vector of the synthetic sample of class k in layer i of the network. θ () indicates the training network;
[0043] Step 4.3 Since batch sizes are different and direct comparison is not possible, the prototype features of the real and synthetic samples are calculated based on the feature vectors obtained in 4.2. The average feature vector of each class is calculated as the prototype feature. C′=C×H×W, where C′ represents the size of the corresponding feature map;
[0044] Step 4.4 uses the mean squared error to calculate the feature distribution matching loss of the prototype features of the real samples and synthetic samples obtained in Step 4.3 in the intermediate layer, as shown in the following formula:
[0045]
[0046] Here, K represents the number of categories in the dataset, and L represents the total number of layers in the training network. This represents the prototype features of a real sample of class k in layer l of the network. This represents the prototype feature of the synthetic sample of class k in the l-th layer of the network.
[0047] The specific process of calculating and judging the loss in step 5 is as follows:
[0048] Step 5.1 Use the prototype features of the synthetic samples of this class from the last layer of the trained network as a classifier to classify all real samples in this class. The correlation between samples is measured by the result of the inner product, expressed by the formula:
[0049]
[0050] Here N′ = N×K, where I contains the logits values of N′×K real samples, N represents the number of real samples, and K represents the number of classes in the dataset. This represents the feature vector value of the real sample in this class at the last layer of the network, also known as the logits value. This represents the prototype characteristics of the synthetic samples of this class;
[0051] Step 5.2 Based on the inner product value obtained in Step 5.1, calculate the softmax value of each real sample in the class to obtain the predicted classification probability value, which is expressed by the formula:
[0052] p i =softmax(I i )
[0053] Step 5.3 Based on the classification probability values obtained in Step 5.2, the cross-entropy loss function is used to obtain the discriminant loss, which is expressed by the formula:
[0054]
[0055] The specific process of iterative optimization of network parameters and synthetic samples in step 6 is as follows:
[0056] Step 6.1 The feature distribution matching loss and discriminative loss obtained in Steps 4.4 and 5.3 are weighted to obtain the final loss function, expressed by the formula:
[0057] L = L m +αL c
[0058] Here, α is a weighting factor;
[0059] Step 6.2 Using the output logits values of the synthetic samples for each class obtained in Step 5.1 at the last network layer, the network parameters θ are iteratively updated using the cross-entropy loss function, expressed by the formula:
[0060]
[0061] Here, M represents the number of synthetic samples, m i This represents the probability value of the synthesized sample compared to the true label;
[0062] Step 6.3 uses the loss function obtained in Step 6.1 to iteratively update the synthetic samples, ultimately obtaining synthetic samples with strong interpretability, expressed by the formula:
[0063]
[0064] A dataset distillation system based on representative sample feature distribution matching includes:
[0065] Initialize clustering module: Use the k-means clustering algorithm to generate sub-clusters, and use the sub-cluster centers as the initialization of real and synthetic samples;
[0066] Data augmentation module: It uses bilinear interpolation to upsample the image, thereby increasing the diversity of image information, and uses a differential twin augmentation strategy to enhance the image.
[0067] Feature distribution matching module: The mean square error of the feature vectors of real samples and synthetic samples in the intermediate layer is used as the feature distribution matching loss;
[0068] Discriminant Loss Module: This module uses the prototype features of the synthetic samples in the last network layer as a classifier to classify real samples, obtaining a discriminant loss and ultimately producing highly interpretable synthetic samples. First, real and synthetic samples are initialized using an initialization clustering module; then, a data augmentation module generates diverse samples to reduce overfitting; during network training, a feature distribution matching module and a discriminant loss module are used to make the feature distribution of the synthetic samples approximate the feature distribution of the real samples, ultimately resulting in highly interpretable synthetic samples. A dataset distillation device based on representative sample feature distribution matching includes:
[0069] Memory: Used to store the computer program that implements the dataset distillation method based on representative sample feature distribution matching;
[0070] A processor is used to implement the dataset distillation method based on representative sample feature distribution matching when executing the computer program.
[0071] A computer-readable storage medium comprising:
[0072] The computer-readable storage medium stores a computer program that, when executed by a processor, enables a dataset distillation method based on matching the feature distribution of representative samples.
[0073] The beneficial effects of this invention are:
[0074] The k-means initialization strategy used in this invention for real and synthetic samples ensures that the selected samples can cover the feature distribution of the entire class as evenly as possible, thereby achieving stable training and rapid optimization of synthetic samples.
[0075] The bilinear interpolation parameterization strategy used in this invention takes into account the correlation between adjacent pixels in the image, which allows the synthesized samples to be optimized from multiple local minima.
[0076] The differential twin augmentation strategy employed in this invention cannot be directly and randomly augmented due to the difference in the number of samples between the real and synthetic samples. Considering the problem of optimizing the synthetic samples through backpropagation, the data augmentation operation must be differentiable, thus achieving effective sample augmentation.
[0077] The feature distribution matching strategy adopted in this invention optimizes the synthetic sample from a hierarchical perspective. This allows the feature distribution of the real sample to be captured, and features to be aligned across different scales, thereby making the feature distribution of the synthetic sample closer to that of the real sample.
[0078] The discriminative loss strategy employed in this invention improves the discriminative ability of the synthesized samples, making their prototype features more practically significant. Attached Figure Description
[0079] Figure 1 This is a flowchart of the method of the present invention.
[0080] Figure 2 This is a diagram of the feature distribution matching loss and discrimination loss modules of the present invention.
[0081] Figure 3 This is a comparison chart showing the classification accuracy of the present invention using different initialization strategies on the CIFAR10 dataset.
[0082] Figure 4 This is the final partially compressed image obtained by the present invention on the CIFAR100 dataset when IPC=10. Detailed Implementation
[0083] The present invention will now be described in further detail with reference to the accompanying drawings.
[0084] See Figure 1 A dataset distillation method based on representative sample feature distribution matching includes the following steps:
[0085] Step 1: Use the k-means algorithm to select representative real and synthetic samples for initialization. This ensures that the selected samples cover the feature distribution of the entire class as evenly as possible, achieving stable training and rapid optimization on the synthetic samples.
[0086] Real samples refer to images in the original dataset, while synthetic samples refer to representative images generated from the original dataset after compression algorithms. In the first round, representative synthetic samples are the cluster centers obtained from the real dataset using the k-means algorithm based on the set IPC (Image Per Class). Subsequent rounds will use the synthetic samples obtained from the previous round of optimization for k-means initialization.
[0087] The datasets used are MNIST, FashionMNIST, CIFAR10, CIFAR100, and TinyImageNet. The K value for each class of the real samples was set to 128, and the number of saved synthetic samples in each class was set to IPC = 1 / 10 / 50.
[0088] Step 1.1 Based on the set mini-batch size K, use it as the K value for the k-means clustering algorithm to divide the real samples into sub-clusters within each class. Obtain representative samples from each class, and use the sample centers of the K sub-clusters as initial samples.
[0089] The k-means clustering algorithm described above is based on the Euclidean distance metric, and the formula is as follows:
[0090]
[0091] Here This represents the center of the sub-cluster, x. i Let J(c,μ) represent each sample, M represent the total number of samples, and J(c,μ) represent the minimum Euclidean distances from all samples in subcluster c to the subcluster center μ.
[0092] Step 1.2: Based on the set IPC (Image Per Class) value, the synthetic samples are also initialized using the k-means clustering algorithm. Each class of real samples is divided into IPC sub-clusters, and the centers of these sub-clusters are used as the initialization samples.
[0093] Step 2: The representative real and synthetic samples obtained in Step 1 are parameterized through data interpolation. By considering the correlation between adjacent pixels in the image, the synthetic samples can be optimized from multiple local minima.
[0094] Step 2.1: For the representative real samples and synthetic samples obtained in Step 1, a bilinear upsampling method is used to partition the data, performing two linear interpolations in the X and Y axes. The formula for linear interpolation is:
[0095]
[0096] Here, (X0,Y0) and (X1,Y1) are known points.
[0097] Step 2.2 Perform a size transformation operation on the representative real samples and synthetic samples obtained in Step 2.1 after partitioning to restore them to the original sample size.
[0098] Step 3: Use the differential twin augmentation strategy to augment the parameterized real samples and synthetic samples obtained in Step 2.
[0099] This makes both real and synthetic samples more interpretable and increases the diversity of the data.
[0100] The processed real and synthetic samples obtained in step 2 are subjected to the same common data augmentation strategies, including cropping, rotation, and color dithering. Cutout and Cutmix data augmentation strategies are then added. Here, considering the problem of backpropagation optimization of the synthetic samples, the entire differential twin augmentation module must be differentiable to ensure that backpropagation can update the synthetic samples. This can be expressed by the formula:
[0101]
[0102] Here, S represents the synthetic sample, A(·) represents the sampled augmentation strategy, D(·) represents the matching target between the real sample and the synthetic sample, and L θ (·) represents the loss function for training the network.
[0103] See Figure 2 Step 4: Input the real samples and synthetic samples obtained in Step 3 into the model.
[0104] The feature distributions between the real and synthetic samples at different hierarchical levels are calculated, and the difference between them is used as the feature distribution matching loss function. This allows for optimization of the synthetic samples from a hierarchical perspective, capturing the feature distributions of the real samples, aligning features across different scales, and further making the feature distributions of the synthetic samples approximate those of the real samples.
[0105] The training model employed a three-layer ConvNet architecture. The network was then trained from scratch using these synthetic datasets and evaluated on real test datasets.
[0106] The trained network architecture consists of three convolutional blocks, each consisting of a 3×3 convolutional layer with 128 kernels, followed by instance regularization, ReLU activation layers, and 2×2 average pooling layers with a stride of 2. Due to the large input size of the dataset, a fourth convolutional block was added during training on the TinyImageNet dataset. The initial learning rate was 0.01, the weight decay was 5e-4, and the momentum was 0.9. The outer ring was trained for 2000 iterations to optimize the synthesized images, and the inner ring was trained for 300 iterations to optimize the network parameters.
[0107] Step 4.1 For the true samples T of the same class k obtained in Step 3 k and synthetic sample S k The input is fed into the L-layer network.
[0108] Step 4.2 Based on Step 4.1, obtain the real sample T respectively. k and synthetic sample S k The feature vectors of the intermediate layers other than the last output layer are represented by the following formula:
[0109]
[0110]
[0111] Here, and Let these represent the feature vectors of the real sample and the synthetic sample of the k-th class, respectively. This represents the feature vector of a real sample of class k in the i-th layer of the network. φ represents the feature vector of the synthetic sample of class k in the i-th layer of the network. θ () indicates the training network.
[0112] Step 4.3: Due to the different batch sizes, comparison is not possible. Therefore, based on the feature vectors obtained in 4.2, the prototype features of the real and synthetic samples are calculated, and the average feature vector of each class is calculated as the prototype feature. Here... C′=C×H×W, where C′ represents the size of the corresponding feature map.
[0113] Step 4.4 uses the mean squared error to calculate the feature distribution matching loss of the prototype features of the real samples and synthetic samples obtained in Step 4.3 in the intermediate layer, as shown in the following formula:
[0114]
[0115] Here, K represents the number of categories in the dataset, and L represents the total number of layers in the training network. This represents the prototype features of a real sample of class k in the l-th layer of the network. This represents the prototype feature of the synthetic sample of class k in the l-th layer of the network.
[0116] See Figure 2 Step 5: Use the prototype features of the synthetic samples to classify the real samples and calculate the correlation between them as the sample discrimination loss.
[0117] Step 5.1 Use the prototype features of the synthetic samples of this class from the last layer of the trained network as a classifier to classify all real samples in this class. The correlation between samples is measured by the result of the inner product, expressed by the formula:
[0118]
[0119] Here N′ = N×K, where I contains the logits values of N′×K real samples, N represents the number of real samples, and K represents the number of classes in the dataset. This represents the feature vector value of the real sample in this class at the last layer of the network, also known as the logits value. This represents the prototype characteristics of the synthetic samples of this class;
[0120] Step 5.2 Based on the inner product value obtained in Step 5.1, calculate the softmax value of each real sample in the class to obtain the predicted classification probability value, which is expressed by the formula:
[0121] p i =softmax(I i )
[0122] Step 5.3 Based on the classification probability values obtained in Step 5.2, the cross-entropy loss function is used to obtain the discriminant loss, which is expressed by the formula:
[0123]
[0124] Step 6: Weight the losses obtained in Steps 4 and 5 to obtain the final optimized loss function. Iterate and optimize the loss function continuously for the network parameters and synthetic samples until the loss function converges. Finally, a synthetic sample with strong interpretability is obtained. Here, the weighting factor α = 1.
[0125] Step 6.1 The feature distribution matching loss and discriminative loss obtained in Steps 4.4 and 5.3 are weighted to obtain the final loss function, expressed by the formula:
[0126] L = L m +αL c
[0127] Here, α is a weighting factor.
[0128] Step 6.2 Using the output logits values of the synthetic samples for each class obtained in Step 5.1 at the last network layer, the network parameters θ are iteratively updated using the cross-entropy loss function, expressed by the formula:
[0129]
[0130] Here, M represents the number of synthetic samples, m i This represents the probability value after comparing the synthetic sample with the real label.
[0131] Step 6.3 uses the loss function obtained in Step 6.1 to iteratively update the synthetic samples, ultimately obtaining synthetic samples with strong interpretability, expressed by the formula:
[0132]
[0133] To evaluate the performance of the synthesized images, the method of this application (Ours) was compared with three core set selection methods: Random, Herding, and K-Center. It was also compared with five state-of-the-art dataset distillation methods: DC, DM, IDM, MTT, and DiM. Experimental results are shown in Table 1.
[0134] As shown in Table 1, although its performance on MNIST and FashionMNIST is slightly lower than DiM, it shows improvement on the CIFAR10 dataset. Accuracy is improved by 1.3% at IPC=1. Furthermore, while DiM's performance drops significantly on larger datasets, the method presented in this application (Ours) can be applied to larger datasets and still maintains performance on the TinyImageNet dataset, demonstrating broad applicability.
[0135] Table 1 shows the classification accuracy across multiple datasets.
[0136]
[0137] To test generalization on synthetic datasets, the performance results of training ConvNet-3 on the CIFAR10 dataset are presented in Table 2. Synthetic data was learned using one architecture (C), and then evaluated on another architecture (T) by training a model from scratch and testing it on real test data. We evaluated several neural architectures, including ConvNet, AlexNet, VGG-11, ResNet10, ResNet18, and DenseNet-121. Batch normalization was applied to all network architectures.
[0138] As can be seen, the method (Ours) in this application has good generalization ability, and can improve the accuracy by about 10% on multiple test models. In particular, when the test model is ResNet10, the accuracy can be improved by about 20% compared to MTT.
[0139] Table 2 Cross-model generalization assessment
[0140]
[0141] See Figure 3 This image compares the classification accuracy using different initialization strategies on the CIFAR10 dataset. The effectiveness of representative sample selection (Random, Central, and Boundary) is verified by comparing several different sampling methods. Using samples from subclusters as initialization for the synthesized image demonstrates that the performance is significantly better than random initialization. A total of 800 iterations were conducted on the CIFAR10 dataset, with IPC set to 10. When the number of training iterations is around 200, the proposed method (Ours) achieves better performance than random sampling, significantly improving training efficiency. Performance further improves with increasing training iterations.
[0142] See Figure 4This is the final partially compressed image obtained by the present invention on the CIFAR100 dataset when IPC=10. The method (Ours) of this application introduces more classification features and diversity into the synthesized image.
[0143] A dataset distillation device based on representative sample feature distribution matching, comprising:
[0144] Memory: Used to store the computer program that implements the dataset distillation method based on representative sample feature distribution matching;
[0145] A processor is used to implement the dataset distillation method based on representative sample feature distribution matching when executing the computer program.
[0146] A computer-readable storage medium comprising:
[0147] The computer-readable storage medium stores a computer program that, when executed by a processor, enables a dataset distillation method based on matching the feature distribution of representative samples.
Claims
1. A dataset distillation method based on representative sample feature distribution matching, characterized in that, Includes the following steps: Step 1: Use the k-means algorithm to select representative real and synthetic samples for initialization; real samples refer to images in the original dataset, while synthetic samples refer to representative images generated from the original dataset after compression. Step 2: The representative real samples and synthetic samples are parameterized by data interpolation; Step 3: Use a differential twin augmentation strategy to augment the parameterized real samples and synthetic samples; Step 4: Input the augmented real samples and synthetic samples into the model, calculate the feature distribution between the real samples and synthetic samples at different levels, and use the difference between the two as the feature distribution matching loss function; Step 5: Use the prototype features of the synthetic samples to classify the real samples, and calculate the correlation between them as the sample discrimination loss; Step 6: Weight the losses obtained in Step 4 and Step 5 to obtain the final optimized loss function. Iterate and optimize the network parameters and synthetic samples until the loss function converges, and finally obtain synthetic samples with strong interpretability. The specific process of calculating the feature distribution matching loss in step 4 is as follows: Step 4.1 For the same type obtained in Step 3 k real samples and synthetic samples The input is fed into the L-layer network; Step 4.2 Based on Step 4.1, obtain the real samples respectively. and synthetic samples The feature vectors of the intermediate layers other than the last output layer are represented by the following formula: and Let these represent the feature vectors of the real sample and the synthetic sample of the k-th class, respectively. This indicates that the real sample of class k is in the network. i The feature vector of the layer, This indicates that the synthetic sample of class k is in the network. i The feature vector of the layer, Indicates the training network; Step 4.3: Calculate the prototype features of the real and synthetic samples based on the feature vectors obtained in 4.2, and calculate the average feature vector of each class as the prototype feature. , This indicates the size of the corresponding feature map; Step 4.4 Calculate the feature distribution matching loss of the prototype features of the real and synthetic samples obtained in Step 4.3 in the intermediate layer using the mean squared error, as shown in the following formula: K represents the number of categories in the dataset, and L represents the total number of layers in the training network. This indicates that the real sample of class k is in the network. l The prototype characteristics of the layer This indicates that the synthetic sample of class k is in the network. l Prototype features of the layer.
2. The dataset distillation method based on representative sample feature distribution matching according to claim 1, characterized in that, The specific process of initialization in step 1 is as follows: Step 1.1 Based on the set mini-batch size K, use it as the K value for the k-means clustering algorithm. Divide the real samples into sub-clusters in each class to obtain representative samples in each class. Use the sample centers of the divided K sub-clusters as the initial samples. The k-means clustering algorithm described above is based on the Euclidean distance metric, and the formula is as follows: This represents a sub-cluster. The center Represents each sample, Indicates the total number of samples. Subclusters All samples reach the center of the sub-cluster The minimum Euclidean distance; Step 1.2 Based on the set IPC (Image Per Class) value, the synthetic samples are also initialized using the k-means clustering algorithm. The real samples of each class are divided into IPC sub-clusters, and the centers of these sub-clusters are used as the initialization samples.
3. The dataset distillation method based on representative sample feature distribution matching according to claim 2, characterized in that, The specific process of parameterization in step 2 is as follows: Step 2.1 The representative real samples and synthetic samples obtained in Step 1 are partitioned using bilinear upsampling. Two linear interpolations are performed in the X and Y axes. The formula for the linear interpolation is: The point is known; Step 2.2 Perform a size transformation operation on the representative real samples and synthetic samples obtained in Step 2.1 after partitioning to restore them to the original sample size, so that the synthetic samples are optimized from multiple local minima.
4. The dataset distillation method based on representative sample feature distribution matching according to claim 3, characterized in that, The specific process of the differential twin enhancement strategy in step 3 is as follows: For the real and synthetic samples after parameterization in step 2, the same common data augmentation strategies are used, including cropping, rotation, color dithering, and the addition of cutout and cutmix data augmentation strategies, as expressed by the formula: S Indicates a synthetic sample. This represents the sampled enhancement strategy. This represents the matching target between real and synthetic samples. This represents the loss function used to train the network; Finally, we obtain the processed real sample. and synthetic samples , k Indicates the category.
5. A dataset distillation method based on representative sample feature distribution matching according to claim 4, characterized in that, The specific process of calculating and determining the loss in step 5 is as follows: Step 5.1 Use the prototype features of the synthetic samples of this class from the last layer of the trained network as a classifier to classify all real samples in this class. The correlation between samples is measured by the result of the inner product, expressed by the formula: , Includes The logits value of a real sample, Indicates the number of real samples. This represents the number of categories in the dataset. This represents the feature vector value of the real sample in this class at the last layer of the network, also known as the logits value. This represents the prototype characteristics of the synthetic samples of this class; Step 5.2 Based on the inner product value obtained in Step 5.1, calculate the inner product of each true sample in this class. The predicted classification probability value is obtained from the value, expressed by the formula: Step 5.3 Based on the classification probability values obtained in Step 5.2, the cross-entropy loss function is used to obtain the discriminant loss, which is expressed by the formula: 。 6. The dataset distillation method based on representative sample feature distribution matching according to claim 5, characterized in that, The specific process of iterative optimization of network parameters and synthetic samples in step 6 is as follows: Step 6.1 Based on the feature distribution matching loss and discrimination loss obtained in Steps 4.4 and 5.3, a weighted average is applied to obtain the final loss function, expressed by the formula: It is a weighting factor; Step 6.2 Using the output logits values of the synthetic samples for each class obtained in Step 5.1 at the last network layer, apply the cross-entropy loss function to the network parameters. Iterative updates can be expressed using a formula: Indicates the number of synthetic samples. This represents the probability value of the synthesized sample compared to the true label; Step 6.3 Use the loss function obtained in Step 6.1 to iteratively update the synthetic samples, finally obtaining synthetic samples with strong interpretability, expressed by the formula: 。 7. A dataset distillation system based on representative sample feature distribution matching for implementing the method of any one of claims 1-6, characterized in that, include: Initialize clustering module: Use the k-means clustering algorithm to generate sub-clusters, and use the sub-cluster centers as the initialization of real and synthetic samples; Data augmentation module: It uses bilinear interpolation to upsample the image, thereby increasing the diversity of image information, and uses a differential twin augmentation strategy to enhance the image. Feature distribution matching module: The mean square error of the feature vectors of real samples and synthetic samples in the intermediate layer is used as the feature distribution matching loss; Discriminant loss module: The prototype features of the synthetic sample in the last network layer are used as a classifier to classify the real samples, obtain the discriminant loss, and finally obtain the synthetic sample with strong interpretability.
8. A dataset distillation device based on representative sample feature distribution matching, characterized in that, include: Memory: for storing a computer program that implements the dataset distillation method based on representative sample feature distribution matching as described in any one of claims 1-6; A processor, configured to execute the computer program to implement the dataset distillation method based on representative sample feature distribution matching as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of a dataset distillation method based on representative sample feature distribution matching, as claimed in any one of claims 1-6.