A data set distillation method and system based on semantic relationship decoupling

Abstract

This invention discloses a dataset distillation method and system based on semantic relation decoupling, belonging to the field of data distillation technology. This invention not only aligns the feature distribution of the distilled data with the category-level statistics of the original data stored in the memory library to achieve efficient, fine-grained category-level optimization of the distilled data, thereby improving the representativeness and semantic consistency of the distilled data, but also addresses the problem of overly concentrated feature distribution in the last layer of the pre-trained model. By simultaneously maximizing the diagonal elements and minimizing the off-diagonal elements of the covariance matrix, it effectively alleviates the gradient vanishing problem in downstream tasks caused by insufficient sample diversity in the feature space, thereby improving the model's generalization ability and learning effect on downstream tasks.

Description

Technical Field

[0001] This invention relates to the field of data distillation technology, and in particular to a data distillation method and system based on semantic relationship decoupling. Background Technology

[0002] The exponential growth of massive datasets has driven significant advances in deep learning, enabling deep neural networks to achieve tremendous success across a wide range of tasks. However, the increased storage requirements and training times resulting from the scaling of datasets present a fundamental challenge. Data distillation has emerged as a key solution, compressing massive datasets into smaller, more representative datasets (i.e., distilled datasets). Such distilled datasets can accelerate model training in downstream tasks while maintaining the training performance of the original dataset across a wide range of model architectures.

[0003] Dataset distillation extracts key information from a large-scale original dataset to generate a small but highly condensed synthetic dataset, thereby significantly reducing training costs while maintaining model performance as much as possible. This synthetic dataset can be used for various purposes, such as efficiently training models with specific functions (e.g., optimization for a particular task), rapid prototyping, hyperparameter tuning, and privacy-preserving model training without revealing the original data. It can also be used in resource-constrained scenarios such as continuous learning and federated learning.

[0004] Traditional methods directly align the feature distributions of the original and distilled data without updating model parameters. These methods typically achieve feature space alignment by extracting features from intermediate network layers and matching them with feature prototypes or statistical properties. For example, some methods utilize class prototypes of intermediate features from deep networks for alignment; others introduce spatial attention mechanisms to align only key features to reduce redundancy; still others reconstruct the prototype matching problem within the regenerating kernel Hilbert space to improve the generalization ability of the matching. However, these methods often use randomly initialized models to extract features. Because the models are not sufficiently trained, the extracted feature embeddings are often inaccurate, thus affecting the quality of the distilled data. To improve the accuracy of feature embedding, subsequent methods introduce multiple pre-trained feature extractors to obtain more robust feature representations. Although using pre-trained models can improve embedding quality, relying solely on class prototype matching still cannot guarantee the diversity of distilled data. Therefore, some methods further align the covariance matrices of class features to preserve richer semantic information and data distribution characteristics. However, this approach of simultaneously processing the prototype and the covariance matrix significantly increases the processing complexity of both the original and distilled data, resulting in high memory consumption and hindering practical deployment and expansion.

[0005] While many dataset distillation methods demonstrate strong performance on medium-sized benchmarks, their scalability to large-scale datasets like ImageNet-1K remains inherently limited. A major bottleneck is the need to load the original data at each step of optimizing the extracted data, resulting in significant memory and computational overhead. Therefore, traditional methods face significant efficiency and performance bottlenecks, often only alleviating efficiency issues in large-scale data distillation through staged optimization. For example, some methods first train a well-converged model on the original data, then optimize the distilled data by aligning local batch statistics based on the model's global batch normalization (BN) parameters. This method improves processing efficiency by decoupling the training and distillation processes. However, such methods typically use the global BN parameters as a uniform optimization objective, leading to convergence in the optimization direction of all synthesized distilled samples, making it difficult to effectively maintain inter-class differences, thus limiting the discriminative power of the distilled data. To enhance inter-class differentiation, some approaches attempt to improve intra-class compactness and inter-class differentiation by aligning BN statistics within classes. However, relying solely on BN statistics (such as mean and variance) is insufficient to fully guarantee the diversity and richness of distillation data. Summary of the Invention

[0006] To enhance the diversity of distillation data, this invention proposes a dataset distillation method and system based on semantic relation decoupling, which effectively preserves the semantic richness of the original dataset by aligning feature distributions in the feature space.

[0007] To achieve the above objectives, the technical solution of the present invention includes the following:

[0008] A dataset distillation method based on semantic relation decoupling, characterized in that the method includes: pre-trained model In the original dataset Training is performed on the model, and the pre-trained model is extracted. Intermediate layer Features to estimate the original dataset Each category In the middle layer Feature distribution on ;in, , Represents the pre-trained model The number of floors, Indicates a Gaussian distribution. Represents the original dataset Medium category The sample is in the middle layer The mean vector on, Represents the original dataset Medium category The sample is in the middle layer The covariance matrix on; Get Category Distillation data And based on the feature distribution Initialize the pre-trained model ; Based on the initialized pre-trained model The data to be distilled Perform iterative distillation to obtain the distillation results.

[0009] Furthermore, by extracting pre-trained models Intermediate layer Features to estimate the original dataset Each category In the middle layer Feature distribution on ,include: Original dataset Medium category samples Input into the pre-trained model To obtain the intermediate layer Feature tensor ; For feature tensors Perform average pooling to obtain the pooled features. ; Based on pooling features Calculate the mean vector Covariance Matrix .

[0010] Furthermore, based on the initialized pre-trained model The data to be distilled Perform iterative distillation to obtain distillation results, including: Distillation data Feed into the pre-trained model after initialization And in calculating cross-entropy loss Simultaneously, acquire distillation data Each category In the middle layer Feature distribution on and layers covariance matrix on ;in, Representing distillation data Medium category Data in the middle layer The mean vector on, Indicates the data to be distilled Medium category Data in the middle layer The covariance matrix on, Indicates the iteration round, distillation data For the data to be distilled ; The mean vectors are respectively Covariance matrix With mean vector Covariance matrix After alignment, calculate the Gaussian alignment loss. ; By maximizing the covariance matrix diagonal variance The variance regularization term is obtained. ; By minimizing the covariance matrix The off-diagonal covariance yields the covariance term. ; Based on cross-entropy loss Gaussian alignment loss Variance regularization term and covariance term The overall loss was obtained. ; Obtain distillation data Subsequently, based on this distillation data Re-execute the distillation data Feed into the pre-trained model after initialization until the set requirements are met; among them, Indicates the learning rate. This represents the gradient.

[0011] Furthermore, the Gaussian distribution alignment loss ;in, This is a hyperparameter.

[0012] Furthermore, the variance regularization term ;in, diagonal variance The Middle One element, diagonal variance Dimensions.

[0013] Furthermore, the covariance term ;in, diagonal variance Dimensions , Let them represent the covariance matrix respectively. Bank number and column number.

[0014] A dataset distillation system based on semantic relation decoupling, the system comprising: The feature distribution acquisition module is used to obtain the pre-trained model In the original dataset Training is performed on the model, and the pre-trained model is extracted. Intermediate layer Features to estimate the original dataset Each category In the middle layer Feature distribution on ;in, , Represents the pre-trained model The number of floors, Indicates a Gaussian distribution. Represents the original dataset Medium category The sample is in the middle layer The mean vector on, Represents the original dataset Medium category The sample is in the middle layer The covariance matrix on; The data acquisition module is used to obtain categories. Distillation data ; The model initialization module is used to initialize the model based on the feature distribution. Initialize the pre-trained model ; The distillation result generation module is used to generate distillation results based on the initialized pre-trained model. The data to be distilled Perform iterative distillation to obtain the distillation results.

[0015] An electronic device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the dataset distillation method based on semantic relation decoupling as described above.

[0016] A computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the dataset distillation method based on semantic relation decoupling as described above.

[0017] A computer program product, characterized in that, when the computer program product is run on a computer device, the computer device executes the dataset distillation method based on semantic relation decoupling as described above.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects.

[0019] (1) Explicitly modeling the relationship between features to enhance data diversity: Unlike the traditional alignment method based on BN statistics, this method aligns the mean and covariance of the original and distilled data by matching Gaussian distribution in the feature space, effectively preserving the relationship between features and semantic diversity. It is the first distillation method that systematically maintains feature correlation.

[0020] (2) Introducing covariance regularization to decouple semantic relations and alleviate gradient vanishing problem: In response to the problem of concentrated feature distribution in the last layer of the pre-trained model, this method enhances feature diversity by constraining the covariance matrix (maximizing the diagonal and minimizing the off-diagonal), avoids gradient starvation in downstream tasks, and improves the model training effect.

[0021] (3) High efficiency and strong generalization: This method achieves excellent distillation performance without introducing additional computational overhead, supports cross-architecture generalization, is applicable to a variety of model structures, and is highly practical.

[0022] In summary, unlike alignment methods based on batch normalization (BN), this invention employs Gaussian distribution matching to align the distributions of the original and distilled data in the feature space, thereby more accurately characterizing the inherent distributional properties of the data. To obtain the Gaussian distribution of the original dataset, this invention designs an offline Gaussian distribution estimation method. This method only requires loading the original data once, extracting feature statistics using a pre-trained model, and storing the mean vector and covariance matrix corresponding to each category in a memory database. Subsequently, by aligning the feature distribution of the distilled data with the category-level statistics of the original data stored in the memory database, efficient and fine-grained category-level optimization of the distilled data is achieved, thereby improving the representativeness and semantic consistency of the distilled data. Furthermore, addressing the problem of overly concentrated distributions (distribution compactness) in the last layer feature distribution of the pre-trained model, this invention further introduces a covariance regularization mechanism. This mechanism effectively alleviates the gradient vanishing problem in downstream tasks caused by insufficient sample diversity in the feature space by simultaneously maximizing the diagonal elements of the covariance matrix (to enhance the variance between features and improve feature diversity) and minimizing the off-diagonal elements (to eliminate redundant correlations between features), thereby improving the model's generalization ability and learning effect on downstream tasks. Attached Figure Description

[0023] Figure 1 Framework diagram of a data distillation method based on semantic relation decoupling.

[0024] Figure 2 CIFAR-100 hyperparameter ablation at IPC=50 ( , , ).

[0025] Figure 3 A comparative visualization of distillation data between the present invention and existing technologies. Detailed Implementation

[0026] To make the technical solutions of the present invention more obvious and understandable, specific examples and accompanying drawings are provided to clearly and completely describe the technical solutions in the embodiments of the present invention.

[0027] The data distillation method based on semantic relation decoupling of the present invention, such as Figure 1 As shown. In the first stage, this invention extracts pre-trained models. Intermediate layer Features to estimate the original dataset Each category The Gaussian distribution, i.e. Then, these Gaussian distribution statistics are stored in a memory. In the second stage, the data is distilled. During the optimization process, this invention extracts the relevant statistics from the memory and inputs them into... In order to ensure consistency between layers, this invention incorporates distillation data. Feature distribution With from Align the corresponding statistics. To solve... To address the compactness issue of the final feature space, this invention introduces an explicit regularization term to constrain the covariance matrix of the distillation data. Specifically, this invention maximizes the diagonal elements (variance loss) of the covariance matrix. And minimize the off-diagonal elements of the covariance matrix (covariance loss). ).

[0028] (I) Framework of distillation method for large-scale datasets.

[0029] To achieve data distillation on large-scale datasets, existing methods involve three key stages: compression, restoration, and relabeling. In the compression stage, the pre-trained model... First, in the original dataset Training is performed on the original dataset to achieve this. Feature compression into pre-trained model During the recovery phase, the pre-trained model Used to optimize distillation data The optimization objective is to optimize the distillation data. With pre-trained models Alignment of encoded information. Specifically, the goal is to minimize the following loss function: (1) in, It is the standard cross-entropy loss function. This is the batch normalization (BN) alignment loss. (BN loss) Ensure distillation data The mean and variance of the standardized feature distribution and the pre-trained model The values ​​stored in the batch normalization layer are aligned. Finally, in the relabeling stage, the distilled data is aligned using soft labels. Upstream and downstream training tasks are performed, where soft labels are derived from pre-trained models. The logits generated.

[0030] (ii) Semantic Gaussian distribution matching.

[0031] While previous work has achieved good results by aligning the mean and variance of the batch normalization (BN) layer in the pre-trained model using Equation (1), these statistics are still insufficient to fully capture the distribution of the original data. Since covariance reflects the relationship between feature dimensions, it is also an important statistical component. Therefore, this invention proposes an offline Gaussian distribution estimation method to better describe the diversity of the original data. Specifically, for each category... The present invention places it in the layer The sample distribution is modeled as a Gaussian distribution. For each category... The present invention uses samples from the dataset to input them into a pre-trained model. Extract its features and generate a feature tensor at each intermediate layer. Considering the high dimensionality of the feature tensor (e.g., the first layer dimension of ResNet-18), Directly calculating the covariance matrix is ​​computationally very expensive. To reduce computational costs and retain key information, this invention employs average pooling: (2) in , Indicates the first Number of channels in the layer and They represent the first The height and width of the layer feature map. After average pooling, the feature dimension is reduced to... .

[0032] Using the pooled features, this invention calculates each category The characteristic statistics, specifically the mean vector. Covariance Matrix They are respectively: (3) (4) in and , Represents the original dataset The middle category is The data.

[0033] To efficiently manage the Gaussian distribution for each category, this invention constructs a memory to store the mean and covariance of each Gaussian distribution. This memory provides fast access during optimization, thereby improving computational efficiency. During the optimization of the distillation data, this invention sequentially optimizes each category... Distillation data Optimize. In optimization... Previously, this invention extracted categories from the memory bank. Gaussian distribution statistics (mean and covariance) are used and assigned to the pre-trained model. Subsequently, using the same method as modeling the original data, this invention... Modeling a Gaussian distribution yields... And these statistics are compared with the pre-trained model. The statistical measures of the original data are aligned. Based on this, the present invention provides... The Gaussian Alignment (GDA) loss is defined for each layer (excluding the last layer) as follows: (5) in It is a hyperparameter used to balance the contributions of the mean and covariance alignment terms.

[0034] (iii) Semantic relation decoupling, diversity enrichment and enhancement strategies.

[0035] Although the layer-by-layer Gaussian distribution alignment in the pre-trained model effectively preserves the diversity of the original data, the last layer... The overly compact and homogeneous features in the model limit the matching space. Therefore, aligning with the features of the last layer of the pre-trained model may lead to insufficient diversity in the distilled data, resulting in gradient starvation in downstream tasks. Taking the typical category of "birds" as an example, in the features learned by the model in the last layer, the "wing shape" of the bird is often highly bound to the "sky (background)"—for example, when the wings are open, the sky background often appears, and the two form a strong correlation in the feature space. This overly strong feature coupling makes it difficult for the distilled data to effectively learn the diverse morphologies of the wings themselves. By introducing covariance regularization to constrain and decouple this kind of redundant cross-dimensional correlation, the strong binding relationship between the background and local features can be effectively weakened, allowing the distilled data to focus more on learning the intrinsic changes of the object itself, such as the different postures and shapes of the wings, thereby enhancing the discriminativeness and diversity of the features.

[0036] This method proposes an explicit covariance regularization for the last layer of features in distillation data. The covariance matrix of the last layer of distillation data is denoted as... This invention introduces two complementary properties to enhance feature representation: (1) maximizing diagonal variance To ensure full utilization of each feature dimension, and (2) minimize off-diagonal covariance to promote feature independence and reduce redundancy. Therefore, the regularizer proposed in this invention consists of two components: (6) (7) Variance regularization term A hinge-based approach is used to enhance feature diversity, while the covariance term... This serves as a regularizer to eliminate cross-correlation, aiming to improve representation efficiency.

[0037] (iv) Overall loss.

[0038] Regarding the overall loss, this invention follows previous work and retains the cross-entropy loss. (See Equation (1)) to maintain intra-class separability. Furthermore, the composite loss function of this invention comprises three key components: Gaussian distribution alignment loss. Variance regularization loss Covariance Regularization Loss The overall loss function takes the following form: (8) in, and These are the weighting coefficients.

[0039] Computational complexity analysis: Traditional methods align feature maps by matching the mean and variance of all BN layers, resulting in a computational complexity of O(n log n). In contrast, the method of this invention uses spatial average pooling to reduce spatial dimensionality and simultaneously aligns the mean and covariance matrices, thereby reducing computational complexity to [value missing]. .

[0040] (v) Algorithm.

[0041] The overall algorithm of this invention mainly includes steps 5.1 to 5.8 as described below, wherein the input data mainly includes the original training dataset. Distilled synthetic dataset Number of training iterations T, pre-trained model ,category Learning rate wait.

[0042] Step 5.1: Initialize the synthesized dataset after distillation ; Step 5.2: By Category Group the samples to form subsets ; Step 5.3: Calculate the characteristic mean See formula (3); Step 5.4: Calculate the covariance See formula (4); Step 5.5: For each category arrive Execution: Use memory Initialize model parameters using statistics: Step 5.6: For each iteration round The Gaussian alignment loss is obtained based on the above formulas (5), (6), and (7), respectively. Variance regularization term and covariance term And combined with cross-entropy loss To generate total loss ; Step 5.7: Update Then, repeat step 5.6; Step 5.8: Output the synthesized dataset after distillation .

[0043] In summary, this invention is the first to explicitly preserve the relationships between features in dataset distillation: it proposes an enhanced distribution alignment method that effectively preserves the correlation between features in large-scale dataset distillation by matching Gaussian distributions in the feature space (aligning the mean and covariance matrix), thereby significantly improving the semantic diversity and representativeness of the distilled data.

[0044] This invention introduces covariance regularization to alleviate the downstream gradient vanishing problem: addressing the issue of concentrated feature distribution and highly similar intra-class samples in the last layer of the pre-trained model, this invention constrains the covariance matrix of the distilled data (increasing diagonal elements and decreasing off-diagonal elements) through a covariance regularization mechanism. This enhances feature diversity while maintaining intra-class compactness, effectively avoiding gradient starvation in downstream tasks and improving model training performance.

[0045] This invention achieves state-of-the-art performance on multiple benchmark datasets without introducing additional computational overhead; due to the rich semantic diversity of distilled data, the model trained on it also exhibits excellent cross-architecture generalization ability and is applicable to different model structures.

[0046] Below, the dataset distillation method of the present invention is evaluated on the following widely used image benchmark datasets: (1) Low-resolution dataset: CIFAR-10 / 100 (32 32 resolution, 60,000 images, 10 / 100 categories) and Tiny-ImageNet (64 (1) 64 resolution, 100,000 images, 200 categories); (2) High-resolution dataset: ImageNet-1K (224 (224 resolution, 1.28 million images, 1,000 categories). These datasets vary in complexity, with ImageNet-1K being the most complex due to its high resolution and large number of images.

[0047] Implementation details: This invention uses ResNet-18 as the default distillation architecture. The covariance matrix hyperparameter is set to [value missing] by default. To promote feature diversity while aligning with a Gaussian distribution, the regularization strategy uses weighted coefficients. Perform variance regularization. Perform covariance regularization.

[0048] Evaluation Metric: The quality of the distilled data is evaluated using Top-1 test accuracy on the original dataset's test set. According to the evaluation strategy, soft labels generated on the distilled data by the pre-trained teacher model are used as the true labels for training new models. These soft labels are dynamically updated by the teacher model in each validation cycle.

[0049] Comparison Methods: The method of this invention is compared with several state-of-the-art benchmark methods. In addition to random sample selection, it is also compared with classic dataset distillation methods, including DC, KIP, DM, CAFE, MTT, and DataDAM. For large-scale dataset distillation, SRe2L and LPLD are selected for comparison.

[0050] Results on CIFAR-10 / 100 and Tiny-ImageNet: Tables 1 and 2 show the comparative analysis of the proposed framework with state-of-the-art methods on the CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. For classic dataset distillation methods, ConvNet represents the method using ConvNet-128, and RN-18 represents the method using ResNet-18. The results reveal two key insights: (1) When using larger-scale pre-trained models (such as ResNet-18), the proposed method achieves significant performance improvements, surpassing SRe2L by 12.7% on CIFAR-10, 12.9% on CIFAR-100, and 14.1% on Tiny-ImageNet (IPC = 50). These significant improvements validate the effectiveness of the Gaussian distribution matching mechanism in capturing important features. (2) Although methods based on the traditional ConvNet-128, such as MTT, are considered state-of-the-art methods for distilling small datasets through two-layer optimization, they perform poorly when scaling to large datasets. In contrast, the method of this invention consistently outperforms these methods in more challenging benchmarks, demonstrating its superior ability to handle dataset scaling, especially on CIFAR-100 and Tiny-ImageNet. Table 1. Comparison of different methods on CIFAR-10 and CIFAR-100. Table 2 Comparison of different methods on Tiny-ImagNet Results for ImageNet-1K: Tables 3 to 5 present a comprehensive evaluation of the proposed method on the ImageNet-1K dataset, using ResNet-18, ResNet-50, and ResNet-101 architectures. The results reveal the following insights: (1) Both LPLD and the proposed method outperform SRe2L, highlighting the effectiveness of class-specific alignment in reducing inter-class similarity and enhancing inter-class separability. (2) The proposed method further surpasses LPLD, demonstrating that the introduction of covariance alignment and regularization significantly improves the diversity of distilled data, thereby achieving superior performance in downstream tasks. Table 3 compares ImageNet-1K using ResNet-18. Table 4 compares ImageNet-1K using ResNet-50. Table 5 compares ImageNet-1K using ResNet-101. Cross-Architecture Evaluation: A more critical evaluation metric for distillation data is its generalization ability across different network architectures. The cross-architecture generalization ability was comprehensively evaluated by testing the performance of the CIFAR-100 distillation dataset (trained on ResNet-18) on a variety of unseen network architectures trained from scratch. The evaluation included seven different architectures: ResNet-50, MobileNetV2, EfficientNetB0, ShuffleNetV2, and VGG-16, as shown in Table 6. The results demonstrate that the method of this invention can achieve generalization without requiring optimization of distillation data across multiple architectures. Table 6 Cross-architecture performance of the CIFAR-100 dataset using ResNet-18 distillation Experimental Analysis: (1) Ablation Experiment: To evaluate the effectiveness of each component in the method of this invention, an ablation experiment was conducted on a CIFAR-100, mainly considering two elements: Gaussian distribution alignment loss ( ) and Diversity Enhancement Regularizer (DER). As shown in Table 7, the analysis reveals four main findings: (1) The novel method of storing the complete Gaussian statistics (mean and covariance) of the pre-trained model in this invention shows superior effectiveness compared to existing methods. The mean alignment (MA) component alone improves the alignment performance of the BN layer of SRe2L by 15.6%, which demonstrates the importance of class-specific alignment. (2) When the covariance alignment (CA) component in the model is used in conjunction with MA, it provides additional performance improvements. This indicates that preserving the relationships between features through covariance matching is crucial for maintaining the diversity of feature dimensions. (3) The variance maximization regularization proposed in this invention is applied to the last layer of features. This ensures sufficient diversity in the semantic dimension, resulting in additional accuracy gains of 0.6% and 0.9%. This validates the hypothesis that feature space expansion promotes better knowledge representation. (4) Off-diagonal covariance minimization regularization This effectively reduced redundancy between dimensions, achieving additional performance improvements of 1.4% and 0.4%. This confirms that promoting the orthogonality of the last layer of feature dimensions can effectively reduce the impact of feature redundancy. Table 7 Ablation studies with CIFAR-100 IPC=10 / 50 (2) To Impact: Hyperparameters This represents the ratio between covariance and mean alignment within the framework of this invention. The covariance matrix, as a statistical descriptor, plays a crucial role in encoding intra-class variation and inter-dimensional correlations across feature dimensions. For example... Figure 2 As shown, on the CIFAR-100 dataset, when IPC=50, it was observed that... Increasing the value from 0 to 50 results in a significant performance improvement. However, if... Beyond this optimal range (increasing from 50 to 200), performance degrades. This phenomenon stems from the inherent trade-off between covariance alignment and mean matching: overemphasizing the preservation of covariance structure may inadvertently weaken the information of various inherent properties.

[0051] (3) To and Impact: Regularization weights (for variance minimization) and (For off-diagonal covariance maximization) different aspects of the feature distribution are addressed separately. In the pre-trained model, the features in the last layer are highly concentrated across classes, while also carrying high-level semantic information. It explicitly limits the diversity of features extracted from samples, while This encourages dimensionality-based decorrelation to enhance feature representation capabilities. For example... Figure 2 As shown, on the CIFAR-100 dataset, the best performance occurs when IPC=50. and In this case, it is worth noting that larger... The value may degrade performance due to over-expanding the feature space, which could affect class discriminability. In contrast, performance is... The parameter remains relatively stable despite changes, indicating that it has low sensitivity.

[0052] Visualization Analysis: To evaluate whether the method of this invention preserves the feature diversity of the original dataset, in Figure 3The document presents a visual comparison of SRe2L and the method of this invention on distillation data on CIFAR-100 and ImageNet-1K. Empirical observations yield two key insights: (1) On low-resolution data like CIFAR-100, the method of this invention demonstrates superior class separability compared to SRe2L, with distilled samples effectively preserving class-specific semantic information. This performance improvement validates the effectiveness of the class-specific Gaussian distribution matching strategy of this invention in maintaining inter-class distinction. (2) On high-resolution data like ImageNet-1K, the method of this invention not only maintains class separability but also captures richer semantic diversity. For example, distilled images of the 'sorrel' class (red-brown horse) retain a variety of fine-grained features, including horse posture, anatomical features, and color variations. This highlights the robustness of the method of this invention across different data scales.

[0053] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A dataset distillation method based on semantic relation decoupling, characterized in that, The method includes: pre-trained model In the original dataset Training is performed on the model, and the pre-trained model is extracted. Intermediate layer Features to estimate the original dataset Each category In the middle layer Feature distribution on ;in, , Represents the pre-trained model The number of floors, Indicates a Gaussian distribution. Represents the original dataset Medium category The sample is in the middle layer The mean vector on, Represents the original dataset Medium category The sample is in the middle layer The covariance matrix on; wherein, the original dataset For image datasets; Get Category Distillation data And based on the feature distribution Initialize the pre-trained model ; wherein, the data to be distilled Image data; Based on the initialized pre-trained model The data to be distilled Perform iterative distillation to obtain the distillation results; Among them, based on the pre-trained model after initialization The data to be distilled Perform iterative distillation to obtain distillation results, including: Distillation data Feed into the pre-trained model after initialization And in calculating cross-entropy loss Simultaneously, acquire distillation data Each category In the middle layer Feature distribution on and layers covariance matrix on ;in, Representing distillation data Medium category Data in the middle layer The mean vector on, Indicates the data to be distilled Medium category Data in the middle layer The covariance matrix on, Indicates the iteration round, distillation data For the data to be distilled ; The mean vectors are respectively Covariance matrix With mean vector Covariance matrix After alignment, calculate the Gaussian alignment loss. ; By maximizing the covariance matrix diagonal variance The variance regularization term is obtained. ; By minimizing the covariance matrix The off-diagonal covariance yields the covariance term. ; Based on cross-entropy loss Gaussian alignment loss Variance regularization term and covariance term The overall loss was obtained. ; Obtain distillation data Subsequently, based on this distillation data Re-execute the distillation data Feed into the pre-trained model after initialization until the set requirements are met; among them, Indicates the learning rate. This represents the gradient.

2. The method according to claim 1, characterized in that, By extracting pre-trained models Intermediate layer Features to estimate the original dataset Each category In the middle layer Feature distribution on ,include: Original dataset Medium category samples Input into the pre-trained model To obtain the intermediate layer Feature tensor ; For feature tensors Perform average pooling to obtain the pooled features. ; Based on pooling features Calculate the mean vector Covariance Matrix .

3. The method according to claim 1, characterized in that, The Gaussian distribution alignment loss ;in, This is a hyperparameter.

4. The method according to claim 1, characterized in that, The variance regularization term ;in, diagonal variance The Middle One element, diagonal variance Dimensions.

5. The method according to claim 1, characterized in that, The covariance term ;in, diagonal variance Dimensions , Let them represent the covariance matrix respectively. Bank number and column number.

6. A dataset distillation system based on semantic relation decoupling, characterized in that, The system includes: The feature distribution acquisition module is used to obtain the pre-trained model In the original dataset Training is performed on the model, and the pre-trained model is extracted. Intermediate layer Features to estimate the original dataset Each category In the middle layer Feature distribution on ;in, , Represents the pre-trained model The number of floors, Indicates a Gaussian distribution. Represents the original dataset Medium category The sample is in the middle layer The mean vector on, Represents the original dataset Medium category The sample is in the middle layer The covariance matrix on; wherein, the original dataset For image datasets; The data acquisition module is used to obtain categories. Distillation data ; wherein, the data to be distilled Image data; The model initialization module is used to initialize the model based on the feature distribution. Initialize the pre-trained model ; The distillation result generation module is used to generate distillation results based on the initialized pre-trained model. The data to be distilled Perform iterative distillation to obtain the distillation results; Among them, based on the pre-trained model after initialization The data to be distilled Perform iterative distillation to obtain distillation results, including: Distillation data Feed into the pre-trained model after initialization And in calculating cross-entropy loss Simultaneously, acquire distillation data Each category In the middle layer Feature distribution on and layers covariance matrix on ;in, Representing distillation data Medium category Data in the middle layer The mean vector on, Indicates the data to be distilled Medium category Data in the middle layer The covariance matrix on, Indicates the iteration round, distillation data For the data to be distilled ; The mean vectors are respectively Covariance matrix With mean vector Covariance matrix After alignment, calculate the Gaussian alignment loss. ; By maximizing the covariance matrix diagonal variance The variance regularization term is obtained. ; By minimizing the covariance matrix The off-diagonal covariance yields the covariance term. ; Based on cross-entropy loss Gaussian alignment loss Variance regularization term and covariance term The overall loss was obtained. ; Obtain distillation data Subsequently, based on this distillation data Re-execute the distillation data Feed into the pre-trained model after initialization until the set requirements are met; among them, Indicates the learning rate. This represents the gradient.

7. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the dataset distillation method based on semantic relationship decoupling as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the dataset distillation method based on semantic relation decoupling as described in any one of claims 1-5.

9. A computer program product, characterized in that, When the computer program product is run on a computer device, the computer device performs the dataset distillation method based on semantic relation decoupling as described in any one of claims 1-5.

Images