Data set distillation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, and provides a data set distillation method and device, electronic equipment and a storage medium, and the method comprises the steps: training a first classification model through an ith round synthetic sample set determined in ith round distillation, and obtaining an ith second classification model; carrying out the (i + 1) th round of active learning in the real sample set through the ith second classification model, and determining the (i + 1) th round of target sample set; the (i + 1) th round of target sample set comprises real samples screened from the real sample set in the (i + 1) th round of active learning; and carrying out (i + 1) th round of distillation by the target classification model according to the (i + 1) th round of target sample set and the ith round of synthetic sample set to obtain an (i + 1) th round of synthetic sample set, and if (i + 1) is equal to N, taking the (i + 1) th round of synthetic sample set as a data distillation result of the real sample set. According to the method provided by the invention, the data set distillation effect can be ensured, and the data set distillation efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data distillation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Dataset distillation (also known as dataset compression) has emerged to utilize large amounts of real datasets (such as real image sets, real text sets, real audio sets, etc.) to synthesize a small amount of synthetic data to approximate the training effect of the real dataset. Dataset distillation is not only crucial for accelerating the training of neural network models and reducing data storage requirements, but also better protects privacy because the synthetic data in the generated dataset is not real data from the real dataset.

[0003] The dataset distillation process involves multiple distillation iterations. In related technologies, every piece of real data in the real dataset is included in each distillation iteration for matching. However, due to the large amount of data in the real dataset, including every piece of real data in each distillation iteration for matching results in a long dataset distillation time and high computational cost. Summary of the Invention

[0004] In view of this, embodiments of this application propose a data set distillation method, apparatus, electronic device, and storage medium to address the problems of long processing time and high computational overhead in related technologies for data set distillation.

[0005] In a first aspect, embodiments of this application provide a dataset distillation method, including:

[0006] The first classification model is trained using the synthetic sample set determined in the i-th round of distillation to obtain the i-th second classification model; wherein the architecture of the first classification model is the same as that of the target classification model used in distillation learning: i is a positive integer;

[0007] The i-th second classification model performs the (i+1)-th round of active learning on the real sample set to determine the (i+1)-th round target sample set; the (i+1)-th round target sample set includes real samples selected from the real sample set in the (i+1)-th round of active learning.

[0008] The target classification model performs distillation in the (i+1)th round based on the target sample set in the (i+1)th round and the synthetic sample set in the (i)th round to obtain the synthetic sample set in the (i+1)th round. The synthetic sample set in the (i+1)th round is obtained by updating the synthetic samples in the synthetic sample set in the (i)th round.

[0009] If i+1 is less than N, increment i by 1 and return to the step of training the first classification model with the synthetic sample set determined in the i-th round of distillation to obtain the i-th second classification model; N is an integer greater than 1.

[0010] If i+1 equals N, the synthesized sample set of the (i+1)th round is used as the data distillation result of the real sample set.

[0011] Secondly, embodiments of this application provide a dataset distillation apparatus, comprising:

[0012] The training module is used to train the first classification model using the synthetic sample set determined in the i-th round of distillation, to obtain the i-th second classification model; wherein the architecture of the first classification model is the same as that of the target classification model used in distillation learning; i is a positive integer;

[0013] An active learning module is used to perform the (i+1)th round of active learning on the real sample set through the i-th second classification model to determine the (i+1)th round target sample set; the (i+1)th round target sample set includes real samples selected from the real sample set in the (i+1)th round of active learning.

[0014] The distillation module is used to perform distillation in the (i+1)th round by the target classification model based on the (i+1)th round target sample set and the (i)th round synthetic sample set to obtain the (i+1)th round synthetic sample set, which is obtained by updating the synthetic samples in the (i)th round synthetic sample set.

[0015] The loop processing module is used to increment i by 1 if i+1 is less than N, and then return to the step of training the first classification model with the first round of synthetic sample set determined by the i-th round of distillation to obtain the i-th second classification model; N is an integer greater than 1.

[0016] The data distillation result determination module is used to take the (i+1)th round of synthesized sample set as the data distillation result of the real sample set if i+1 equals N.

[0017] In some embodiments, the active learning module includes:

[0018] A determining unit is configured to determine the (i+1)th round real sample set in the real sample set; the (i+1)th round real sample set includes all or part of the real samples in the real sample set.

[0019] The first computing unit is used to calculate the loss gradient of each real sample in the (i+1)th round of the real sample set on the i-th second classification model;

[0020] Clustering unit, used to cluster the real samples in the (i+1)th round real sample set according to the loss gradient, to obtain the clustering result;

[0021] A filtering unit is used to filter out real samples in the (i+1)th round of real sample set whose loss gradient distance with the cluster center is less than a distance threshold, based on the cluster center in the clustering result.

[0022] An addition unit is used to add the selected real samples to the target sample set of the (i+1)th round.

[0023] In some embodiments, the first computing unit includes:

[0024] The first classification unit is used to classify each real sample in the (i+1)th round of real sample set by the i-th second classification model to obtain the first predicted category of each real sample in the (i+1)th round of real sample set.

[0025] The first prediction loss determination unit is used to calculate the first prediction loss of each real sample in the (i+1)th round real sample set based on the labeled category and the first prediction category of each real sample in the (i+1)th round real sample set.

[0026] The loss gradient determination unit is used to calculate the gradient of the first predicted loss of each real sample in the (i+1)th round of the real sample set relative to the i-th second classification model, so as to obtain the loss gradient of each real sample in the (i+1)th round of the real sample set on the i-th second classification model.

[0027] In some embodiments, the i-th second classification model includes a feature extraction network and a classifier; the loss gradient of the real sample on the i-th second classification model includes the loss gradient of the real sample on the classifier; the loss gradient determination unit is further configured to:

[0028] Calculate the gradient of the first prediction loss of each real sample in the (i+1)th round real sample set relative to the classifier, and obtain the loss gradient of each real sample in the (i+1)th round real sample set on the classifier.

[0029] In some embodiments, the determining unit is further configured to: take the difference between the real sample set and the target sample set of the i-th round as the real sample set of the (i+1)-th round.

[0030] In some embodiments, the target sample set for the (i+1)th round may also include real samples from the target sample set for the i-th round.

[0031] In some embodiments, the distillation module includes:

[0032] The acquisition unit is used to acquire a reference classification model that has the same architecture and model parameters as the target classification model.

[0033] The first loss gradient calculation unit is used to calculate the first loss gradient of the target sample set in the (i+1)th round on the target classification model;

[0034] The second loss gradient calculation unit is used to calculate the second loss gradient of the i-th round of synthetic sample set on the reference classification model;

[0035] A loss gradient difference determination unit is used to determine the loss gradient difference between the first loss gradient and the second loss gradient;

[0036] An iterative unit is used to iteratively update the synthetic samples in the i-th round synthetic sample set with the goal of minimizing the difference in the loss gradient, until the distillation cycle stopping condition is met; when the distillation cycle stopping condition is met, the updated i-th round synthetic sample set is used as the (i+1)-th round synthetic sample set.

[0037] In some embodiments, the first loss gradient calculation unit includes:

[0038] The second classification unit is used to classify each real sample in the target sample set of the (i+1)th round by the target classification model to obtain the second predicted category of each real sample in the target sample set of the (i+1)th round.

[0039] The second prediction loss determination unit is used to determine the second prediction loss of the target sample set in the (i+1)th round based on the labeling category and the second prediction category of each real sample in the target sample set in the (i+1)th round.

[0040] The second calculation unit is used to calculate the gradient of the second prediction loss of the target sample set in the (i+1)th round relative to the target classification model, so as to obtain the first loss gradient of the target sample set in the (i+1)th round on the target classification model.

[0041] In some embodiments, the second prediction loss determination unit includes:

[0042] The sub-prediction loss determination unit is used to calculate the loss of each real sample in the target sample set of the (i+1)th round by performing loss calculation on the labeled category and the second prediction category of each real sample in the target sample set of the (i+1)th round.

[0043] The weighted average unit is used to perform a weighted average of the sub-prediction losses of the real samples in the target sample set of the (i+1)th round to obtain the second prediction loss of the target sample set of the (i+1)th round.

[0044] In some embodiments, the loss gradient difference determination unit is further configured to:

[0045] Calculate the cosine similarity between the first loss gradient and the second loss gradient;

[0046] Based on the cosine similarity, the difference in loss gradient between the first loss gradient and the second loss gradient is determined, and the difference in loss gradient is negatively correlated with the cosine similarity.

[0047] In some embodiments, the dataset distillation apparatus further includes:

[0048] An initial synthetic sample set acquisition module is used to acquire an initial synthetic sample set, which includes multiple synthetic samples and the annotation labels of each synthetic sample;

[0049] The first training module is used to train the first classification model using the initialized synthetic sample set to obtain the first second classification model;

[0050] The first active learning module is used to perform the first round of active learning on the real sample set through the first second classification model to obtain the first round of target sample set;

[0051] The first distillation module is used to perform a first round of distillation by the target classification model based on the first round target sample set and the initial synthetic sample set to obtain a first round synthetic sample set.

[0052] In some embodiments, the real sample is a real image, and the synthetic sample is a synthetic image.

[0053] Thirdly, embodiments of this application provide an electronic device, including: a processor; and a memory storing computer instructions, which, when executed by the processor, implement the above-described method.

[0054] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the aforementioned method.

[0055] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the above-described method.

[0056] In this application, the changing real data requirements at different stages of the data distillation process are fully considered. Specifically, the importance of real samples in the real sample set dynamically changes at different stages of the data distillation process. Therefore, before distillation, the first classification model is trained using the i-th round of synthetic sample set determined by the i-th round of distillation to obtain the i-th second classification model. This i-th second classification model reflects the distillation state after the i-th round of distillation and reflects the subsequent dataset distillation requirements. Then, the i-th second classification model performs the (i+1)-th round of active learning on the real sample set to determine the (i+1)-th round target sample set. The target classification model then performs the (i+1)-th round of distillation based on the (i+1)-th round target sample set and the i-th round synthetic sample set to obtain the (i+1)-th round synthetic sample set. The i-th second classification model reflects the distillation state after the i-th round of distillation. Thus, in the (i+1)-th round of active learning, the i-th second classification model dynamically selects high-value real samples with the current distillation state from the real sample set to meet the needs of subsequent distillation learning. This ensures that the target sample set of the (i+1)-th round is used for the (i+1)-th round of distillation, and that the determined synthetic sample set for the (i+1)-th round effectively covers the effective information in the real sample set, guaranteeing the dataset distillation effect. Furthermore, in this application's scheme, the target sample set determined in the (i+1)-th round of active learning is used for the (i+1)-th round of distillation, rather than using the entire real sample set. This significantly reduces the time and computational cost of each round of distillation, improving the efficiency of dataset distillation.

[0057] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application.

[0060] Figure 2 This is a flowchart illustrating a dataset distillation method according to an embodiment of this application.

[0061] Figure 3 This is a flowchart of step 220 in one embodiment of this application.

[0062] Figure 4 This is a flowchart of step 230 in one embodiment of this application.

[0063] Figure 5 This is a flowchart illustrating a distillation process according to an embodiment of this application.

[0064] Figure 6 This is a flowchart illustrating the steps preceding step 210 according to an embodiment of this application.

[0065] Figure 7 This is a schematic diagram illustrating a dataset distillation method according to an embodiment of this application.

[0066] Figure 8 This is a schematic diagram showing real samples selected at different stages in the feature space and gradient space according to an embodiment of this application.

[0067] Figure 9 This is a block diagram of an intent classification device shown in one embodiment of this application.

[0068] Figure 10 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0069] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0070] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0071] In this document, "multiple" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. In the following description, references to "some embodiments or some embodiment methods" describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0072] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application. For example... Figure 1As shown, the application scenario includes an electronic device that can upload a real sample set to the electronic device and deploy a target classification model and a first classification model on the electronic device. The electronic device can distill the real sample set according to the method of this application and output the data distillation result of the real sample set.

[0073] like Figure 1 As shown, in the scheme of this application, the process involved in one round of active learning and one round of distillation includes: training a first classification model using the synthetic sample set determined in the previous round of distillation to obtain a second classification model, which can reflect the current data distillation state; then, using the second classification model to perform active learning on the real dataset in the current round, so as to select representative real samples from the real sample set and add them to the target sample set, that is, to select real samples with high value for subsequent distillation and add them to the target sample set; then, using the target classification model to perform distillation using the determined target sample set and the synthetic sample set determined in the previous round of distillation, so as to iteratively update the synthetic sample set determined in the previous round of distillation and obtain the synthetic sample set of the current round of distillation.

[0074] If the number of rounds of active learning does not reach the set threshold N (N is an integer greater than 1), repeat the above active learning and distillation process until the number of rounds of active learning reaches the set threshold N, and use the synthetic sample set determined by the Nth round of distillation as the data distillation result of the real sample set.

[0075] In this case, the number of synthetic samples in the data distillation results of the real sample set is less than the number of real samples in the real sample set. Subsequently, the synthetic sample set can be used instead of the real sample set to train the model, resulting in a performance that is essentially equivalent to that of a model trained using the real sample set.

[0076] Electronic devices can be servers, terminals, edge service devices, or other devices with computing capabilities. Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0077] In some embodiments, an electronic device can implement the dataset distillation method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP); it can be a small program, i.e., a program that only needs to be downloaded into a browser environment to run; or it can be a small program or plugin that can be embedded into any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0078] Figure 2 This is a flowchart illustrating a dataset distillation method according to an embodiment of this application. The dataset distillation method can be performed by an electronic device, such as a terminal or server as described above. Figure 2 As shown, the method includes steps 210 to 250, which are described in detail below:

[0079] Step 210: Train the first classification model using the synthetic sample set determined in the i-th round of distillation to obtain the i-th second classification model; wherein the architecture of the first classification model is the same as that of the first classification model used in distillation learning; i is a positive integer.

[0080] The purpose of data distillation is to create a small synthetic dataset from a large set of real samples, so that a model trained on the small synthetic dataset can achieve comparable training results to a model trained on the large set of real samples. The resulting small synthetic dataset is the result of data distillation from the real dataset.

[0081] The i-th round of synthetic sample set includes multiple synthetic samples and the labeled categories of each synthetic sample. The real sample set includes multiple real samples and the labeled categories of each real sample, i.e., the actual categories to which the real samples belong.

[0082] The scheme of this application can be applied to image set distillation, where the real samples in the real sample set are real images, and the synthetic samples in the synthetic sample set (e.g., the i-th round of synthetic sample set) are synthetic images. It can also be applied to text set distillation, where the real samples in the real sample set are real text (e.g., text actually created by the creator), and the synthetic samples in the synthetic sample set (e.g., the i-th round of synthetic sample set) are synthetic text. Furthermore, it can be applied to audio set distillation, where the real samples in the real sample set are real audio (e.g., collected human voice, collected environmental audio, etc.), and the synthetic samples in the synthetic sample set are synthetic audio.

[0083] The first classification model is a neural network model suitable for classifying real samples in a real sample set. For example, if the real sample is a real image, the first classification model is an image classification model; if the real sample is real text, the first classification model is a text classification model; if the real sample is real audio, the first classification model is an audio classification model.

[0084] The classification model used in the distillation process is called the target classification model. The target classification model and the first classification model perform the same classification task, such as image classification, text classification, or audio classification, etc., without further specific limitations.

[0085] In some embodiments, it is assumed that the classification model to be trained using the real sample set or the data distillation results of the real sample set is called the third classification model. The target classification model can be another model with the same classification task as the third classification model, or the target classification model can also be the third classification model.

[0086] The first classification model has the same architecture as the target classification model, which means that the first classification model and the target classification model have the same structure, including the number of neural network layers, the type of corresponding neural network layers (e.g., both are fully connected network layers, both are convolutional network layers, both are pooling network layers), the number of neurons and channels in the corresponding neural network layers are the same, and the activation functions deployed in the corresponding neural network layers are the same, etc.

[0087] In some embodiments, the target classification model after parameter initialization can be copied before distillation learning, or the target classification model during the distillation learning process can be copied, and the copied model can be used as the first classification model.

[0088] Considering that the scheme of this application involves multiple trainings of the first classification model, and the training data used in different training sessions are different, the classification model obtained by training the first classification model through the i-th round of synthetic sample set is called the i-th second classification model.

[0089] In some embodiments, step 210 includes the following 1)-3):

[0090] 1) The first classification model is used to classify each synthetic sample in the i-th round of synthetic sample set to obtain the reference predicted category of each synthetic sample in the i-th round of synthetic sample set.

[0091] The first classification model can extract features from each synthetic sample in the i-th round of synthetic sample set to obtain the sample features of the synthetic sample. Then, it performs classification prediction based on the sample features of the synthetic sample and outputs the reference predicted category of the synthetic sample.

[0092] 2) Calculate the reference prediction loss based on the labeled category and reference prediction category of each synthetic sample in the i-th round synthetic sample set.

[0093] The first loss function can be used to calculate the loss between the labeled category and the reference prediction category of each synthetic sample in the i-th round synthetic sample set, thus obtaining the reference prediction loss. Examples of first loss functions include cross-entropy loss, absolute value loss, and mean squared error loss, etc., which are not specifically limited here.

[0094] 3) Update the parameters of the first classification model according to the reference prediction loss until the training termination condition is met. The first classification model with updated parameters after the training termination condition is met is used as the i-th second classification model.

[0095] In the above process, the first classification model is trained with the goal of minimizing the reference prediction loss function. The training objective of the first classification model can be described as follows: Formula 1:

[0096]

[0097] Among them, S i L represents the synthetic sample set of the i-th round; CE (θ,S i () indicates that, based on the parameters θ of the first classification model, for the synthetic sample set S in the i-th round... i The reference prediction loss is calculated based on the reference prediction category predicted by the synthetic samples in the model; for example, it could be the reference prediction loss calculated using the cross-entropy loss function. AC These are the parameters of the i-th second classification model.

[0098] The training termination condition can be the convergence of the loss function of the first classification model, for example, when the reference prediction loss calculated using the loss function is less than the loss threshold, or when the number of iterations of the first classification model reaches the threshold. No specific limitation is made here.

[0099] Step 220: Perform the (i+1)th round of active learning on the real sample set using the i-th second classification model to determine the (i+1)th round target sample set; the (i+1)th round target sample set includes the real samples selected from the real sample set in the (i+1)th round of active learning.

[0100] In this application, representative real samples are selected from the real sample set through active learning and added to the target sample set in the (i+1)th round. A representative real sample can be understood as a real sample carrying a high amount of information. Real samples with higher information content are better at improving the model's training effect. Correspondingly, the synthetic samples obtained by synthesizing samples using these high-information real samples also have a more significant effect on improving the model's training effect.

[0101] Since the i-th secondary classification model is determined through training on the i-th round of synthetic sample set determined in the i-th round of distillation, it reflects the distillation state after the i-th round of distillation. In other words, the performance of the i-th secondary classification model corresponds to the distillation state after the i-th round of distillation; or, the performance of the i-th secondary classification model is equivalent to the performance of the classification model after training it using the i-th round of synthetic sample set. Thus, in step 220, the i-th secondary classification model can dynamically select real samples with high value to the current distillation state from the real sample set. That is, the selected real samples are those that improve the distillation state. All real samples in the (i+1)-th round target sample set are selected from the real sample set.

[0102] In some embodiments, in the (i+1)th round of active learning, all real samples in the real sample set can be used as the basis for filtering. In other embodiments, since some real samples have already been filtered out in previous rounds of active learning, to improve filtering efficiency and avoid duplicate filtering, in the (i+1)th round of active learning, other real samples in the real sample set excluding the filtered real samples can be used as the basis for filtering. For ease of description, the set of real samples used as the basis for filtering in the (i+1)th round of active learning is called the (i+1)th round real sample set.

[0103] In some embodiments, in the (i+1)th round of active learning, the loss gradient of each real sample in the (i+1)th round of real sample set on the i-th second classification model can be calculated. Then, based on the loss gradient, representative real samples are selected from the (i+1)th round of real sample set and added to the (i+1)th round of target sample set. The specific process can be described below.

[0104] In other embodiments, during the (i+1)th round of active learning, the i-th secondary classification model can classify each real sample in the (i+1)th round of real sample set, obtaining a category prediction result for each real sample in the (i+1)th round of real sample set. The category prediction result includes the predicted category for the real sample and a confidence level for that predicted category. The confidence level for the predicted category indicates the degree of certainty of the i-th secondary classification model that the real sample belongs to that predicted category. A higher confidence level indicates a higher degree of certainty from the i-th secondary classification model that the real sample belongs to that predicted category; conversely, a lower confidence level indicates a lower degree of certainty from the i-th secondary classification model that the real sample belongs to that predicted category.

[0105] Based on this, and using the labeled categories of each real sample in the (i+1)th round of the real sample set, as well as the category prediction results of each real sample, we can filter out real samples from the (i+1)th round of the real sample set where the predicted category indicated by the category prediction result is different from the corresponding labeled category, or where the predicted category indicated by the category prediction result is the same as the corresponding labeled category, but the confidence level of the predicted category in the category prediction result is lower than the confidence threshold. Alternatively, we can filter out a portion of real samples from those where the predicted category is different from the corresponding labeled category, or where the predicted category indicated by the category prediction result is the same as the corresponding labeled category, but the confidence level of the predicted category in the category prediction result is lower than the confidence threshold. These filtered real samples are added to the target sample set for the (i+1)th round. The filtered real samples are those that are uncertain for the i-th second classification model, and these uncertain real samples are more valuable for subsequent distillation processes. The confidence threshold set for different rounds of active learning can be the same or different. In some embodiments, the higher the round of active learning, the higher the set confidence threshold.

[0106] Step 230: The target classification model performs distillation in the (i+1)th round based on the target sample set in the (i+1)th round and the synthetic sample set in the (i)th round to obtain the synthetic sample set in the (i+1)th round. The synthetic sample set in the (i+1)th round is obtained by updating the synthetic samples in the synthetic sample set in the (i)th round.

[0107] The purpose of distillation is to ensure that training the model using real samples from the (i+1)th round target sample set yields essentially the same results as training the model using the (i+1)th round synthetic sample set. In other words, it's equivalent to transferring the information carried by the real samples from the (i+1)th round target sample set to the synthetic samples from the (i+1)th round synthetic sample set.

[0108] Since the real samples in the target sample set of round i+1 are only a portion of the real samples in the real sample set, using the target sample set of round i+1 for distillation learning can significantly reduce computational overhead compared to using all the real samples in the real sample set for distillation learning.

[0109] In some embodiments, the target classification model can be replicated to obtain a reference classification model with the same architecture and parameters as the target classification model. Then, in step 230, the synthetic samples in the synthetic sample set of the (i+1)th round can be iteratively updated at least once, with the objective of minimizing the difference between the first loss gradient of the target sample set in the target classification model and the second loss gradient of the synthetic sample set in the i-th round on the reference classification model, until the distillation cycle stops. The specific process is described below.

[0110] In some embodiments, during the (i+1)th distillation round, the parameters of the synthetic samples and the reference classification model in the synthetic sample set of the i-th round can be iteratively updated.

[0111] In some embodiments, the global prediction loss of the target classification model for the (i+1)th round target sample set and the global prediction loss of the target classification model for the i-th round synthetic sample set can be determined. The goal is to minimize the global prediction loss between the two, and the synthetic sample set for at least the i-th round is iteratively updated to obtain the (i+1)th round synthetic sample set. The global loss of the target classification model for the (i+1)th round target sample set can be determined according to the following processes ①-③:

[0112] ① The target classification model is used to classify each real sample in the target sample set of round i+1 to obtain the second predicted category of each real sample in the target sample set of round i+1.

[0113] ② Based on the labeled category and the second predicted category of each real sample in the target sample set of round i+1, determine the second sub-prediction loss of each real sample in the target sample set of round i+1. In this process, the second sub-prediction loss can be calculated using cross-entropy loss function, mean squared error loss function, absolute value loss function, etc., in a similar manner as above.

[0114] ③ Take the weighted average of the second sub-prediction loss of the real samples in the target sample set of the (i+1)th round to obtain the global loss of the target classification model for the target sample set of the (i+1)th round.

[0115] In some embodiments, the target classification model can be copied to obtain a reference classification model with the same architecture and model parameters as the target classification model. Based on this, step 230 may include the following (1)-(6):

[0116] (1) The feature extraction network in the target classification model extracts features from each real sample in the target sample set of round i+1 to obtain the sample features of each real sample in the target sample set of round i+1; the classifier in the target classification model classifies each real sample in the target sample set of round i+1 according to the sample features of each real sample in the target sample set of round i+1 to obtain the second predicted category of each real sample in the target sample set of round i+1.

[0117] (2) The feature extraction network in the reference classification model extracts features from each synthetic sample in the i-th round of synthetic sample set to obtain the sample features of each synthetic sample in the i-th round of synthetic sample set; the classifier in the reference classification model classifies each synthetic sample in the i-th round of synthetic sample set according to the sample features of each synthetic sample in the i-th round of synthetic sample set to obtain the third predicted category of each synthetic sample in the i-th round of synthetic sample set.

[0118] Among them, (1)-(2) above can be carried out in parallel, which can shorten the distillation processing time.

[0119] (3) Based on the sample features of each real sample in the target sample set of round i+1 and the sample features of each synthetic sample in the synthetic sample set of round i, determine the feature distribution loss between the target sample set of round i+1 and the synthetic sample set of round i.

[0120] (4) Based on the second predicted category and labeled category of each real sample in the target sample set of round i+1, determine the global loss of the target classification model for the target sample set of round i+1, and based on the third predicted category and labeled category of each synthetic sample in the synthetic sample set of round i, determine the global prediction loss of the reference classification model for the synthetic sample set of round i.

[0121] (5) Determine the difference in prediction loss based on the third predicted category and labeled category of each synthetic sample in the i-th round of synthetic sample set, and the global prediction loss of the reference classification model for the i-th round of synthetic sample set;

[0122] (6) With the goal of minimizing the weighted result of the difference in prediction loss and the feature distribution loss, the synthetic samples in the synthetic sample set of the i-th round are updated at least iteratively until the distillation termination condition is met.

[0123] It is worth mentioning that during a round of distillation, the synthesized samples in the i-th round of synthesized sample set may be updated iteratively multiple times. The i-th round of synthesized sample set after the previous iteration serves as the basis for the next iteration update.

[0124] After step 230, if i+1 is less than N, execute step 240, increment i by 1, and return to step 210; N is an integer greater than 1; if i+1 is equal to N, execute step 250, and use the synthesized sample set of the (i+1)th round as the data distillation result of the real sample set.

[0125] If i+1 is less than N, increment i by 1 and continue the training of the first classification model, active learning, and distillation process described above. Here, N is the maximum number of active learning rounds, or the maximum number of distillation rounds.

[0126] In this application, the changing real data requirements at different stages of the data distillation process are fully considered. Specifically, the importance of real samples in the real sample set dynamically changes at different stages of the data distillation process. Therefore, before distillation, the first classification model is trained using the synthetic sample set determined in the i-th round of distillation to obtain the i-th second classification model. This i-th second classification model reflects the distillation state after the i-th round of distillation and reflects the requirements of subsequent dataset distillation. Then, the i-th second classification model performs the (i+1)-th round of active learning on the real sample set to determine the (i+1)-th round target sample set. The target classification model then performs the (i+1)-th round of distillation based on the (i+1)-th round target sample set and the synthetic sample set from the i-th round, obtaining the (i+1)-th round synthetic sample set. The i-th second classification model reflects the distillation state after the i-th round of distillation. Thus, in the (i+1)-th round of active learning, the i-th second classification model can dynamically select high-value real samples with the current distillation state from the real sample set to meet the needs of subsequent distillation learning. This ensures that the target sample set of the (i+1)-th round is used for the (i+1)-th round of distillation, and the determined synthetic sample set for the (i+1)-th round effectively covers the effective information in the real sample set. Furthermore, in this application's scheme, the target sample set determined in the (i+1)-th round of active learning is used for the (i+1)-th round of distillation, rather than using the entire real sample set. This significantly reduces the time and computational cost of each round of distillation, improving the efficiency of dataset distillation.

[0127] In some embodiments, such as Figure 3 As shown, step 220 includes steps 310-350, which are detailed below:

[0128] Step 310: Determine the real sample set for the (i+1)th round from the real sample set; the real sample set for the (i+1)th round includes all or part of the real samples in the real sample set.

[0129] The real sample set in round i+1 serves as the basis for real sample selection in round i+1 active learning. The real samples selected in round i+1 active learning are derived from the real sample set in round i+1.

[0130] In some embodiments, the real sample set can be used as the real sample set for the (i+1)th round. In other embodiments, the difference between the real sample set and the target sample set for the i-th round is used as the real sample set for the (i+1)th round. The target sample set for the i-th round is the target sample set determined in the i-th round of active learning; that is, the real samples in the target sample set for the i-th round are the real samples that have already been selected. In this way, using the difference between the real sample set and the target sample set for the i-th round as the basis for selecting real samples in the (i+1)th round of active learning avoids repeatedly evaluating the value of the same real sample for subsequent distillation in multiple rounds of active learning, thus improving processing efficiency.

[0131] Step 320: Calculate the loss gradient of each real sample in the (i+1)th round of the real sample set on the i-th second classification model.

[0132] The loss gradient of a real sample on the i-th second classification model reflects the rate of change of the loss function in the parameter space of the second classification model. That is, the loss gradient of a real sample on the i-th second classification model is the gradient of the loss corresponding to the real sample with respect to the model parameters of the second classification model.

[0133] In some embodiments, the loss gradient of each real sample on the i-th second classification model can be determined according to the following steps A1-A3:

[0134] Step A1: The i-th second classification model classifies each real sample in the (i+1)-th round of real sample set to obtain the first predicted category of each real sample in the (i+1)-th round of real sample set.

[0135] The i-th second classification model can extract features from each real sample in the (i+1)-th round of real sample set to obtain the sample features of the real samples. Then, it performs classification prediction based on the sample features of the real samples and outputs the first predicted category of the real samples.

[0136] Step A2: Calculate the first prediction loss for each real sample in the (i+1)th round of the real sample set based on the labeled category and the first prediction category of each real sample in the (i+1)th round of the real sample set.

[0137] The second loss function can be used to calculate the loss between the labeled category and the first predicted category of each real sample in the (i+1)th round real sample set, so as to obtain the first prediction loss of each real sample in the (i+1)th round real sample set. The second loss function can be, for example, the absolute value loss function, the mean squared error loss function, the cross-entropy loss function, etc., and no specific limitation is made here.

[0138] Step A3: Calculate the gradient of the first prediction loss of each real sample in the (i+1)th round of the real sample set relative to the i-th second classification model, and obtain the loss gradient of each real sample in the (i+1)th round of the real sample set on the i-th second classification model.

[0139] The gradient of the first prediction loss relative to the second classification model can be the gradient of the first prediction loss with respect to the model parameters of the second classification model. In some embodiments, the partial derivative of the first prediction loss of a real sample with respect to the model parameters of the second classification model can be taken, and the result is the loss gradient of that real sample on the second classification model.

[0140] In some embodiments, the gradient of the first prediction loss of a real sample relative to all the model parameters of the second classification model can be used as the loss gradient of the real sample on the second classification model.

[0141] In other embodiments, the i-th second classification model includes a feature extraction network and a classifier; the loss gradient of a real sample in the i-th second classification model includes the loss gradient of the real sample in the classifier; in step A3, the gradient of the first predicted loss of each real sample in the (i+1)-th round of the real sample set relative to the classifier is calculated to obtain the loss gradient of each real sample in the (i+1)-th round of the real sample set in the classifier. In this case, the loss gradient of a real sample in the classifier of the i-th second classification model is equivalent to the loss gradient of the real sample in the i-th second classification model.

[0142] For example, if the first prediction loss above is calculated using the cross-entropy loss function, the loss gradient of the j-th real sample in the (i+1)-th round of the real sample set on the classifier in the i-th second classification model can be expressed as:

[0143]

[0144] in, θ represents the loss gradient of the j-th real sample in the (i+1)-th round of the real sample set on the classifier in the i-th second classification model. c Let θ(x) be the parameter of the classifier in the i-th second classification model. j ) represents the first predicted category of the j-th real sample in the (i+1)-th round of the real sample set, and θ represents the parameters of the i-th second classification model; y j Let be the labeled category of the j-th real sample in the (i+1)-th round of the real sample set. CE (θ(x j ), y j That is, the first prediction loss calculated according to the cross-entropy loss function for the j-th real sample in the (i+1)-th round of the real sample set.

[0145] Expanding the right side of the equals sign in formula 2 above, we get the following formula (3):

[0146]

[0147] Where, θ e Let θ represent the parameters of the feature extraction network in the i-th second classification model. e (x j This can be understood as the sample features extracted by the feature extraction network in the i-th second classification model for the j-th real sample in the (i+1)-th round of real sample set.

[0148] In the above embodiment, the loss gradient of a real sample on the classifier in the i-th second classification model is used as the loss gradient of the real sample in the i-th second classification model, without involving the calculation of the gradient of the feature extraction network of the real sample in the i-th second classification model. This can greatly save computational costs and significantly reduce computational overhead without losing the overall gradient characteristics.

[0149] Step 330: Based on the loss gradient, cluster the real samples in the (i+1)th round of the real sample set to obtain the clustering result.

[0150] In step 330, the loss gradients of each real sample in the (i+1)th round of the real sample set are used. The real samples in the (i+1)th round of the real sample set are then clustered in the gradient space (i.e., the space where the loss gradients reside). During the clustering process in the gradient space, real samples in the (i+1)th round of the real sample set whose loss gradients are closest to each other are grouped into the same class. The clustering results indicate the cluster centers of each cluster category and the cluster centers to which each real sample in the (i+1)th round of the real sample set belongs; one cluster center represents one cluster category.

[0151] In some embodiments, K-means clustering can be used to cluster the real samples in the (i+1)th round of the real sample set in the gradient space to obtain the clustering result. This can maintain the diversity of gradients. For example, clustering can be performed as follows: ① Randomly select the loss gradients of K real samples from the (i+1)th round of the real sample set as K initial cluster centers; ② Calculate the distance between the loss gradients of other real samples in the (i+1)th round of the real sample set and each initial cluster center, and assign the other real samples in the (i+1)th round of the real sample set to the nearest cluster center; ③ Calculate the new centroid in each cluster category (e.g., calculate the mean of the loss gradients of all real samples belonging to the cluster category, and use the mean as the new centroid; or randomly select the loss gradient of a real sample from each cluster category as the new centroid), and use it as the new cluster center; ④ Repeat the above process ②-③ based on the new cluster centers until the change in the cluster center between two iterations does not exceed the magnitude threshold, or the number of clustering iterations reaches the number threshold.

[0152] The total number of cluster categories (K) can be set according to actual needs. For example, the total number of labeled categories involved in the real samples in the real sample set can be used as K, thus ensuring that the real samples selected later cover real samples under all labeled categories.

[0153] In some embodiments, density-based clustering algorithms (e.g., density-based clustering with noise) can also be used to cluster the real samples in the (i+1)th round of the real sample set according to the loss gradient.

[0154] Step 340: Based on the cluster centers in the clustering results, select real samples from the real sample set in the (i+1)th round whose loss gradient distance from the cluster centers is less than the distance threshold.

[0155] For each cluster center, in the (i+1)th round of real sample set, real samples whose loss gradient distance to the cluster center is less than the distance threshold can be selected.

[0156] The loss gradient distance between a real sample and its cluster center can be either the Euclidean distance between the loss gradient of the real sample and the loss gradient represented by the cluster center of the cluster to which the real sample belongs, or it can be the cosine distance. The distance threshold can also be set as needed. The distance threshold set for different rounds of active learning can be the same or different.

[0157] In some embodiments, the active learning rounds can be negatively correlated with the distance threshold. For example, when the active learning rounds are small (e.g., less than the first round threshold), a larger distance threshold is used to filter real samples, which can quickly filter out some valuable real samples; when the active learning rounds are large (e.g., not less than the first round threshold), a smaller distance threshold is used to filter real samples, which can filter out high-value real samples.

[0158] Step 350: Add the selected real samples to the target sample set for round i+1.

[0159] In some embodiments, the target sample set for the (i+1)th round of active learning may include only the real samples selected in the (i+1)th round of active learning. For example, the real samples in the (i+1)th round of active learning may be all real samples in the real sample set. In this case, since each round of active learning selects from the real sample set, the same real samples may exist in the target sample sets determined in different rounds of active learning.

[0160] In some embodiments, the (i+1)th round of active learning's real sample set is the difference between the real sample set and the target sample set of the i-th round. The (i+1)th round's target sample set may include real samples selected from the (i+1)th round's real sample set in the (i+1)th round of active learning and real samples from the target sample set of the i-th round. In other words, the (i+1)th round's target sample set is the real samples selected from the (i+1)th round's real sample set in the (i+1)th round of active learning and samples from the target sample set of the i-th round. Thus, since each round of active learning selects real samples from the set of previously unselected real samples (i.e., the difference between the real sample set and the target sample set of the i-th round), it eliminates the need to repeatedly evaluate the value of already selected real samples for subsequent distillation in multiple rounds of active learning, reducing computational overhead.

[0161] Furthermore, since the target sample set of round i+1 can include real samples selected from the real sample set of round i+1 in the active learning round i+1 and real samples from the target sample set of round i, the real samples selected in the previous round are also retained in the target sample set of round i+1 and used for round i+1 distillation. This can prevent the synthetic samples from overfitting to new samples in the subsequent distillation process and avoid catastrophic forgetting of the knowledge already captured in the distillation process. This ensures the quality of the synthetic sample set of round i+1 determined by the subsequent distillation. The subsequent use of the synthetic sample set of round i+1 to continuously train the first classification model can also better reflect the information state contained in the synthetic sample set and provide a better sampling importance estimate than discarding the existing target set.

[0162] In the above embodiment, the loss gradient of the real samples in the i-th round of the real sample set used as the current round of active learning is applied to the second classification model. Based on the loss gradient, the real samples in the i-th round of the real sample set are clustered in the gradient space. Based on the clustering results, real samples that are closer to the cluster center are selected and added to the target sample set. In this way, the target sample set used for distillation is dynamically adjusted to meet the ever-changing distillation learning requirements, thereby improving distillation efficiency and enhancing the quality of the synthesized sample set throughout the distillation process.

[0163] Since the clustering results in gradient space are used to select real samples that are close to the cluster centers and add them to the target sample set, initially, almost all real samples have large loss gradients, and the distribution of loss gradients among multiple real samples is relatively uniform. After clustering multiple real samples according to the loss gradient, the distribution of cluster centers is also relatively uniform. Real samples that are close to the cluster centers are representative of the overall real samples at this stage. However, as the dataset distillation process progresses, in the later stages, the loss gradients of most real samples are small, with only a few uncertain real samples having large loss gradients. After clustering multiple real samples according to the loss gradient, most real samples will be densely distributed in a small number of clusters, while real samples with large loss gradients will be clustered into scattered clusters. In this case, the principle of selecting real samples that are close to the cluster centers is followed, and real samples with large loss gradients are selected. Therefore, according to this scheme...

[0164] In some embodiments, such as Figure 4 As shown, step 230 includes the following steps 410-450:

[0165] Step 410: Obtain a reference classification model with the same architecture and parameters as the target classification model.

[0166] The target classification model can be replicated to obtain a reference classification model with the same architecture and parameters as the target classification model.

[0167] Step 420: Calculate the first loss gradient of the target sample set in the (i+1)th round on the target classification model.

[0168] The first loss gradient of the target sample set in the (i+1)th round on the target classification model reflects the overall change in the prediction loss of the target sample set in the (i+1)th round on the target classification model.

[0169] Specifically, the first loss gradient can be determined by following steps B1-B3:

[0170] Step B1: The target classification model classifies each real sample in the target sample set of round i+1 to obtain the second predicted category of each real sample in the target sample set of round i+1.

[0171] Step B2: Determine the second prediction loss for the target sample set in the (i+1)th round based on the labeled category and the second predicted category of each real sample in the target sample set in the (i+1)th round.

[0172] The second prediction loss for the target sample set in round i+1 reflects the difference between the second predicted category and the labeled category of the target classification model for the entire target sample set in round i+1.

[0173] In some embodiments, the labeled category and the second predicted category of each real sample in the target sample set of the (i+1)th round can be used to calculate the loss, so as to obtain the sub-prediction loss of each real sample in the target sample set of the (i+1)th round; then the sub-prediction losses of all real samples in the target sample set of the (i+1)th round are added together, and the sum is used as the second prediction loss.

[0174] The sub-prediction loss of a real sample reflects the difference between the labeled class and the second predicted class of that real sample. The sub-prediction loss of each real sample in the (i+1)th round target sample set can be obtained by calculating the loss between the labeled class and the second predicted class using a third loss function. Examples of third loss functions include absolute value loss, mean squared error loss, and cross-entropy loss, etc., which are not specifically limited here.

[0175] In other embodiments, the loss can be calculated using the labeled category and the second predicted category of each real sample in the (i+1)th round target sample set to obtain the sub-prediction loss of each real sample in the (i+1)th round target sample set; then, the sub-prediction losses of the real samples in the (i+1)th round target sample set are weighted and averaged to obtain the second prediction loss of the (i+1)th round target sample set. For example, the second prediction loss of the (i+1)th round target sample set can be obtained by averaging the sub-prediction losses of the real samples in the (i+1)th round target sample set.

[0176] Step B3: Calculate the gradient of the second prediction loss of the target sample set in the (i+1)th round relative to the target classification model, and obtain the first loss gradient of the target sample set in the (i+1)th round on the target classification model.

[0177] One approach is to take the partial derivative of the second prediction loss with respect to the model parameters of the target classification model, thus obtaining the first loss gradient of the target sample set in the (i+1)th round on the target classification model. In other words, the first loss gradient of the target sample set in the (i+1)th round on the target classification model refers to the gradient of the second prediction loss of the target sample set in the (i+1)th round relative to the model parameters of the target classification model.

[0178] Step 430: Calculate the second loss gradient of the i-th round of synthetic sample set on the reference classification model.

[0179] Similarly, the second loss gradient can be determined by following steps C1-C3:

[0180] Step C1: Classify each synthetic sample in the i-th round synthetic sample set using the reference classification model to obtain the third predicted category of each synthetic sample in the i-th round synthetic sample set.

[0181] Step C2: Determine the third prediction loss of the synthetic sample set in the i-th round based on the labeled category and the third prediction category of each synthetic sample in the i-th round synthetic sample set.

[0182] The third prediction loss of the i-th round of synthetic sample set reflects the difference between the third predicted class and the labeled class of the reference classification model for the i-th round of synthetic sample set as a whole.

[0183] In some embodiments, the labeled category and the third prediction category of each synthetic sample in the i-th round synthetic sample set can be used to calculate the loss, so as to obtain the sub-prediction loss of each synthetic sample in the i-th round synthetic sample set; then the sub-prediction losses of all synthetic samples in the i-th round synthetic sample set are added together, and the sum is used as the third prediction loss of the i-th round synthetic sample set.

[0184] The sub-prediction loss of each synthetic sample in the i-th round of synthetic sample set can be obtained by calculating the loss between the labeled category and the third predicted category of each synthetic sample in the i-th round of synthetic sample set. Examples of third loss functions include absolute value loss, mean squared error loss, and cross-entropy loss, etc., and no specific limitation is made here.

[0185] In other embodiments, the labeled category and the third predicted category of each synthetic sample in the i-th round of synthetic sample set can be used to calculate the loss, resulting in the sub-prediction loss of each synthetic sample in the i-th round of synthetic sample set. Then, the sub-prediction losses of the synthetic samples in the i-th round of synthetic sample set can be weighted and averaged to obtain the second prediction loss of the i-th round of synthetic sample set. For example, the second prediction loss of the i-th round of synthetic sample set can be obtained by averaging the sub-prediction losses of the synthetic samples in the i-th round of synthetic sample set.

[0186] Step C3: Calculate the gradient of the third prediction loss of the i-th round synthetic sample set relative to the reference classification model, and obtain the second loss gradient of the i-th round synthetic sample set on the reference classification model.

[0187] The second loss gradient of the i-th round synthetic sample set on the reference classification model can be obtained by taking the partial derivative of the third prediction loss with respect to the model parameters of the reference classification model. In other words, the second loss gradient of the i-th round synthetic sample set on the reference classification model refers to the gradient of the third prediction loss of the i-th round synthetic sample set relative to the model parameters of the reference classification model.

[0188] Step 440: Determine the difference in loss gradients between the first loss gradient and the second loss gradient.

[0189] In some embodiments, the Euclidean distance between the first loss gradient and the second loss gradient can be calculated as the difference in loss gradients between the first loss gradient and the second loss gradient.

[0190] In some embodiments, considering that the gradients (first loss gradient, second loss gradient) contain directional information, in step 440, the cosine similarity between the first loss gradient and the second loss gradient can be calculated; based on the cosine similarity, the loss gradient difference between the first loss gradient and the second loss gradient is determined, and the loss gradient difference is negatively correlated with the cosine similarity. For example, the difference between 1 and the cosine similarity can be used as the loss gradient difference between the first loss gradient and the second loss gradient.

[0191] Step 450: With the goal of minimizing the difference in loss gradient, iteratively update the synthetic samples in the synthetic sample set of the i-th round until the distillation cycle stops. If the distillation cycle stops, the updated synthetic sample set of the i-th round is used as the synthetic sample set of the (i+1)-th round.

[0192] In other words, by iteratively updating the synthetic samples in the i-th round of the synthetic sample set, the difference between the gradients of the updated i-th round of the synthetic sample set and the target sample set in the (i+1)-th round on a model with the same architecture and parameters is minimized. The iterative objective of iteratively updating the synthetic samples in the i-th round of the synthetic sample set in step 450 can be described by the following formula 4:

[0193]

[0194] S i+1 S represents the (i+1)th round of synthesized sample set; i Denotes the synthetic sample set of the i-th round; Let L(θ1, θ2) represent the target sample set for the (i+1)th round; θ1 represents the parameters of the reference classification model; θ2 represents the parameters of the target classification model; L(θ1, S) i ) represents the third prediction loss of the synthetic sample set in the i-th round; This represents the second prediction loss for the target sample set in the (i+1)th round; This represents the second loss gradient of the i-th round of synthetic sample set on the reference classification model; Denotes the target sample set in the (i+1)th round. The first loss gradient on the target classification model; This represents the difference in loss gradients between the first and second loss gradients;

[0195] Among them, the third prediction loss L(θ1, S) of the i-th round of synthetic sample set i It can be calculated according to the following formula 5:

[0196]

[0197] Where s represents the synthetic sample set S in the i-th round. i Synthetic samples in Let θ1(s) represent the labeled class of the synthetic sample s, and let θ1(s) represent the third predicted class of the synthetic sample s. |S represents the sub-prediction loss of the synthetic sample s; i | represents the synthetic sample set S in the i-th round. i The number of synthetic samples.

[0198] The second prediction loss for the target sample set in round i+1 It can be calculated using the following formula 6:

[0199]

[0200] Where x represents the target sample set in the (i+1)th round. In the context of the real sample x, y is the labeled class of the real sample x, θ2(x) represents the second predicted class of the real sample x, and l(θ2(x), y) represents the sub-prediction loss of the real sample x. Denotes the target sample set in the (i+1)th round. The number of real samples in the sample.

[0201] Figure 5 This is a flowchart illustrating a distillation process according to an embodiment of this application. Taking the i-th distillation process in each round of distillation as an example, as follows... Figure 5 As shown, it includes:

[0202] Step 510: Obtain the synthetic sample set of the i-th round before the t-th update, and use it as the sample set to be updated.

[0203] It is understandable that when t = 1, the synthetic sample set obtained in step 510 for the i-th round is the synthetic sample set determined by the i-th round of distillation. When t is greater than 1, the synthetic sample set obtained in step 510 is the synthetic sample set for the i-th round after the (t-1)-th update.

[0204] Step 520: Calculate the second loss gradient of the sample set to be updated on the reference classification model.

[0205] Step 530: Calculate the difference between the second loss gradient of the sample set to be updated on the reference classification model and the first loss gradient of the target sample set in the (i+1)th round on the target classification model.

[0206] Step 540: In the direction of reducing the difference in loss gradient, update the sample set to be updated at least t-th time.

[0207] In step 540, the parameters of the sample set to be updated and the reference classification model can be updated in the t-th iteration in the direction of reducing the difference in loss gradient.

[0208] Step 550: Determine if the distillation cycle stopping condition is met; if not, proceed to step 560, increment t by 1, and return to step 510. If the condition is met, proceed to step 570, and output the synthesized sample set for the (i+1)th round.

[0209] In some embodiments, the distillation cycle stopping condition may be that the difference in loss gradient is less than a difference threshold; the distillation cycle stopping condition may also be that the number of iterations is not less than an iteration number threshold. In some embodiments, when iterating and updating the reference classification model, the distillation cycle stopping condition may also be that the performance difference of the reference classification model on the test set does not exceed a difference threshold, and the number of iterations is not less than an iteration number threshold.

[0210] In some embodiments, such as Figure 6 As shown, prior to step 210, the method further includes:

[0211] Step 610: Obtain the initial synthetic sample set, which includes multiple synthetic samples and the annotation labels of each synthetic sample.

[0212] In some embodiments, random sampling can be performed based on the labels of each real sample in the real sample set to obtain M real samples. Assume the total number of different labels in the real sample set is X, M is greater than X, and the total number of different labels in the M real samples is equal to X. In some embodiments, the number of samples under each category can be set, for example, P samples under each label category. Then, P real samples are sampled from the real samples for each label category, for a total of M = P × X. Next, each label from the M real samples is input into the sample generation model. The label guides the sample generation model to generate samples, resulting in a synthetic sample output by the model. The label used to guide the generation of the synthetic sample is used as the label of the synthetic sample. The label of the synthetic sample and the synthetic sample are associated and added to a first set. This first set is used as the initial synthetic sample set.

[0213] In some embodiments, feature extraction can be performed on M real samples to obtain the sample features of each real sample in the M real samples; then, for each real sample in the M real samples, the label and sample features of the real sample are input into the sample generation model, and the sample generation model is guided to generate samples by the label and sample features of the real samples, and outputs a synthetic sample, so that the synthetic sample can retain the sample features of the corresponding real sample, inherit the label of the real sample, and the synthetic sample is different from the real sample.

[0214] The sample generation model is applicable to the same data modality as the real sample. For example, if the real sample modality is an image modality (i.e., the real sample is a real image), the sample generation model is an image generation model; if the real sample modality is a text modality (i.e., the real sample is real text), the sample generation model is a text generation model; if the real sample modality is an audio modality (i.e., the real sample is real audio), the sample generation model is an audio generation model.

[0215] Step 620: Train the first classification model by initializing the synthetic sample set to obtain the first second classification model.

[0216] Step 630: The first round of active learning is performed on the real sample set using the first second classification model to obtain the first round of target sample set.

[0217] The process of training the first classification model in step 620 is similar to that described above, and the process of the first round of active learning in step 630 is also similar to that described above, so it will not be repeated here. It is worth mentioning that in step 630, the real sample set is used as the real sample set for the first round.

[0218] Step 630: The target classification model performs the first round of distillation based on the first round target sample set and the initialized synthetic sample set to obtain the first round synthetic sample set.

[0219] The process of the first round of distillation is similar to that described above and will not be repeated here. After determining the first round of synthetic sample set, the first classification model can be trained using the first round of synthetic sample set to carry out the subsequent active learning-distillation cycle.

[0220] In other embodiments, random sampling can be performed based on the labels of each real sample in the real sample set to obtain M real samples. Assume that the total number of different labels involved in the real sample set is X, M is greater than X, and the total number of different labels involved in the M real samples is equal to X. The selected M real samples are used as the first round target sample set. Subsequently, the target classification model performs the first round distillation based on the first round target sample set and the initialized synthetic sample set to obtain the first round synthetic sample set. Based on this, the first classification model can be trained using the first round synthetic sample set to carry out the subsequent active learning-distillation-active learning-distillation cycle.

[0221] Figure 7 This is a schematic diagram illustrating a dataset distillation method according to an embodiment of this application, as shown below. Figure 7 As shown, the method in this application involves an active learning cycle and a distillation cycle. The active learning cycle iteratively selects a target sample set from the real sample set that is informative for learning from the synthetic data. Each round of data distillation involves one round of active learning and one round of distillation.

[0222] In the active loop, the first classification model is trained using the current synthetic sample set (i.e., the (i-1)th round synthetic sample set) to obtain the (i-1)th second classification model. This i-th second classification model reflects the distillation state after the i-th round of distillation. Based on this, the (i-1)th second classification model can be used to dynamically select the most valuable real image for the current distillation state to determine the target sample set for the i-th round (i.e., to perform the i-th round of active learning).

[0223] As described above, in the i-th round of active learning, the loss gradient of each real sample in the i-th round of real sample set on the (i-1)-th second classification model is calculated. To reduce computational overhead, the loss gradient of each real sample on the classifier of the (i-1)-th second classification model can be regarded as the loss gradient of the real sample on the (i-1)-th second classification model. Then, based on the loss gradient, the real samples in the i-th round of real sample set are clustered in the gradient space to obtain the clustering results. Based on the cluster centers in the clustering results, real samples with a loss gradient distance of less than a distance threshold from the cluster centers are selected from the i-th round of real sample set. The selected real samples are added to the i-th round of target sample set.

[0224] The pseudocode for implementing the i-th round of active learning is shown in Table 1 below:

[0225]

[0226] Table 1

[0227] In Table 1 above, C represents the K cluster centers determined by clustering based on the loss gradient.k This represents the k-th cluster center. Let x represent the j-th real sample. j With the kth cluster center c k The distance between them; This represents the selected real samples. This represents the selected real samples. The labeling category; represents the set of real samples selected during the i-th round of active learning; DC represents the real sample set.

[0228] The active learning in this application is performed after the distillation cycle converges to meet the evolution of the learning requirements of the synthetic sample set. Therefore, in this application, this process is referred to as stepwise information matching.

[0229] In this embodiment, the real samples selected in the i-th round of active learning are merged into the target sample set of the (i-1)-th round and distilled. Thus, the target sample set of the i-th round is represented as:

[0230]

[0231] In this embodiment, the target sample set for the i-th round includes the real samples selected in the i-th round of active learning and the target sample set for the (i-1)-th round. The real samples in Formula 7. In Formula 7, q represents... The real sample in, y q The labeled category is the real sample q.

[0232] Correspondingly, the selected real samples are removed from the real sample set to reduce the computational cost of the algorithm; therefore, the real sample set of the (i+1)th round, which forms the basis for the (i+1)th round of active learning, can be represented as:

[0233]

[0234] It is understandable that DC on the right side of the equals sign in Formula 8 represents the real sample set of the i-th round, which serves as the basis for the i-th round of active learning.

[0235] In this embodiment, since the real samples selected in the previous rounds of active learning are retained in the target sample set of the i-th round, and the real samples selected in the previous rounds are also retained in the target sample set of the i-th round and used for the i-th round of distillation, it can prevent the synthetic samples from overfitting to new samples in the subsequent distillation process, and avoid catastrophic forgetting of the knowledge already captured in the distillation process. This ensures the quality of the synthetic sample set of the i-th round determined by the subsequent distillation. The subsequent use of the synthetic sample set of the i-th round to continuously train the first classification model can also better reflect the information state contained in the synthetic sample set and provide a better sampling importance estimate than discarding the existing target set.

[0236] After determining the true sample set for the i-th round, a distillation cycle can be entered to perform the i-th round of distillation. In this embodiment, the objective is to minimize the difference in loss gradient between the first loss gradient of the target sample set in the i-th round on the target classification model and the second loss gradient of the synthetic sample set in the (i-1)-th round on the reference classification model. The synthetic samples in the synthetic sample set in the (i-1)-th round are iteratively updated until the distillation cycle stops, thus obtaining the synthetic sample set for the i-th round.

[0237] Wherein, the synthetic sample set S in the i-th round i It can be represented as:

[0238]

[0239] θ represents the parameters of the reference classification model and the parameters of the target classification model; L(θ, S) i-1 ) represents the (i-1)th round of synthesized sample set S i-1 The third predicted loss; This represents the second prediction loss for the target sample set in the i-th round; S represents the synthetic sample set in the (i-1)th round. i-1 The second loss gradient on the reference classification model; Denotes the target sample set in the i-th round. The first loss gradient on the target classification model; This represents the difference in loss gradients between the first and second loss gradients.

[0240] Considering that the loss gradient (first loss gradient and second loss gradient) contains directional information, the difference in loss gradients above can be determined based on the cosine similarity between the first loss gradient and the second loss gradient. Since the cosine similarity is negatively correlated with the difference in loss gradients, minimizing the difference in loss gradients between the first loss gradient and the second loss gradient is equivalent to maximizing the cosine similarity between the first loss gradient and the second loss gradient.

[0241] Among them, the synthetic sample set S in the (i-1)th round i-1 The third prediction loss L(θ, S) i-1 It can be calculated using the following formula 10:

[0242]

[0243] Where s represents the synthetic sample set S in the (i-1)th round. i-1 Synthetic samples in Let be the labeled class of the synthetic sample s, and θ(s) represent the third predicted class of the synthetic sample s. |S represents the sub-prediction loss of the synthetic sample s; i-1| represents the (i-1)th round of synthesized sample set S i-1 The number of synthetic samples.

[0244] Wherein, the second prediction loss of the target sample set in the i-th round It can be calculated using the following formula 11:

[0245]

[0246] Where x represents the target sample set in the i-th round. In the context of the real samples, y is the labeled class of the real sample x, θ(x) represents the second predicted class of the real sample x, and l(θ(x), y) represents the sub-prediction loss of the real sample x. Denotes the target sample set in the i-th round. The number of real samples in the sample.

[0247] In a single distillation cycle, the stopping condition for the distillation cycle can be: the performance difference of the reference classification model on the test set is less than a performance difference threshold, and the number of iterations is greater than or equal to the iteration number threshold N. min That is, the distillation cycle stopping condition is expressed as:

[0248] Δm(V, θ) S ,t)<∈)∧(t≥N min )

[0249] Where V represents the test set, and the samples in the test set are real samples, θ S For the parameters of the reference classification model, t is the number of iterations updated in the current synthetic sample set; Δm(V, θ) S ,t) represents the performance difference of the reference classification model on the test set; ∈ represents the performance difference threshold. When the distillation cycle stopping condition is met, we can infer that the synthetic sample set has converged to a stable state, indicating that the relevant knowledge has been absorbed.

[0250] After completing the i-th round of distillation, if i is less than N, the active learning process of distillation described above continues; if i is equal to N, the i-th round of synthetic sample set determined by the i-th round of distillation is used as the data distillation result of the real sample set.

[0251] The pseudocode for implementing the above active loop + distillation loop process is shown in Table 2 below:

[0252]

[0253]

[0254] Table 2

[0255] In this application, the principle of active learning is introduced into the dataset distillation process, and a real sample set constructed through the active learning method is maintained outside the distillation process. In each round of active learning, a first classification model with the same architecture as the target learning model is trained on the current synthetic sample set to obtain a second classification model, which is used to evaluate the current distillation state.

[0256] Based on this evaluation, the second classification model dynamically selects the most valuable real samples for the current distillation state from the real sample set and adds them to the target sample set. During the real sample selection process, the diversity and uncertainty of the real samples can be considered simultaneously. Specifically, in this scheme, the loss gradient of the real samples in the i-th round of the current active learning set on the second classification model is used. Based on the loss gradient, the real samples in the i-th round of the real sample set are clustered in the gradient space, and the real samples closest to the cluster centers are selected and added to the target sample set based on the clustering results.

[0257] In related technologies, the changing distillation requirements of real data during the data distillation process are not considered. It is mistakenly assumed that the importance of real samples remains constant throughout the distillation process. As a result, the diversity and information content of the selected real samples gradually decrease, leading to an imbalance in the information representation of the synthetic sample set. In this application, a second classification model is obtained by training a first classification model with the same architecture as the target learning model on the current synthetic sample set. This second classification model is used to evaluate the current distillation state and selects real samples from the real sample set that meet the requirements of subsequent distillations. This dynamically adjusts the target sample set used for distillation to meet the constantly changing distillation learning needs, thereby improving distillation efficiency and enhancing the quality of the synthetic sample set throughout the entire distillation process.

[0258] This scheme uses clustering results in gradient space to select real samples that are close to the cluster centers and add them to the target sample set. In the early stages, almost all real samples have large loss gradients, and the distribution of these gradients is relatively uniform. After clustering multiple real samples according to their loss gradients, the cluster centers are also relatively evenly distributed. The real samples selected that are close to the cluster centers are representative of the overall real samples at this stage. However, as the dataset distillation process progresses, in the later stages, most real samples have small loss gradients, with only a few uncertain real samples having large loss gradients. After clustering multiple real samples according to their loss gradients, most real samples will be densely distributed in a small number of clusters, while real samples with large loss gradients will be clustered into scattered clusters. For example, some real samples with large loss gradients may be located in the gradient space and thus become cluster centers, or be located near scattered cluster centers. In this case, the principle of selecting real samples close to the cluster centers is applied, and real samples with large loss gradients are selected. Therefore, according to this scheme, representative real samples can be selected in the early stage for subsequent distillation, and highly uncertain real samples can be focused on in the later stage. Real samples with high value for the current distillation state can be screened and added to the target sample set, which can effectively improve the subsequent distillation effect.

[0259] Figure 8 This is a schematic diagram illustrating real samples selected at different stages in the feature space and gradient space, according to an embodiment of this application. Figure 8 In the diagram, black dots represent the selected real samples, and gray dots represent the real samples in the real sample set. Figure 8 In Figure 'a', the distribution of real samples in the feature space of the overall real sample set in the early, middle, and late stages, as well as the distribution of the selected real samples in the feature space, are shown. Figure 8 As can be seen from 'a' in the diagram, the real samples selected in the early and middle stages are located in regions with relatively dense features in the feature space. This means the selected real samples have a high information content and are representative real samples. Therefore, according to the method of this application, representative real samples can be selected in the early stages for subsequent distillation. In the later stages, more of the selected real samples are located in regions with sparse features in the feature space. In other words, in the later stages, the method of this application will select highly uncertain real samples.

[0260] from Figure 8Figure b shows the distribution of real samples in the gradient space of the overall real sample set in the early, middle and late stages, as well as the distribution of the selected real samples in the gradient space. It can be seen that, regardless of whether it is the early, middle or late stage, the selected real samples in the gradient space are the real samples whose corresponding loss gradients are close to the cluster centers.

[0261] In addition, Figure 7 The scheme in the embodiments is tested on multiple real image sets to evaluate the performance of the dataset distillation method provided in this application. The selected real image sets have various resolutions and include: MNIST (image resolution 28×28), Fashion-MNIST (image resolution 28×28), SVHN dataset (image resolution 32×32, where the real images are RGB house number images), CIFAR10 (images distributed across 10 classes), CIFAR100 (real images distributed across 100 classes), and a subset of ImageNet (image resolution 128×128).

[0262] Following classic dataset distillation settings, the compression ratio is measured using IPC (Images per class), and the number of synthetic images per class in the synthetic sample set is limited, typically set to 1, 10, and 50. Five randomly initialized image classification models are trained on each synthetic sample set for 1000 iterations, and the average accuracy and standard deviation of the image classification models after training are recorded. All experiments are performed on a single V100 GPU.

[0263] For the active learning setup in the experiment, a total of 10 rounds of active learning were performed, with 4% of new data (i.e., real images) added in each round, and finally 40% of the original training set (real image set) was used for distillation. Each round of distillation consisted of 500 distillation iterations.

[0264] Table 3 below shows the average accuracy of the image classification model after training the model on the synthetic sample set determined by distillation of the three real datasets SVHN, CIFAR10, and CIFAR100, using the method provided in this application and other dataset distillation methods in related technologies. In Table 3, the SVHN, CIFAR10, and CIFAR100 datasets were distilled according to three requirements: IPC values ​​of 1, 10, and 50.

[0265]

[0266] Table 3

[0267] Table 4 below shows the average accuracy of the image classification model after training the image classification model on the synthetic sample set determined by distillation on the MNIST dataset and the Fashion-MNIST dataset, using the method provided in this application and other dataset distillation methods in related technologies. In Table 4, the MNIST dataset and the Fashion-MNIST dataset were distilled according to three requirements: IPC values ​​of 1, 10, and 50.

[0268]

[0269] Table 4

[0270] As shown in Tables 3 and 4 above, the overall matching method and methods 1-19 are all existing methods in related technologies. The overall matching method refers to using the entire provided real sample set for multiple rounds of distillation matching to determine the synthetic sample set. As can be seen from Tables 3 and 4 above, after training the image classification model using the synthetic sample set obtained by distillation using the method provided in this application, the average accuracy of the image classification model is generally higher than that of methods 1-19. It can be seen that the method provided in this application can effectively guarantee the validity of the synthetic sample set determined by distillation.

[0271] Because the method provided in this application incorporates active learning, it selects a portion of the real samples that meet the current distillation requirements from the real sample set before each distillation round, instead of using all real samples in the real sample set for each distillation round. This method can significantly reduce computational overhead and improve distillation efficiency. Experiments on multiple real datasets show that the method in this application can achieve good performance even when using only 20-40% of the actual data volume for distillation.

[0272] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described in the above embodiments of this application.

[0273] Figure 9 This is a block diagram of a dataset distillation apparatus according to an embodiment of this application, such as... Figure 9As shown, the dataset distillation apparatus includes: a training module 910, used to train a first classification model using the i-th round of synthetic sample set determined in the i-th round of distillation, to obtain the i-th second classification model; wherein the architecture of the first classification model is the same as that of the target classification model used in distillation learning; i is a positive integer; an active learning module 920, used to perform the (i+1)-th round of active learning on the real sample set using the i-th second classification model, to determine the (i+1)-th round target sample set; the (i+1)-th round target sample set includes real samples selected from the real sample set in the (i+1)-th round of active learning; and a distillation module 930, used to train the target classification model using the root... Distillation is performed on the (i+1)th round target sample set and the i-th round synthetic sample set to obtain the (i+1)th round synthetic sample set. The (i+1)th round synthetic sample set is obtained by updating the synthetic samples in the i-th round synthetic sample set. The loop processing module 940 is used to increment i by 1 if i+1 is less than N, and return to the step of training the first classification model with the i-th round synthetic sample set determined by the i-th round distillation to obtain the i-th second classification model. N is an integer greater than 1. The data distillation result determination module 950 is used to take the (i+1)th round synthetic sample set as the data distillation result of the real sample set if i+1 is equal to N.

[0274] In some embodiments, the active learning module 920 includes: a determining unit, configured to determine a real sample set for the (i+1)th round in the real sample set; the real sample set for the (i+1)th round includes all or part of the real samples in the real sample set; a first calculation unit, configured to calculate the loss gradient of each real sample in the (i+1)th round on the i-th second classification model; a clustering unit, configured to cluster the real samples in the (i+1)th round based on the loss gradient to obtain a clustering result; a filtering unit, configured to filter out real samples in the (i+1)th round from which the loss gradient distance between the real samples and the cluster centers is less than a distance threshold; and an adding unit, configured to add the filtered real samples to the target sample set for the (i+1)th round.

[0275] In some embodiments, the first calculation unit includes: a first classification unit, configured to classify each real sample in the (i+1)th round of the real sample set using the i-th second classification model to obtain a first predicted category for each real sample in the (i+1)th round of the real sample set; a first prediction loss determination unit, configured to calculate the first prediction loss for each real sample in the (i+1)th round of the real sample set based on the labeled category and the first predicted category; and a loss gradient determination unit, configured to calculate the gradient of the first prediction loss of each real sample in the (i+1)th round of the real sample set relative to the i-th second classification model to obtain the loss gradient of each real sample in the (i+1)th round of the real sample set on the i-th second classification model.

[0276] In some embodiments, the i-th second classification model includes a feature extraction network and a classifier; the loss gradient of the real sample on the i-th second classification model includes the loss gradient of the real sample on the classifier; the loss gradient determination unit is further configured to: calculate the gradient of the first predicted loss of each real sample in the i+1-th round of the real sample set relative to the classifier, thereby obtaining the loss gradient of each real sample in the i+1-th round of the real sample set on the classifier.

[0277] In some embodiments, the determining unit is further configured to: take the difference between the real sample set and the target sample set of the i-th round as the real sample set of the (i+1)-th round.

[0278] In some embodiments, the target sample set for the (i+1)th round also includes real samples from the target sample set for the i-th round.

[0279] In some embodiments, the distillation module 930 includes: an acquisition unit for acquiring a reference classification model with the same architecture and model parameters as the target classification model; a first loss gradient calculation unit for calculating the first loss gradient of the target sample set in the (i+1)th round on the target classification model; a second loss gradient calculation unit for calculating the second loss gradient of the synthetic sample set in the i-th round on the reference classification model; a loss gradient difference determination unit for determining the loss gradient difference between the first loss gradient and the second loss gradient; and an iteration unit for iteratively updating the synthetic samples in the synthetic sample set in the i-th round with the goal of minimizing the loss gradient difference until the distillation cycle stops; and, if the distillation cycle stops, using the updated synthetic sample set in the i-th round as the synthetic sample set in the (i+1)th round.

[0280] In some embodiments, the first loss gradient calculation unit includes: a second classification unit, configured to classify each real sample in the target sample set of the (i+1)th round by the target classification model to obtain a second predicted category for each real sample in the target sample set of the (i+1)th round; a second prediction loss determination unit, configured to determine a second prediction loss for the target sample set of the (i+1)th round based on the labeled category and the second predicted category of each real sample in the target sample set of the (i+1)th round; and a second calculation unit, configured to calculate the gradient of the second prediction loss of the target sample set of the (i+1)th round relative to the target classification model to obtain a first loss gradient of the target sample set of the (i+1)th round on the target classification model.

[0281] In some embodiments, the second prediction loss determination unit includes: a sub-prediction loss determination unit, configured to calculate the loss of each real sample in the target sample set of the (i+1)th round by performing loss calculation on the labeled category and the second prediction category, to obtain the sub-prediction loss of each real sample in the target sample set of the (i+1)th round; and a weighted averaging unit, configured to perform weighted averaging on the sub-prediction losses of the real samples in the target sample set of the (i+1)th round, to obtain the second prediction loss of the target sample set of the (i+1)th round.

[0282] In some embodiments, the loss gradient difference determination unit is further configured to: calculate the cosine similarity between the first loss gradient and the second loss gradient; and determine the loss gradient difference between the first loss gradient and the second loss gradient based on the cosine similarity, wherein the loss gradient difference is negatively correlated with the cosine similarity.

[0283] In some embodiments, the dataset distillation apparatus further includes: an initial synthetic sample set acquisition module, configured to acquire an initial synthetic sample set, the initial synthetic sample set including multiple synthetic samples and labels of each synthetic sample; a first training module, configured to train a first classification model using the initial synthetic sample set to obtain a first second classification model; a first active learning module, configured to perform a first round of active learning on a real sample set using the first second classification model to obtain a first round target sample set; and a first distillation module, configured to perform a first round of distillation by the target classification model based on the first round target sample set and the initial synthetic sample set to obtain a first round synthetic sample set.

[0284] In some embodiments, real samples are real images, and synthetic samples are synthetic images.

[0285] Figure 10 A schematic diagram of an electronic device suitable for implementing embodiments of this application is shown. This electronic device is used to implement the dataset distillation method provided in this application. It should be noted that... Figure 10 The computer system 1300 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0286] like Figure 10As shown, the computer system 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in read-only memory (ROM) 1302 or programs loaded from storage portion 1308 into random access memory (RAM) 1303. The RAM 1303 also stores various programs and data required for system operation. The CPU 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0287] The following components are connected to I / O interface 1305: an input section 1306 including a keyboard, mouse, microphone, etc.; an output section 1307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 13013 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 13013 performs communication processing via a network such as the Internet. Drive 1310 is also connected to I / O interface 1305 as needed. Removable media 1311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1310 as needed so that computer instructions read from it can be loaded into storage section 1308 as needed.

[0288] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising computer instructions. When these computer instructions are executed by the central processing unit (CPU) 1301, the methods described in any of the above method embodiments are implemented.

[0289] This application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the methods described in any of the above method embodiments.

[0290] It should be noted that the computer-readable storage medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0291] In the embodiments of this application, the terms "module" or "unit" refer to computer instructions or a portion of computer instructions that have a predetermined function and work together with other related parts to achieve a predetermined goal. These instructions can be implemented, wholly or partially, using software, hardware (e.g., processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that functions as a whole.

[0292] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed preferred embodiments as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A dataset distillation method, characterized in that, include: The first classification model is trained using the synthetic sample set determined in the i-th round of distillation to obtain the i-th second classification model; wherein the architecture of the first classification model is the same as that of the target classification model used in distillation learning; i is a positive integer; The i-th second classification model performs the (i+1)-th round of active learning on the real sample set to determine the (i+1)-th round target sample set; the (i+1)-th round target sample set includes real samples selected from the real sample set in the (i+1)-th round of active learning. The target classification model performs distillation in the (i+1)th round based on the target sample set in the (i+1)th round and the synthetic sample set in the (i)th round to obtain the synthetic sample set in the (i+1)th round. The synthetic sample set in the (i+1)th round is obtained by updating the synthetic samples in the synthetic sample set in the (i)th round. If i+1 is less than N, increment i by 1 and return to the step of training the first classification model with the synthetic sample set determined in the i-th round of distillation to obtain the i-th second classification model; N is an integer greater than 1. If i+1 equals N, the synthesized sample set of the (i+1)th round is used as the data distillation result of the real sample set.

2. The method according to claim 1, characterized in that, The step of determining the target sample set for the (i+1)th round by performing active learning on the real sample set using the i-th second classification model includes: Determine the (i+1)th round real sample set from the real sample set; the (i+1)th round real sample set includes all or part of the real samples in the real sample set. Calculate the loss gradient of each real sample in the (i+1)th round of the real sample set on the i-th second classification model; Based on the loss gradient, the real samples in the (i+1)th round of real sample set are clustered to obtain the clustering result; Based on the cluster centers in the clustering results, select real samples from the (i+1)th round of real sample set whose loss gradient distance from the cluster centers is less than a distance threshold; The selected real samples are added to the target sample set of the (i+1)th round.

3. The method according to claim 2, characterized in that, The calculation of the loss gradient of each real sample in the (i+1)th round of the real sample set on the i-th second classification model includes: The i-th second classification model is used to classify each real sample in the (i+1)-th round of real sample set to obtain the first predicted category of each real sample in the (i+1)-th round of real sample set; Based on the labeled category and the first predicted category of each real sample in the (i+1)th round real sample set, calculate the first prediction loss of each real sample in the (i+1)th round real sample set; Calculate the gradient of the first prediction loss of each real sample in the (i+1)th round of the real sample set relative to the i-th second classification model, and obtain the loss gradient of each real sample in the (i+1)th round of the real sample set on the i-th second classification model.

4. The method according to claim 3, characterized in that, The i-th second classification model includes a feature extraction network and a classifier; the loss gradient of the real sample on the i-th second classification model includes the loss gradient of the real sample on the classifier; The step of calculating the gradient of the first predicted loss of each real sample in the (i+1)th round of the real sample set relative to the i-th second classification model, to obtain the loss gradient of each real sample in the (i+1)th round of the real sample set on the i-th second classification model, includes: Calculate the gradient of the first prediction loss of each real sample in the (i+1)th round real sample set relative to the classifier, and obtain the loss gradient of each real sample in the (i+1)th round real sample set on the classifier.

5. The method according to claim 2, characterized in that, Determining the (i+1)th round of real sample set in the real sample set includes: The difference between the real sample set and the target sample set of the i-th round is taken as the real sample set of the (i+1)-th round.

6. The method according to claim 5, characterized in that, The target sample set for the (i+1)th round also includes real samples from the target sample set for the i-th round.

7. The method according to any one of claims 1 to 6, characterized in that, The process of distilling the target sample set in the (i+1)th round based on the target sample set in the (i+1)th round and the synthetic sample set in the (i)th round to obtain the synthetic sample set in the (i+1)th round includes: Obtain a reference classification model with the same architecture and parameters as the target classification model; Calculate the first loss gradient of the target sample set in the (i+1)th round on the target classification model; Calculate the second loss gradient of the i-th round of synthetic sample set on the reference classification model; Determine the difference in loss gradients between the first loss gradient and the second loss gradient; With the goal of minimizing the difference in loss gradient, the synthetic samples in the synthetic sample set of the i-th round are iteratively updated at least until the distillation cycle stops. If the distillation cycle stops, the updated synthetic sample set of the i-th round is used as the synthetic sample set of the (i+1)-th round.

8. The method according to claim 7, characterized in that, The calculation of the first loss gradient of the target sample set in the (i+1)th round on the target classification model includes: The target classification model is used to classify each real sample in the target sample set of the (i+1)th round to obtain the second predicted category of each real sample in the target sample set of the (i+1)th round. The second prediction loss of the target sample set in the (i+1)th round is determined based on the labeled category and the second predicted category of each real sample in the target sample set in the (i+1)th round. Calculate the gradient of the second prediction loss of the target sample set in the (i+1)th round relative to the target classification model to obtain the first loss gradient of the target sample set in the (i+1)th round on the target classification model.

9. The method according to claim 8, characterized in that, The step of determining the second prediction loss of the target sample set in the (i+1)th round based on the labeled category and the second predicted category of each real sample in the target sample set includes: The loss is calculated by performing loss calculation on the labeled category and the second predicted category of each real sample in the target sample set of the (i+1)th round, to obtain the sub-prediction loss of each real sample in the target sample set of the (i+1)th round. The second prediction loss of the target sample set in the (i+1)th round is obtained by weighted averaging of the sub-prediction losses of the real samples in the target sample set in the (i+1)th round.

10. The method according to claim 7, characterized in that, Determining the difference in loss gradients between the first loss gradient and the second loss gradient includes: Calculate the cosine similarity between the first loss gradient and the second loss gradient; Based on the cosine similarity, the difference in loss gradient between the first loss gradient and the second loss gradient is determined, and the difference in loss gradient is negatively correlated with the cosine similarity.

11. The method according to any one of claims 1 to 6, characterized in that, Before training the first classification model using the i-th round of synthetic sample set determined in the i-th round of distillation to obtain the i-th second classification model, the method further includes: Obtain an initial synthetic sample set, which includes multiple synthetic samples and the annotation labels of each synthetic sample; The first classification model is trained using the initial synthetic sample set to obtain the first second classification model; The first second classification model is used to perform the first round of active learning on the real sample set to obtain the first round of target sample set; The target classification model performs a first round of distillation based on the first round target sample set and the initial synthetic sample set to obtain the first round synthetic sample set.

12. The method according to any one of claims 1 to 6, characterized in that, The real sample is a real image, and the synthetic sample is a synthetic image.

13. A dataset distillation apparatus, characterized in that, include: The training module is used to train the first classification model using the synthetic sample set determined in the i-th round of distillation, to obtain the i-th second classification model; wherein the architecture of the first classification model is the same as that of the target classification model used in distillation learning; i is a positive integer; An active learning module is used to perform the (i+1)th round of active learning on the real sample set through the i-th second classification model to determine the (i+1)th round target sample set; the (i+1)th round target sample set includes real samples selected from the real sample set in the (i+1)th round of active learning. The distillation module is used to perform distillation in the (i+1)th round by the target classification model based on the (i+1)th round target sample set and the (i)th round synthetic sample set to obtain the (i+1)th round synthetic sample set, which is obtained by updating the synthetic samples in the (i)th round synthetic sample set. The loop processing module is used to increment i by 1 if i+1 is less than N, and then return to the step of training the first classification model with the first round of synthetic sample set determined by the i-th round of distillation to obtain the i-th second classification model; N is an integer greater than 1. The data distillation result determination module is used to take the (i+1)th round of synthesized sample set as the data distillation result of the real sample set if i+1 equals N.

14. An electronic device, characterized in that, include: processor; A memory storing computer instructions that, when executed by the processor, implement the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method described in any one of claims 1 to 12.

16. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the method described in any one of claims 1 to 12.

Images