Data set distillation methods and apparatus based on cross-model generalization and diffusion models

Abstract

This disclosure relates to a dataset distillation method and apparatus based on cross-model generalization and diffusion models. The method includes: acquiring multiple image processing models corresponding to the application scenarios of the target dataset to obtain a model pool; selecting alternative image processing models from the model pool; inputting the features of the current sample and prompts for the current sample into the diffusion model to obtain alternative samples; inputting the alternative samples into the alternative image processing model to obtain the current output of the alternative image processing model; wherein the target image processing model and the alternative image processing model are isomorphic; calculating the gradient matching loss between the target image processing model and the alternative image processing model based on the current output of the alternative image processing model and the current output of the target image processing model; training the diffusion model based on the gradient matching loss; and obtaining the alternative dataset after distillation of the target dataset in response to the completion of training.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically to a dataset distillation method, apparatus, electronic device, and storage medium based on cross-model generalization and diffusion models. Background Technology

[0002] Dataset distillation is a dataset condensation technique that aims to condense a target dataset T with N samples into an alternative dataset S with M samples (M is much smaller than N, and the M samples are generally new samples different from the N samples). The alternative dataset S can be used for deep learning model training, and its training cost is much lower than that of the target dataset T. Furthermore, the performance of a model trained on S is basically equivalent to that of a model trained on T.

[0003] However, the dataset distillation methods in related technologies cannot simultaneously achieve the excellent performance of dataset distillation and the good generalization ability of different structural models. Summary of the Invention

[0004] The dataset distillation method, apparatus, electronic device, and storage medium based on cross-model generalization and diffusion models provided in this exemplary disclosure can at least solve the above-mentioned technical problems and other technical problems not mentioned above.

[0005] According to one aspect of this disclosure, a dataset distillation method based on cross-model generalization and diffusion models is provided. The method includes: acquiring a target dataset, wherein the target dataset is an image dataset; acquiring multiple image processing models corresponding to application scenarios of the target dataset to obtain a model pool; for each sample from a plurality of samples selected from the target dataset according to a first preset rule, performing the following operations: selecting an alternative image processing model from the model pool; inputting the current sample into the target image processing model to obtain the current output of the target image processing model, wherein the target image processing model is a pre-trained model isomorphic to the alternative image processing model, and the current output of the target image processing model includes the... The features of the current sample are used as inputs to a diffusion model along with a prompt word for the current sample to obtain a substitute sample, wherein the prompt word is used to instruct the diffusion model to generate the substitute sample based on the features of the current sample; the substitute sample is input into a substitute image processing model to obtain the current output of the substitute image processing model; based on the current output of the substitute image processing model and the current output of the target image processing model, the gradient matching loss between the target image processing model and the substitute image processing model is calculated; the diffusion model is trained based on the gradient matching loss; in response to the completion of training, the set of all obtained substitute samples is used as the substitute dataset after distillation of the target dataset.

[0006] Optionally, the method further includes: filtering the models in the model pool based on a second preset rule to obtain a first model pool, wherein the second preset rule is that, for multiple image processing models with the same basic architecture, if the difference in the number of network layers is less than a first preset threshold, the image processing model with the fewest network layers is selected.

[0007] Optionally, the method further includes: for each image processing model in the first model pool, embedding and mapping the sub-networks in each image processing model, and calculating the pairwise similarity between each image processing model, wherein the pairwise similarity between each image processing model is obtained by calculating the similarity between sub-networks of a preset number of layers between two image processing models, wherein the preset number of layers is the number of network layers of the image processing model with fewer network layers between the two image processing models; selecting the image processing model with the fewest network layers from each group of image processing models in the first model pool to obtain a second model pool, wherein each group of image processing models contains at least two image processing models with a similarity greater than a second preset threshold.

[0008] Optionally, the alternative image processing model is a plurality of image processing models selected from the second model pool, and the target image processing model is a plurality of pre-trained image processing models isomorphic to the alternative image processing model; wherein, calculating the gradient matching loss between the target image processing model and the alternative image processing model includes: for each of the plurality of isomorphic target image processing models and alternative image processing models composed of a plurality of target image processing models and a plurality of alternative image processing models, calculating the gradient matching loss between the plurality of isomorphic target image processing models and alternative image processing models; and calculating the total gradient matching loss based on the gradient matching loss between the plurality of isomorphic target image processing models and alternative image processing models.

[0009] Optionally, in the first stage of training the diffusion model, the alternative image processing model is a plurality of image processing models randomly selected from the second model pool; in the second stage of training the diffusion model, based on the magnitude of the gradient matching loss corresponding to each image processing model in the second model pool, the selection probability corresponding to each image processing model in the second model pool is determined, and based on the selection probability corresponding to each image processing model in the second model pool, a plurality of image processing models are selected from the second model pool as the alternative image processing models.

[0010] Optionally, the step of calculating the total gradient matching loss based on the gradient matching loss between multiple sets of isomorphic target image processing models and alternative image processing models includes: adding the gradient matching losses between multiple sets of isomorphic target image processing models and alternative image processing models to obtain the total gradient matching loss.

[0011] Optionally, the step of calculating the total gradient matching loss based on the gradient matching loss between multiple sets of isomorphic target image processing models and alternative image processing models includes: weighted summation of the gradient matching losses between multiple sets of isomorphic target image processing models and alternative image processing models to obtain the total gradient matching loss.

[0012] According to another aspect of this disclosure, a dataset distillation apparatus based on cross-model generalization and diffusion models is also provided. The apparatus includes: a target dataset acquisition unit configured to acquire a target dataset, wherein the target dataset is an image dataset; a model pool generation unit configured to acquire multiple image processing models corresponding to the application scenarios of the target dataset, thereby obtaining a model pool; and a data distillation unit configured to perform the following operations for each sample from a plurality of samples selected from the target dataset according to a first preset rule: selecting an alternative image processing model from the model pool; inputting the current sample into the target image processing model to obtain the current output of the target image processing model, wherein the target image processing model is a pre-trained model isomorphic to the alternative image processing model. The current output of the image processing model includes the features of the current sample; the features of the current sample and a cue word for the current sample are input into the diffusion model to obtain a substitute sample, wherein the cue word is used to instruct the diffusion model to generate the substitute sample based on the features of the current sample; the substitute sample is input into the substitute image processing model to obtain the current output of the substitute image processing model; based on the current output of the substitute image processing model and the current output of the target image processing model, the gradient matching loss between the target image processing model and the substitute image processing model is calculated; the diffusion model is trained based on the gradient matching loss; the data generation unit is configured to: in response to training completion, use the set of all obtained substitute samples as a substitute dataset after distillation of the target dataset.

[0013] Optionally, the apparatus further includes: a first model pool filtering unit, configured to: filter models in the model pool based on a second preset rule to obtain a first model pool, wherein the second preset rule is that, for multiple image processing models with the same basic architecture, if the difference in the number of network layers is less than a first preset threshold, the image processing model with the fewest network layers is selected; wherein the data distillation unit is configured to: select the alternative image processing model from the first model pool.

[0014] Optionally, the apparatus further includes: a second model pool screening unit, configured to: for each image processing model in the first model pool, embed and map the sub-networks in each image processing model respectively, and calculate the pairwise similarity between each image processing model, wherein the pairwise similarity between each image processing model is obtained by calculating the similarity between sub-networks of a preset number of layers between two image processing models, wherein the preset number of layers is the number of network layers of the image processing model with fewer network layers between the two image processing models; select the image processing model with the fewest network layers from each group of image processing models in the first model pool to obtain a second model pool, wherein each group of image processing models contains at least two image processing models with similarity greater than a second preset threshold; wherein the data distillation unit is configured to: select the alternative image processing model from the second model pool.

[0015] Optionally, the alternative image processing model is a plurality of image processing models selected from the second model pool, and the target image processing model is a plurality of pre-trained image processing models isomorphic to the alternative image processing model; wherein, the data distillation unit is configured to: calculate the gradient matching loss between the target image processing model and the alternative image processing model in each of the plurality of isomorphic target image processing models and alternative image processing models composed of a plurality of target image processing models and a plurality of alternative image processing models; and calculate the total gradient matching loss based on the gradient matching loss between the plurality of isomorphic target image processing models and alternative image processing models.

[0016] Optionally, the apparatus further includes: a model selection unit, configured to: in a first stage of training the diffusion model, the alternative image processing model is a plurality of image processing models randomly selected from the second model pool; in a second stage of training the diffusion model, based on the magnitude of the gradient matching loss corresponding to each image processing model in the second model pool, determine the selection probability corresponding to each image processing model in the second model pool, and select a plurality of image processing models from the second model pool as the alternative image processing models based on the selection probability corresponding to each image processing model in the second model pool.

[0017] Optionally, the data distillation unit is configured to add the gradient matching losses between multiple sets of isomorphic target image processing models and alternative image processing models to obtain the total gradient matching loss.

[0018] Optionally, the data distillation unit is configured to: perform a weighted summation of the gradient matching losses between multiple sets of isomorphic target image processing models and alternative image processing models to obtain the total gradient matching loss.

[0019] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; at least one memory storing computer-executable instructions, wherein, when executed by the at least one processor, the computer-executable instructions cause the at least one processor to perform the dataset distillation method based on cross-model generalization and diffusion models as described above.

[0020] According to another aspect of the embodiments of this disclosure, a computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the dataset distillation method based on cross-model generalization and diffusion models as described above.

[0021] According to another aspect of the embodiments of this disclosure, a system is also provided that includes at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the dataset distillation method based on cross-model generalization and diffusion models as described above.

[0022] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the dataset distillation method based on cross-model generalization and diffusion models as described in any of the above embodiments.

[0023] The technical solutions provided in this disclosure offer at least the following beneficial effects:

[0024] According to the dataset distillation method, apparatus, electronic device and storage medium based on cross-model generalization and diffusion model disclosed herein, by combining gradient matching method and diffusion model, an alternative dataset applicable to a large number of network structures can be obtained through a single training, and the model trained on the alternative dataset can achieve excellent performance. The method has significantly improved versatility, greatly reduced training cost and greatly improved model training performance, and can be deployed and promoted in practical applications.

[0025] In addition, by introducing multi-expert constraints, good generalization ability can be achieved for different models. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0027] Figure 1 A schematic flowchart illustrating a dataset distillation method based on cross-model generalization and diffusion models in an exemplary embodiment of this disclosure is shown.

[0028] Figure 2 A flowchart illustrating a diffusion model dataset distillation method based on multi-expert fusion optimization in an exemplary embodiment of this disclosure is shown.

[0029] Figure 3 A block diagram of a dataset distillation apparatus based on cross-model generalization and diffusion models is shown in an exemplary embodiment of this disclosure;

[0030] Figure 4 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0032] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0033] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0034] Dataset distillation techniques in related technologies can be divided into two types.

[0035] The first type is the gradient matching-based dataset distillation method. This type of method is generally designed for a specific network structure and performs gradient matching on the target dataset and the alternative dataset during the optimization process. Backpropagation is performed using gradient-based loss (e.g., MSE, Mean Squared Error Gradient Loss, mean squared error gradient loss, etc.) to optimize and obtain a suitable alternative dataset. The advantage of this type of method is that it has a good dataset distillation effect for a specific network (the performance of the model trained on the alternative dataset is not much different from that of the model trained on the target dataset). However, its disadvantages are also prominent: (1) Dataset distillation needs to be performed again for new network models, that is, each network structure needs a corresponding alternative dataset; (2) Alternative datasets for different network structures cannot be used interchangeably, that is, if the alternative dataset of network structure A is used to train network structure B, it will cause the performance of network structure B model to degrade severely, that is, poor generalization ability.

[0036] The second type is the dataset distillation method based on the diffusion model. This type of method uses visual common sense obtained by training the diffusion model on a large-scale dataset to directly generate alternative datasets using "textualized images". For example, it directly inputs prompts related to "airplane" to generate airplane-type images, and then combines some common data augmentation methods (random cropping, random flipping, etc.) to improve the diversity of the dataset. The advantages of this type of method are low training cost and no dependence on a specific network structure; the disadvantages are (1) because it is not optimized for a specific network, the training performance of the alternative dataset is poor, and there is a large performance gap with the first method; (2) its performance makes it impossible to promote in practical applications and it is still in the laboratory stage.

[0037] To address the aforementioned issues, this disclosure provides a dataset distillation method, apparatus, electronic device, and storage medium based on cross-model generalization and diffusion models. By combining gradient matching methods and diffusion models, alternative datasets applicable to a wide range of network structures can be obtained through a single training iteration. Furthermore, models trained on these alternative datasets achieve superior performance. This method significantly enhances versatility, drastically reduces training costs, and significantly improves model training performance, enabling its deployment and widespread adoption in practical applications.

[0038] Below, we will refer to Figures 1 to 4 This disclosure provides a detailed description of a dataset distillation method, apparatus, electronic device, and storage medium based on cross-model generalization and diffusion models.

[0039] Figure 1 This illustration shows a flowchart of a dataset distillation method based on cross-model generalization and diffusion models in an exemplary embodiment of this disclosure.

[0040] Reference Figure 1 In step 101, the target dataset is obtained, wherein the target dataset is an image dataset.

[0041] Specifically, the target dataset is the dataset to be distilled. It should be noted that in some other embodiments, the target dataset may also be a text dataset, but this disclosure uses the distillation of an image dataset as an example for illustration.

[0042] In step 102, multiple image processing models corresponding to the application scenarios of the target dataset are obtained to form a model pool. Specifically, the model pool can be obtained by iterating through the scenarios that the corresponding image processing models of the target dataset might be used in. For example, when the target dataset is a face recognition dataset, there may be several server-side models and several lightweight client-side models. The union of these model structures can then be the initial model pool. It is understood that the target dataset is not limited to the listed face recognition datasets, but can also be datasets used in any other scenario, such as object detection datasets used in object detection scenarios.

[0043] In real-world AI applications, various network architectures are required to adapt to different environments. For example, cloud servers can use complex, high-performance models, while IoT (Internet of Things) devices, limited by computing power, can only use lightweight models. To achieve good generalization capabilities across different models, representative model architectures can be selected for application.

[0044] According to an exemplary embodiment of this disclosure, models in the model pool can be filtered based on a second preset rule to obtain a first model pool. The second preset rule can be that, for multiple image processing models with the same basic architecture, if the difference in the number of network layers is less than a first preset threshold, the image processing model with the fewest network layers is selected. Alternative image processing models can be selected from the first model pool.

[0045] Specifically, since the initial model pool may contain a large number of models with different structures (e.g., ResNet18, ResNet34, ResNet50, MobileNetV2, etc., but not limited to these), a preliminary screening of the model pool can be performed based on the second preset rule. The first preset threshold can be 50%, but is not limited to this. Therefore, for similar model structures, if the difference in the number of network layers between two models is less than 50%, then considering the computational cost, only the one with the smaller number can be retained; for example, between ResNet18 and ResNet34, only ResNet18 can be retained; however, specific examples are not limited to this. Based on the above rules, the preliminary model pool, i.e., the first model pool, can be obtained.

[0046] According to an exemplary embodiment of this disclosure, for each image processing model in the first model pool, the sub-networks in each image processing model can be embedded and mapped separately, and the pairwise similarity between each image processing model can be calculated. The pairwise similarity between each image processing model can be obtained by calculating the similarity between sub-networks of a preset number of layers between two image processing models. The preset number of layers can be the number of network layers of the image processing model with fewer network layers between the two image processing models. The image processing model with the fewest network layers can be selected from each group of image processing models in the first model pool to obtain a second model pool. Each group of image processing models contains at least two image processing models with a similarity greater than a second preset threshold. An alternative image processing model can be selected from the second model pool.

[0047] Specifically, the first model pool can be further refined. This can be done using NAS (Neural Architecture Search) related network similarity calculation methods. For example, after embedding different operations (i.e., sub-networks) within the model, the overall similarity between each pair of models (networks) can be calculated. During the similarity calculation, only the similarity between the first N layers (i.e., a preset number of layers) of two networks can be calculated, where N can be the number of layers in the shallower network. For a pair of models with a similarity higher than a second preset threshold (e.g., the second preset threshold can be set to 0.75), only the model with the shallower network layer can be selected. Finally, the refined model pool, the second model pool, is obtained. Its size is significantly smaller than the initial model pool, but its representativeness is basically the same as the initial model pool.

[0048] In step 103, for each of the multiple samples selected from the target dataset according to the first preset rule, steps 103-1 to 103-5 are executed.

[0049] Specifically, the first preset rule can be random selection, but it is not limited to this. It can also be other feasible selection rules, such as selection based on the representativeness of the sample.

[0050] In step 103-1, an alternative image processing model is selected from the model pool.

[0051] In step 103-2, the current sample is input into the target image processing model to obtain the current output of the target image processing model. The target image processing model is a pre-trained model that is isomorphic to the alternative image processing model. The current output of the target image processing model includes the features of the current sample.

[0052] Specifically, the current sample can be a sample randomly selected from the target dataset. In the exemplary embodiments of this disclosure, the overall model structure can include a diffusion model, a target model, and an alternative model; the target model is a model pre-trained based on the target dataset; the alternative model is a model trained based on the alternative dataset; the target model and the alternative model are isomorphic. Since this disclosure uses a distilled image dataset as an example, both the target model and the alternative model can be image processing models. The output of the target image processing model can also be the corresponding classification result and / or features, which are related to the corresponding application scenario, but are not limited to this.

[0053] In step 103-3, the features of the current sample and the prompt words for the current sample are input into the diffusion model to obtain alternative samples, wherein the prompt words are used to instruct the diffusion model to generate alternative samples based on the features of the current sample.

[0054] Specifically, the prompt words can instruct the diffusion model to generate alternative samples based on various features of the current sample. These features include, but are not limited to, type features, style features, or features in vector space, and can also be other feasible features. For example, the prompt words can generate cartoon-type images. The diffusion model can be the SDv1.5 model or any other feasible diffusion model. It is understood that the features of the current sample and the prompt words for the current sample are the conditional inputs to the diffusion model, which can include, but is not limited to, noise. Inputting the features of the current sample obtained by inputting the current sample into the target image processing model and then inputting these features into the diffusion model can be used to distinguish the architectures of different models.

[0055] In step 103-4, the substitute sample is input into the substitute image processing model to obtain the current output of the substitute image processing model.

[0056] Specifically, the output of the alternative image processing model can be the corresponding classification result and / or features, which are related to the corresponding application scenario, but are not limited to this.

[0057] In step 103-5, the gradient matching loss between the target image processing model and the alternative image processing model is calculated based on the current output of the alternative image processing model and the current output of the target image processing model.

[0058] Specifically, gradient matching loss refers to the degree of matching between the gradients of two models. Gradient matching methods in related technologies typically use MSE (Mean Search Estimate) to calculate the loss of the gradients of the two networks, then backpropagate it to the image space of the target dataset for dataset distillation in the pixel space of the target dataset. This disclosure, however, utilizes a diffusion model with visual common sense for gradient backpropagation and optimization. Unlike gradient matching methods in related technologies, this disclosure backpropagates the gradient to the diffusion model, achieving dataset distillation through optimization. The resulting alternative dataset exhibits superior performance and is more targeted.

[0059] Gradient matching loss between the target image processing model and the alternative image processing model The calculation method can be as follows:

[0060]

[0061] in, The gradient of the teacher network (i.e., the target image processing model) is given. The gradient of the student network (i.e., the alternative image processing model) is given. It can be understood that the gradient is the derivative of the loss function with respect to the model parameters, indicating the direction and magnitude of parameter updates. The loss function can be defined based on the model's output and the true labels. The gradient matching loss between the target image processing model and the alternative image processing model can be considered as the gradient matching loss of each model.

[0062] In steps 103-6, the diffusion model is trained based on gradient matching loss.

[0063] Specifically, based on the model structure and loss function described above, the SGD optimizer can be used for optimization until the loss function converges. It can be understood that during the training of the diffusion model, the loss can be obtained by defining the loss based on the output of the alternative image processing model and the corresponding true label.

[0064] Each image processing model in the model pool can be considered an expert model. The steps described above enable dataset distillation under gradient matching using a diffusion model for an expert model (i.e., the isomorphic target image processing model and the alternative image processing model can be considered the same model). The dataset distillation method based on cross-model generalization and diffusion model can be extended to multiple expert models to achieve gradient matching training under multi-expert constraints.

[0065] According to an exemplary embodiment of the present disclosure, the alternative image processing model is a plurality of image processing models selected from a second model pool, and the target image processing model is a plurality of pre-trained image processing models that are isomorphic to the alternative image processing model.

[0066] Specifically, during training, the target image processing model and the alternative image processing model can be replaced with multiple models, such as three target image processing models and three alternative image processing models that are isomorphic to the target image processing model. To obtain multiple models, selection can be made from a second pool of refined models to achieve multi-expert model selection.

[0067] According to an exemplary embodiment of this disclosure, in the first stage of training the diffusion model, the alternative image processing model can be a plurality of image processing models randomly selected from the second model pool; in the second stage of training the diffusion model, the selection probability of each image processing model in the second model pool can be determined based on the magnitude of the gradient matching loss corresponding to each image processing model in the second model pool, and a plurality of image processing models can be selected from the second model pool as alternative image processing models based on the selection probability of each image processing model in the second model pool.

[0068] Specifically, in the first stage of training the diffusion model, i.e. the early stage of training (e.g., the first 15%-20% of epochs, but not limited to this), random selection can be performed; in the second stage of training the diffusion model, i.e. the middle and late stages of training, adaptive selection can be performed based on the magnitude of the gradient matching loss of each model in the second model pool. For example, models with larger loss values ​​can be selected with higher probability, while models with smaller loss values ​​can be selected with lower probability.

[0069] According to an exemplary embodiment of this disclosure, in the process of calculating the gradient matching loss between the target image processing model and the alternative image processing model, for each set of isomorphic target image processing models and alternative image processing models composed of multiple target image processing models and multiple alternative image processing models, the gradient matching loss between the set of isomorphic target image processing models and alternative image processing models can be calculated; the total gradient matching loss can be calculated based on the gradient matching loss between the multiple sets of isomorphic target image processing models and alternative image processing models.

[0070] When using multiple expert models to perform gradient matching simultaneously, there will be multiple loss function values, which need to be fused before overall backpropagation can be performed.

[0071] According to an exemplary embodiment of this disclosure, the gradient matching loss between multiple sets of isomorphic target image processing models and alternative image processing models can be added together to obtain a total gradient matching loss.

[0072] According to an exemplary embodiment of this disclosure, the gradient matching loss between multiple isomorphic target image processing models and alternative image processing models can be weighted and summed to obtain the total gradient matching loss.

[0073] There are various ways to fuse multi-expert gradient losses, including direct addition, weighted addition, and more, but not limited to these. The total gradient matching loss... The calculation method can be shown below:

[0074]

[0075] in, The gradient of the first target image processing model. The gradient of the first alternative image processing model, The gradient of the second target image processing model. The gradient of the second alternative image processing model, The gradient of the third target image processing model. The gradient of the third alternative image processing model. , , As weight.

[0076] In step 104, in response to the completion of training, the set of all the obtained alternative samples is used as the alternative dataset after distillation of the target dataset.

[0077] Specifically, the number of samples in the alternative dataset can be increased or decreased by adjusting the number of runs of the diffusion model. Ultimately, a general alternative dataset that can be adapted to multiple models can be obtained by training the diffusion model, allowing for the deployment and application of the dataset based on the alternative dataset. After obtaining the final alternative dataset, it can be used as a training dataset for subsequent model training, such as for face recognition tasks, but is not limited to this.

[0078] According to exemplary embodiments of this disclosure, the diffusion model dataset distillation method based on multi-expert fusion optimization may include, as follows: Figure 2 The steps shown, Figure 2 This diagram illustrates a flowchart of a diffusion model dataset distillation method based on multi-expert fusion optimization in an exemplary embodiment of this disclosure. (Refer to...) Figure 2 First, a multi-expert (model) pool can be constructed to summarize and filter the model structures used for the application scenarios targeted by the target dataset. Then, a diffusion model distillation method based on cross-model generalization can be performed. This step can be performed using one target image processing model and an alternative image processing model, or multiple target image processing models and alternative image processing models, to complete gradient matching training under multi-expert constraints. Finally, the obtained alternative dataset can be used to deploy and apply the dataset.

[0079] Figure 3 A block diagram of a dataset distillation apparatus based on cross-model generalization and diffusion models is shown in an exemplary embodiment of this disclosure.

[0080] Reference Figure 3 The exemplary embodiments of this disclosure also provide a dataset distillation apparatus 300 based on cross-model generalization and diffusion models, which may include, but is not limited to, a target dataset acquisition unit 301, a model pool generation unit 302, a data distillation unit 303, and a data generation unit 304.

[0081] The target dataset acquisition unit 301 can acquire the target dataset, which is an image dataset.

[0082] The model pool generation unit 302 can obtain multiple image processing models corresponding to the application scenarios of the target dataset to obtain a model pool.

[0083] The data distillation unit 303 can perform the following operations for each sample from a plurality of samples selected from the target dataset according to a first preset rule: selecting an alternative image processing model from a model pool; inputting the current sample into the target image processing model to obtain the current output of the target image processing model, wherein the target image processing model is a pre-trained model isomorphic to the alternative image processing model, and the current output of the target image processing model includes the features of the current sample; inputting the features of the current sample and a prompt word for the current sample into a diffusion model to obtain an alternative sample, wherein the prompt word is used to instruct the diffusion model to generate an alternative sample based on the features of the current sample; inputting the alternative sample into the alternative image processing model to obtain the current output of the alternative image processing model; calculating the gradient matching loss between the target image processing model and the alternative image processing model based on the current output of the alternative image processing model and the current output of the target image processing model; and training the diffusion model based on the gradient matching loss.

[0084] The data generation unit 304 can, in response to the completion of training, use the set of all the obtained alternative samples as an alternative dataset after distillation of the target dataset.

[0085] According to an exemplary embodiment of this disclosure, the dataset distillation apparatus 300 based on cross-model generalization and diffusion models may also include, but is not limited to, a first model pool screening unit (not shown in the figure), which can screen models in the model pool based on a second preset rule to obtain a first model pool. The second preset rule is that for multiple image processing models with the same basic architecture, if the difference in the number of network layers is less than a first preset threshold, the image processing model with the fewest network layers is selected. The data distillation unit 303 can select alternative image processing models from the first model pool.

[0086] According to an exemplary embodiment of this disclosure, the dataset distillation apparatus 300 based on cross-model generalization and diffusion models may also include, but is not limited to, a second model pool screening unit (not shown in the figure). For each image processing model in the first model pool, the sub-networks in each image processing model are embedded and mapped respectively, and the similarity between each pair of image processing models is calculated. The similarity between each pair of image processing models is obtained by calculating the similarity between sub-networks of a preset number of layers between two image processing models. The preset number of layers is the number of network layers of the image processing model with fewer network layers between the two image processing models. The image processing model with the fewest network layers is selected from each group of image processing models in the first model pool to obtain the second model pool. Each group of image processing models contains at least two image processing models with similarity greater than a second preset threshold. The data distillation unit 303 can select alternative image processing models from the second model pool.

[0087] According to an exemplary embodiment of this disclosure, the alternative image processing model is a plurality of image processing models selected from a second model pool, and the target image processing model is a plurality of pre-trained image processing models that are isomorphic to the alternative image processing model; wherein, the data distillation unit 303 can calculate the gradient matching loss between each set of isomorphic target image processing models and alternative image processing models in a plurality of sets of isomorphic target image processing models and alternative image processing models composed of a plurality of target image processing models and a plurality of alternative image processing models; and calculate the total gradient matching loss based on the gradient matching loss between the plurality of isomorphic target image processing models and alternative image processing models.

[0088] According to exemplary embodiments of the present disclosure, the dataset distillation apparatus 300 based on cross-model generalization and diffusion models may also include, but is not limited to, a model selection unit (not shown in the figure), which may, in the first stage of training the diffusion model, replace the image processing model with multiple image processing models randomly selected from the second model pool; in the second stage of training the diffusion model, determine the selection probability of each image processing model in the second model pool based on the magnitude of the gradient matching loss corresponding to each image processing model in the second model pool, and select multiple image processing models from the second model pool as replacement image processing models based on the selection probability of each image processing model in the second model pool.

[0089] According to an exemplary embodiment of the present disclosure, the data distillation unit 303 can add the gradient matching losses between multiple sets of isomorphic target image processing models and alternative image processing models to obtain the total gradient matching loss.

[0090] According to an exemplary embodiment of the present disclosure, the data distillation unit 303 can perform a weighted summation of the gradient matching losses between multiple sets of isomorphic target image processing models and alternative image processing models to obtain the total gradient matching loss.

[0091] It is understood that the specific implementation process of the dataset distillation device 300 based on the cross-model generalization and diffusion model described above is largely the same as that of the dataset distillation method based on the cross-model generalization and diffusion model described above, and will not be repeated here. The dataset distillation device 300 based on the cross-model generalization and diffusion model can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these devices may correspond to dedicated integrated circuits, pure software code, or modules combining software and hardware. Furthermore, one or more functions implemented by these devices may also be uniformly executed by components in a physical entity device (e.g., a processor, client, or server).

[0092] Figure 4 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

[0093] Reference Figure 4 The electronic device 400 includes at least one memory 401 and at least one processor 402. The at least one memory 401 stores a set of computer-executable instructions. When the set of computer-executable instructions is executed by the at least one processor 402, a dataset distillation method based on a cross-model generalization and diffusion model according to an exemplary embodiment of the present disclosure is performed.

[0094] As an example, electronic device 400 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 400 is not necessarily a single electronic device, but may be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 400 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0095] In electronic device 400, processor 402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0096] The processor 402 can execute instructions or code stored in the memory 401, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0097] The memory 401 may be integrated with the processor 402, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 401 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory 401 and the processor 402 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 402 to read files stored in the memory.

[0098] In addition, the electronic device 400 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 400 can be interconnected via a bus and / or network.

[0099] According to exemplary embodiments of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one computing device, they cause at least one computing device to perform the above-described dataset distillation method based on cross-model generalization and diffusion models.

[0100] Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers. It should be noted that the instructions can also be used to perform additional steps in addition to those described above, or to perform more specific processing while performing the above steps. The details of these additional steps and further processing have already been mentioned in the description of the relevant methods, so they will not be repeated here to avoid repetition.

[0101] Another embodiment of this disclosure relates to a system comprising at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by at least one computing device, cause at least one computing device to perform the aforementioned dataset distillation method based on cross-model generalization and diffusion models.

[0102] It should be noted that the system according to the exemplary embodiments of this disclosure may rely entirely on the operation of computer programs or instructions to achieve the corresponding functions. That is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., a lib library) to achieve the corresponding functions.

[0103] On the other hand, when the above system is implemented in software, firmware, middleware or microcode, the program code or code segment used to perform the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0104] According to exemplary embodiments of this disclosure, the storage device may be integrated with the computing device, for example, by arranging RAM or flash memory within an integrated circuit microprocessor. Alternatively, the storage device may include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The storage device and the computing device may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the computing device to read instructions stored in the storage device.

[0105] Another embodiment of this disclosure relates to a computer program product, including a computer program / instruction that, when executed by a processor, implements the dataset distillation method based on cross-model generalization and diffusion models as described above.

[0106] According to the dataset distillation method, apparatus, electronic device and storage medium based on cross-model generalization and diffusion model provided in this disclosure, by combining gradient matching method and diffusion model, an alternative dataset applicable to a large number of network structures can be obtained through a single training, and the model trained on the alternative dataset can achieve excellent performance. The method has significantly improved versatility, greatly reduced training cost and greatly improved model training performance, and can be deployed and promoted in practical applications.

[0107] In addition, by introducing multi-expert constraints, good generalization ability can be achieved for different models.

[0108] The foregoing has described various exemplary embodiments of this disclosure. It should be understood that the foregoing description is exemplary only and not exhaustive, and this disclosure is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A dataset distillation method based on cross-model generalization and diffusion models, characterized in that, The method includes: Obtain the target dataset, wherein the target dataset is an image dataset; Obtain multiple image processing models corresponding to the application scenarios of the target dataset to obtain a model pool; The models in the model pool are filtered based on the second preset rule to obtain the first model pool. The second preset rule is that for multiple image processing models with the same basic architecture, if the difference in the number of network layers is less than the first preset threshold, the image processing model with the fewest network layers is selected. For each image processing model in the first model pool, after embedding and mapping the sub-networks in each image processing model, the similarity between each pair of image processing models is calculated. The similarity between each pair of image processing models is obtained by calculating the similarity between the sub-networks of two image processing models with a preset number of layers. The preset number of layers is the number of network layers of the image processing model with fewer network layers between the two image processing models. The image processing model with the fewest network layers is selected from each group of image processing models in the first model pool to obtain the second model pool, wherein each group of image processing models contains at least two image processing models with a similarity greater than a second preset threshold. For each sample among multiple samples selected from the target dataset according to a first preset rule, perform the following operations: An alternative image processing model is selected from the second model pool; The current sample is input into the target image processing model to obtain the current output of the target image processing model, wherein the target image processing model is a pre-trained model isomorphic to the alternative image processing model, and the current output of the target image processing model includes the features of the current sample; The features of the current sample and the prompt words for the current sample are input into the diffusion model to obtain alternative samples, wherein the prompt words are used to instruct the diffusion model to generate the alternative samples based on the features of the current sample; The substitute sample is input into the substitute image processing model to obtain the current output of the substitute image processing model; Based on the current output of the alternative image processing model and the current output of the target image processing model, calculate the gradient matching loss between the target image processing model and the alternative image processing model; The diffusion model is trained based on the gradient matching loss. In response to the completion of training, the set of all the obtained alternative samples is used as the alternative dataset after distillation of the target dataset.

2. The dataset distillation method based on cross-model generalization and diffusion models as described in claim 1, characterized in that, The alternative image processing model is a plurality of image processing models selected from the second model pool, and the target image processing model is a plurality of pre-trained image processing models that are isomorphic to the alternative image processing model. The step of calculating the gradient matching loss between the target image processing model and the alternative image processing model includes: For each set of isomorphic target image processing models and alternative image processing models, which consists of multiple target image processing models and multiple alternative image processing models, calculate the gradient matching loss between the set of isomorphic target image processing models and alternative image processing models. The total gradient matching loss is calculated based on the gradient matching loss between multiple isomorphic target image processing models and alternative image processing models.

3. The dataset distillation method based on cross-model generalization and diffusion models as described in claim 2, characterized in that, In the first stage of training the diffusion model, the alternative image processing model is a plurality of image processing models randomly selected from the second model pool; In the second stage of training the diffusion model, the selection probability of each image processing model in the second model pool is determined based on the magnitude of the gradient matching loss corresponding to each image processing model in the second model pool. Based on the selection probability corresponding to each image processing model in the second model pool, multiple image processing models are selected from the second model pool as the alternative image processing models.

4. The dataset distillation method based on cross-model generalization and diffusion models as described in claim 2, characterized in that, The gradient matching loss based on multiple isomorphic target image processing models and alternative image processing models is used to calculate the total gradient matching loss, which includes: The gradient matching loss between multiple isomorphic target image processing models and alternative image processing models is summed to obtain the total gradient matching loss.

5. The dataset distillation method based on cross-model generalization and diffusion models as described in claim 2, characterized in that, The gradient matching loss based on multiple isomorphic target image processing models and alternative image processing models is used to calculate the total gradient matching loss, which includes: The gradient matching loss between multiple isomorphic target image processing models and alternative image processing models is weighted and summed to obtain the total gradient matching loss.

6. A dataset distillation apparatus based on cross-model generalization and diffusion models, characterized in that, The device includes: The target dataset acquisition unit is configured to acquire a target dataset, wherein the target dataset is an image dataset; The model pool generation unit is configured to: acquire multiple image processing models corresponding to the application scenarios of the target dataset, and obtain a model pool; The first model pool filtering unit is configured to: filter the models in the model pool based on a second preset rule to obtain a first model pool, wherein the second preset rule is that for multiple image processing models with the same basic architecture, if the difference in the number of network layers is less than a first preset threshold, the image processing model with the fewest network layers is selected. The second model pool selection unit is configured to: for each image processing model in the first model pool, embed and map the sub-networks in each image processing model, and calculate the pairwise similarity between each image processing model, wherein the pairwise similarity between each image processing model is obtained by calculating the similarity between the sub-networks of two image processing models with a preset number of layers, wherein the preset number of layers is the number of network layers of the image processing model with fewer network layers between the two image processing models; select the image processing model with the fewest network layers from each group of image processing models in the first model pool to obtain the second model pool, wherein each group of image processing models contains at least two image processing models with a similarity greater than a second preset threshold; The data distillation unit is configured as follows: For each sample among multiple samples selected from the target dataset according to a first preset rule, perform the following operations: An alternative image processing model is selected from the second model pool; The current sample is input into the target image processing model to obtain the current output of the target image processing model, wherein the target image processing model is a pre-trained model isomorphic to the alternative image processing model, and the current output of the target image processing model includes the features of the current sample; The features of the current sample and the prompt words for the current sample are input into the diffusion model to obtain alternative samples, wherein the prompt words are used to instruct the diffusion model to generate the alternative samples based on the features of the current sample; The substitute sample is input into the substitute image processing model to obtain the current output of the substitute image processing model; Based on the current output of the alternative image processing model and the current output of the target image processing model, calculate the gradient matching loss between the target image processing model and the alternative image processing model; The diffusion model is trained based on the gradient matching loss. The data generation unit is configured to, in response to training completion, use the set of all obtained alternative samples as an alternative dataset after distillation of the target dataset.

7. An electronic device, characterized in that, include: At least one processor; At least one memory that stores computer-executable instructions. Wherein, when the computer-executable instructions are executed by the at least one processor, the at least one processor causes the at least one processor to execute the dataset distillation method based on cross-model generalization and diffusion models as described in any one of claims 1-5.

8. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is executed by at least one processor, it causes the at least one processor to perform the dataset distillation method based on cross-model generalization and diffusion models as described in any one of claims 1-5.

Images