Data set distillation method and device based on cross-model generalization and diffusion model

Abstract

The invention relates to a data set distillation method and device based on cross-model generalization and a diffusion model, and the method comprises the steps: obtaining a plurality of image processing models corresponding to an application scene of a target data set, and obtaining a model pool; selecting from the model pool to obtain a replacement image processing model; inputting the features of the current sample and the cue words for the current sample into a diffusion model to obtain a replacement sample; inputting the alternative sample into the alternative image processing model to obtain the current output of the alternative image processing model; wherein the target image processing model and the alternative image processing model are isomorphic; calculating gradient matching loss between the target image processing model and the alternative image processing model based on the current output of the alternative image processing model and the current output of the target image processing model; training a diffusion model based on gradient matching loss; and in response to completion of training, obtaining a replacement data set after distillation of the target data set.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a dataset distillation method, device, electronic device, and storage medium based on cross-model generalization and diffusion models. Background Art

[0002] Dataset distillation is a dataset concentration technique that aims to condense a target dataset T of N samples into a surrogate dataset S of M samples (M is much smaller than N, and M samples are generally new and different from the N samples). The surrogate dataset S can be used to train deep learning models at a much lower training cost than the target dataset T. Furthermore, the performance of models trained on S is roughly equivalent to that of models trained on T.

[0003] However, the dataset distillation methods in related technologies cannot take into account both the excellent performance of dataset distillation and the good generalization ability of different structural models. Summary of the Invention

[0004] The dataset distillation method, device, electronic device and storage medium based on cross-model generalization and diffusion models provided by the exemplary embodiments of the present disclosure can at least solve the above-mentioned technical problems and other technical problems not mentioned above.

[0005] According to one aspect of the present disclosure, a data set distillation method based on cross-model generalization and diffusion models is provided, the method comprising: obtaining a target data set, wherein the target data set is an image data set; obtaining a plurality of image processing models corresponding to the application scenarios of the target data set to obtain a model pool; for each sample of a plurality of samples selected from the target data set according to a first preset rule, performing the following operations: selecting a replacement image processing model from the model pool; inputting the current sample into the target image processing model to obtain a current output of the target image processing model, wherein the target image processing model is a pre-trained model isomorphic to the replacement image processing model, and the current output of the target image processing model includes the The invention relates to a method for obtaining a replacement sample by inputting the feature of the current sample and a prompt word for the current sample into a diffusion model to obtain a replacement sample, wherein the prompt word is used to instruct the diffusion model to generate the replacement sample based on the feature of the current sample; inputting the replacement sample into the replacement image processing model to obtain a current output of the replacement image processing model; calculating a gradient matching loss between the target image processing model and the replacement image processing model based on the current output of the replacement image processing model and the current output of the target image processing model; training the diffusion model based on the gradient matching loss; and in response to completion of the training, using the obtained set of all replacement samples as a replacement dataset after distillation of the target dataset.

[0006] Optionally, the method further includes: screening the models in the model pool based on a second preset rule to obtain a first model pool, wherein the second preset rule is that for multiple image processing models with the same basic architecture, when the difference in the number of network layers is less than a first preset threshold, the image processing model with the least number of network layers is selected.

[0007] Optionally, the method also includes: for each image processing model in the first model pool, embedding and mapping the sub-networks in each image processing model respectively, and calculating the similarity between each pair of image processing models, wherein the similarity between each pair of image processing models is obtained by calculating the similarity between the sub-networks of a preset number of layers between the two image processing models, and the preset number of layers is the number of network layers of the image processing model with fewer network layers between the two image processing models; selecting the image processing model with the least number of network layers from each group of image processing models in the first model pool to obtain a second model pool, wherein each group of image processing models contains at least two image processing models whose similarity is greater than a second preset threshold.

[0008] Optionally, the replacement image processing model is a plurality of image processing models selected from the second model pool, and the target image processing model is a plurality of pre-trained image processing models that are isomorphic to the replacement image processing model; wherein, the calculation of the gradient matching loss between the target image processing model and the replacement image processing model includes: for each group of isomorphic target image processing models and replacement image processing models in multiple groups of isomorphic target image processing models and replacement image processing models consisting of multiple target image processing models and multiple replacement image processing models, calculating the gradient matching loss between the group of isomorphic target image processing models and replacement image processing models; and calculating the total gradient matching loss based on the gradient matching losses between the multiple groups of isomorphic target image processing models and replacement image processing models.

[0009] Optionally, in the first stage of training the diffusion model, the replacement image processing model is a plurality of image processing models randomly selected from the second model pool; in the second stage of training the diffusion model, based on the size of the gradient matching loss corresponding to each image processing model in the second model pool, the corresponding selection probability of each image processing model in the second model pool is determined, and based on the corresponding selection probability of each image processing model in the second model pool, multiple image processing models are selected from the second model pool as the replacement image processing models.

[0010] Optionally, the total gradient matching loss is calculated based on the gradient matching loss between multiple groups of isomorphic target image processing models and alternative image processing models, including: adding the gradient matching losses between multiple groups of isomorphic target image processing models and alternative image processing models to obtain the total gradient matching loss.

[0011] Optionally, the total gradient matching loss is calculated based on the gradient matching loss between multiple groups of isomorphic target image processing models and alternative image processing models, including: weighted addition of the gradient matching losses between multiple groups of isomorphic target image processing models and alternative image processing models to obtain the total gradient matching loss.

[0012] According to another aspect of the present disclosure, a data set distillation device based on a cross-model generalization and diffusion model is also provided, the device comprising: a target data set acquisition unit, configured to: acquire a target data set, wherein the target data set is an image data set; a model pool generation unit, configured to: acquire a plurality of image processing models corresponding to the application scenario of the target data set to obtain a model pool; a data distillation unit, configured to: for each sample of a plurality of samples selected from the target data set according to a first preset rule, perform the following operations: select a replacement image processing model from the model pool; input the current sample into the target image processing model to obtain the current output of the target image processing model, wherein the target image processing model is a pre-trained model isomorphic to the replacement image processing model, and the target The current output of the image processing model includes the features of the current sample; the features of the current sample and the prompt word for the current sample are input into the diffusion model to obtain a replacement sample, wherein the prompt word is used to instruct the diffusion model to generate the replacement sample based on the features of the current sample; the replacement sample is input into the replacement image processing model to obtain the current output of the replacement image processing model; based on the current output of the replacement image processing model and the current output of the target image processing model, the gradient matching loss between the target image processing model and the replacement image processing model is calculated; the diffusion model is trained based on the gradient matching loss; the data generation unit is configured to: in response to the completion of training, use the set of all replacement samples obtained as a replacement data set after distillation of the target data set.

[0013] Optionally, the device also includes: a first model pool screening unit, configured to: screen the models in the model pool based on a second preset rule to obtain a first model pool, wherein the second preset rule is that for multiple image processing models with the same basic architecture, when the difference in the number of network layers is less than a first preset threshold, the image processing model with the least number of network layers is selected; wherein the data distillation unit is configured to: select the alternative image processing model from the first model pool.

[0014] Optionally, the device also includes: a second model pool screening unit, configured to: for each image processing model in the first model pool, embed and map the sub-networks in each image processing model respectively, and then calculate the similarity between each pair of image processing models, wherein the similarity between each pair of image processing models is obtained by calculating the similarity between the sub-networks of a preset number of layers between the two image processing models, and the preset number of layers is the number of network layers of the image processing model with fewer network layers between the two image processing models; select the image processing model with the least number of network layers from each group of image processing models in the first model pool to obtain a second model pool, wherein each group of image processing models contains at least two image processing models whose similarity is greater than a second preset threshold; wherein the data distillation unit is configured to: select the replacement image processing model from the second model pool.

[0015] Optionally, the replacement image processing model is a plurality of image processing models selected from the second model pool, and the target image processing model is a plurality of pre-trained image processing models that are isomorphic to the replacement image processing model; wherein the data distillation unit is configured to: for each group of isomorphic target image processing models and replacement image processing models in multiple groups of isomorphic target image processing models and replacement image processing models consisting of multiple target image processing models and multiple replacement image processing models, calculate the gradient matching loss between the group of isomorphic target image processing models and replacement image processing models; and calculate the total gradient matching loss based on the gradient matching losses between the multiple groups of isomorphic target image processing models and replacement image processing models.

[0016] Optionally, the device also includes: a model selection unit, configured to: in the first stage of training the diffusion model, the replacement image processing model is a plurality of image processing models randomly selected from the second model pool; in the second stage of training the diffusion model, based on the size of the gradient matching loss corresponding to each image processing model in the second model pool, determine the selection probability corresponding to each image processing model in the second model pool, and based on the selection probability corresponding to each image processing model in the second model pool, select multiple image processing models in the second model pool as the replacement image processing models.

[0017] Optionally, the data distillation unit is configured to: add the gradient matching losses between multiple groups of isomorphic target image processing models and replacement image processing models to obtain the total gradient matching loss.

[0018] Optionally, the data distillation unit is configured to: perform weighted addition of gradient matching losses between multiple groups of isomorphic target image processing models and replacement image processing models to obtain the total gradient matching loss.

[0019] According to another aspect of an embodiment of the present disclosure, an electronic device is also provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein when the computer-executable instructions are executed by the at least one processor, the at least one processor is prompted to execute any of the above-described dataset distillation methods based on cross-model generalization and diffusion models.

[0020] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is also provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute any of the dataset distillation methods based on cross-model generalization and diffusion models as described above.

[0021] According to another aspect of an embodiment of the present disclosure, a system is also provided, comprising at least one computing device and at least one storage device storing instructions, wherein, when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute any of the above-described dataset distillation methods based on cross-model generalization and diffusion models.

[0022] According to another aspect of an embodiment of the present disclosure, a computer program product is also provided, including a computer program / instruction, which, when executed by a processor, implements the dataset distillation method based on cross-model generalization and diffusion model as described in any one of the above.

[0023] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects: According to the dataset distillation method, device, electronic device and storage medium based on cross-model generalization and diffusion model disclosed in the present invention, by combining the gradient matching method and the diffusion model, an alternative dataset that is universally applicable to a large number of network structures can be obtained through one training, and the performance of the model trained based on the alternative dataset can achieve excellent performance; the general ability of this method is significantly improved, the training cost is greatly reduced, and the model training performance is greatly improved, and it can be deployed and promoted in practical applications.

[0024] In addition, by introducing multi-expert constraints, good generalization ability can be achieved for different models. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0026] Figure 1 A schematic diagram illustrating a flow chart of a dataset distillation method based on cross-model generalization and diffusion models in an exemplary embodiment of the present disclosure is shown; Figure 2 A schematic diagram illustrating a flow chart of a diffusion model dataset distillation method based on multi-expert fusion optimization in an exemplary embodiment of the present disclosure is shown; Figure 3 A block diagram of a dataset distillation apparatus based on cross-model generalization and diffusion models in an exemplary embodiment of the present disclosure is shown; Figure 4 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0028] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0029] It should be noted that the phrase "at least one of the items" in this disclosure includes three types of parallel situations: "any one of the items", "a combination of any multiple items of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" includes the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing steps 1 and 2.

[0030] Dataset distillation techniques in related technologies can be divided into two types.

[0031] The first type is the dataset distillation method based on gradient matching. This type of method is generally targeted at a specific network structure and performs gradient matching on the target dataset and the surrogate dataset during the optimization process. Backpropagation is performed through the gradient metric loss (such as MSE, Mean Squared Error Gradient Loss, mean squared error gradient loss, etc.), and the appropriate surrogate dataset is optimized. The advantage of this type of method is that it has a good dataset distillation effect for a specific network (the performance of the model trained on the surrogate dataset is not much different from the performance of the model trained on the target dataset). However, its disadvantages are also prominent: (1) the dataset needs to be re-distilled for the new network model, that is, each network structure must have a corresponding surrogate dataset; (2) the surrogate datasets of different network structures cannot be used interchangeably, that is, if the surrogate dataset of network structure A is used to train network structure B, the performance of the model of network structure B will be seriously degraded, that is, the generalization ability is poor.

[0032] The second type is a dataset distillation method based on the diffusion model. This type of method uses the visual common sense obtained by the diffusion model trained on large-scale datasets to directly use "vintage images" to generate alternative datasets. For example, directly inputting prompt words related to "airplane" to generate aircraft-type images, and then combining some common data augmentation methods (random cropping, random flipping, etc.) to improve the diversity of the dataset. The advantages of this type of method are low training cost and no dependence on a specific network structure; the disadvantages are: (1) due to the lack of optimization for a specific network, the training performance of the alternative dataset is poor, and there is a large performance gap with the first method; (2) its performance makes it impossible to be promoted in practical applications and is still in the laboratory stage.

[0033] In order to solve the above problems, the present disclosure provides a dataset distillation method, device, electronic device and storage medium based on cross-model generalization and diffusion model. By combining the gradient matching method and the diffusion model, an alternative dataset that is universally applicable to a large number of network structures can be obtained through one training, and the performance of the model trained based on the alternative dataset can achieve excellent performance; the general capability of this method is significantly improved, the training cost is greatly reduced, and the model training performance is greatly improved, and it can be deployed and promoted in practical applications.

[0034] Below, we will refer to Figures 1 to 4 The present invention specifically describes the dataset distillation method, device, electronic device and storage medium based on cross-model generalization and diffusion model.

[0035] Figure 1 A flowchart illustrating a dataset distillation method based on cross-model generalization and diffusion models in an exemplary embodiment of the present disclosure is shown.

[0036] Reference Figure 1 In step 101, a target data set is obtained, where the target data set is an image data set.

[0037] Specifically, the target dataset is the dataset to be distilled. It should be noted that in some other embodiments, the target dataset may also be a text dataset, but the present disclosure uses the distillation of an image dataset as an example for illustration.

[0038] In step 102, a plurality of image processing models corresponding to the application scenarios of the target dataset are obtained to obtain a model pool. Specifically, the scenarios in which the corresponding image processing models of the target dataset may be used can be traversed to obtain a plurality of different types of corresponding image processing models. For example, when the target dataset is a face recognition dataset, there may be several server-side models and several lightweight client-side models. In this case, the union of these model structures can be the initial model pool. It is understood that the target dataset is not limited to the face recognition dataset listed above, but can also be a dataset used in any other scenario, such as an object detection dataset used in an object detection scenario.

[0039] Actual AI applications require a variety of network architectures to adapt to different environments. For example, cloud servers can use complex, high-performance models, while IoT (Internet of Things) devices can only use lightweight models due to computing power constraints. To achieve good generalization across different models, it is important to select representative model architectures for the application.

[0040] According to an exemplary embodiment of the present disclosure, the models in the model pool can be screened based on the second preset rule to obtain a first model pool, wherein the second preset rule can be that, for multiple image processing models with the same basic architecture, when the difference in the number of network layers is less than a first preset threshold, the image processing model with the least number of network layers is selected; and an alternative image processing model can be selected from the first model pool.

[0041] Specifically, since the initial model pool may contain a large number of models with different structures (for example, ResNet18, ResNet34, ResNet50, MobileNetV2, etc., but not limited to this), the model pool can be initially screened based on the second preset rule. The first preset threshold can be 50%, but is not limited to this. Therefore, for similar model structures, if the difference in the number of network layers of two models is less than 50%, then based on computational considerations, only the smaller one can be retained; for example, for ResNet18 and ResNet34, only ResNet18 can be retained; but the specific example is not limited to this. Based on the above rules, the model pool after initial screening can be obtained, that is, the first model pool.

[0042] According to an exemplary embodiment of the present disclosure, for each image processing model in the first model pool, the sub-networks in each image processing model can be embedded and mapped respectively, and the similarity between each pair of image processing models can be calculated, wherein the similarity between each pair of image processing models can be obtained by calculating the similarity between the sub-networks with a preset number of layers between the two image processing models, and the preset number of layers can be the number of network layers of the image processing model with fewer network layers between the two image processing models; the image processing model with the least number of network layers can be selected from each group of image processing models in the first model pool to obtain a second model pool, wherein each group of image processing models contains at least two image processing models with a similarity greater than a second preset threshold; and an alternative image processing model can be selected from the second model pool.

[0043] Specifically, the first model pool can be fine-screened. This can be done based on network similarity calculation methods related to NAS (Neural Architecture Search). For example, after embedding and mapping different operations in the model (i.e., sub-networks in the model), the overall similarity between each model (network) can be calculated. During the similarity calculation process, only the similarity between the first N layers (i.e., a preset number of layers) of the two networks can be calculated, where N can be the number of layers of the shallower network. For a pair of models whose similarity exceeds a second preset threshold (e.g., the second preset threshold can be set to 0.75), only the model with the shallower network layer can be selected. Finally, a fine-screened model pool, i.e., the second model pool, is obtained. Its size is significantly smaller than the initial model pool, but its representativeness is basically the same as the initial model pool.

[0044] In step 103 , steps 103 - 1 to 103 - 5 are performed for each sample of a plurality of samples selected from the target data set according to a first preset rule.

[0045] Specifically, the first preset rule may be random selection, but is not limited thereto, and may also be other feasible selection rules, for example, selection may be made based on the representativeness of the sample.

[0046] In step 103 - 1 , a replacement image processing model is selected from a model pool.

[0047] In step 103-2, the current sample is input into the target image processing model to obtain the current output of the target image processing model, wherein the target image processing model is a pre-trained model isomorphic to the replacement image processing model, and the current output of the target image processing model includes the features of the current sample.

[0048] Specifically, the current sample can be a sample randomly selected from the target dataset. In the exemplary embodiments of the present disclosure, the overall model structure can include a diffusion model, a target model, and a surrogate model. The target model is a model pre-trained based on the target dataset; the surrogate model is a model trained based on the surrogate dataset; and the target model and the surrogate model are isomorphic. Since the present disclosure uses a distilled image dataset as an example, the target model and the surrogate model can both be image processing models. The output of the target image processing model can also be corresponding classification results and / or features, which are related to the corresponding application scenario, but are not limited to this.

[0049] In step 103 - 3 , the features of the current sample and the prompt word for the current sample are input into the diffusion model to obtain a replacement sample, wherein the prompt word is used to instruct the diffusion model to generate a replacement sample based on the features of the current sample.

[0050] Specifically, the prompt word can instruct the diffusion model to generate a replacement sample based on various features of the current sample. These features include, but are not limited to, type features, style features, or features in vector space, and can also be other feasible features. The prompt word can, for example, generate a cartoon-like image. The diffusion model can be the SDv1.5 model or any other feasible diffusion model. It is understood that the features of the current sample and the prompt word for the current sample serve as conditional inputs to the diffusion model, which can include, but is not limited to, noise. Inputting the features of the current sample obtained by inputting the current sample into the target image processing model into the diffusion model can be used to distinguish between different model architectures.

[0051] In step 103 - 4 , the replacement sample is input into the replacement image processing model to obtain the current output of the replacement image processing model.

[0052] Specifically, the output of the replacement image processing model may be a corresponding classification result and / or feature, which is related to the corresponding application scenario, but is not limited thereto.

[0053] In step 103 - 5 , based on the current output of the substitute image processing model and the current output of the target image processing model, a gradient matching loss between the target image processing model and the substitute image processing model is calculated.

[0054] Specifically, the gradient matching loss is the degree of matching between the gradients of the two models. The gradient matching method in the related art generally uses MSE to calculate the gradient loss of the two networks, and then directly back-propagates it to the image space of the target dataset, and performs dataset distillation in the pixel space of the target dataset. The present disclosure can use a diffusion model with visual common sense to perform gradient back-propagation and optimization. Unlike the gradient matching method in the related art, the present disclosure can back-propagate the gradient to the diffusion model, and achieve the purpose of dataset distillation by optimizing the diffusion model. The performance of the alternative dataset obtained is better and more targeted.

[0055] Gradient matching loss between the target image processing model and the alternative image processing model The calculation method can be as follows:

[0056] in, is the gradient of the teacher network (i.e., the target image processing model), is the gradient of the student network (i.e., the replacement image processing model). It can be understood that the gradient is the derivative of the loss function with respect to the model parameters, indicating the direction and magnitude of the parameter update. The loss function can be defined based on the model output and the true label. The gradient matching loss between the target image processing model and the replacement image processing model can be considered as the gradient matching loss corresponding to the target image processing model and the replacement image processing model.

[0057] In step 103 - 6 , the diffusion model is trained based on the gradient matching loss.

[0058] Specifically, based on the above model structure and loss function, an SGD optimizer can be used for optimization until the loss function converges. It is understood that during the training of the diffusion model, the alternative image processing model can be trained based on the output of the alternative image processing model and the loss defined by the corresponding true label.

[0059] Each image processing model in the model pool can be considered an expert model. The above steps enable dataset distillation under gradient matching using the diffusion model for a single expert model (i.e., the isomorphic target image processing model and the replacement image processing model can be considered the same model). The dataset distillation method based on cross-model generalization and the diffusion model can be extended to multiple expert models to achieve gradient matching training under multi-expert constraints.

[0060] According to an exemplary embodiment of the present disclosure, the replacement image processing model is a plurality of image processing models selected from the second model pool, and the target image processing model is a plurality of pre-trained image processing models that are isomorphic to the replacement image processing model.

[0061] Specifically, during training, the target image processing model and the alternative image processing model can be replaced with multiple models, for example, three target image processing models and three alternative image processing models that are isomorphic to the target image processing model. To obtain multiple models, they can be selected from the carefully screened second model pool to achieve multi-expert model selection.

[0062] According to an exemplary embodiment of the present disclosure, in the first stage of training the diffusion model, the replacement image processing model can be a plurality of image processing models randomly selected from the second model pool; in the second stage of training the diffusion model, the selection probability corresponding to each image processing model in the second model pool can be determined based on the size of the gradient matching loss corresponding to each image processing model in the second model pool, and based on the selection probability corresponding to each image processing model in the second model pool, a plurality of image processing models can be selected from the second model pool as replacement image processing models.

[0063] Specifically, in the first stage of training the diffusion model, i.e., the initial stage of training (e.g., the first 15%-20% of epochs, but not limited thereto), random selection can be performed; in the second stage of training the diffusion model, i.e., the middle and late stages of training, adaptive selection can be performed based on the size of the gradient matching loss of each model in the second model pool. For example, a model with a large loss value can have a higher selection probability, while a model with a small loss value can have a lower selection probability.

[0064] According to an exemplary embodiment of the present disclosure, in the process of calculating the gradient matching loss between the target image processing model and the alternative image processing model, for each group of isomorphic target image processing models and alternative image processing models in multiple groups of isomorphic target image processing models and alternative image processing models consisting of multiple target image processing models and multiple alternative image processing models, the gradient matching loss between the group of isomorphic target image processing models and alternative image processing models can be calculated; and the total gradient matching loss can be calculated based on the gradient matching losses between the multiple groups of isomorphic target image processing models and alternative image processing models.

[0065] When using multiple expert models for gradient matching simultaneously, there will be multiple loss function values, which need to be fused before overall backpropagation can be performed.

[0066] According to an exemplary embodiment of the present disclosure, the gradient matching losses between multiple groups of isomorphic target image processing models and replacement image processing models may be added to obtain a total gradient matching loss.

[0067] According to an exemplary embodiment of the present disclosure, the gradient matching losses between multiple groups of isomorphic target image processing models and replacement image processing models may be weightedly added to obtain a total gradient matching loss.

[0068] There are many ways to fuse the multi-expert gradient loss, including direct addition, weighted addition, etc., but not limited to these. The calculation method can be as follows:

[0069] in, is the gradient of the first target image processing model, is the gradient of the first alternative image processing model, is the gradient of the second target image processing model, is the gradient of the second alternative image processing model, is the gradient of the third target image processing model, is the gradient of the third alternative image processing model, 、 、 is the weight.

[0070] In step 104 , in response to the completion of the training, the obtained set of all replacement samples is used as a replacement dataset after distillation of the target dataset.

[0071] Specifically, the number of samples in the replacement dataset can be increased or decreased by adjusting the number of diffusion model runs. Ultimately, by training the diffusion model, a universal replacement dataset that can be adapted to multiple models can be obtained, allowing for dataset deployment and application based on the replacement dataset. Once the final replacement dataset is obtained, it can be used as a training dataset for subsequent model training, such as, but not limited to, face recognition tasks.

[0072] According to an exemplary embodiment of the present disclosure, the diffusion model dataset distillation method based on multi-expert fusion optimization may include the following: Figure 2 The steps shown, Figure 2 A flow chart of a diffusion model dataset distillation method based on multi-expert fusion optimization in an exemplary embodiment of the present disclosure is shown. Figure 2 First, a multi-expert (model) pool can be constructed to summarize the model structures corresponding to the application scenarios facing the target dataset, or to screen them; then a diffusion model distillation method based on cross-model generalization can be performed. This step can be performed using one target image processing model and an alternative image processing model respectively, or using multiple target image processing models and alternative image processing models respectively to complete the gradient matching training under multi-expert constraints; finally, the obtained alternative dataset can be used to implement the deployment and application of the dataset.

[0073] Figure 3 A block diagram of a dataset distillation apparatus based on cross-model generalization and diffusion models in an exemplary embodiment of the present disclosure is shown.

[0074] Reference Figure 3 , an exemplary embodiment of the present disclosure also provides a dataset distillation device 300 based on a cross-model generalization and diffusion model, which may include but is not limited to a target dataset acquisition unit 301, a model pool generation unit 302, a data distillation unit 303, and a data generation unit 304.

[0075] The target data set acquisition unit 301 may acquire a target data set, wherein the target data set is an image data set.

[0076] The model pool generation unit 302 can obtain a variety of image processing models corresponding to the application scenarios of the target data set to obtain a model pool.

[0077] The data distillation unit 303 can perform the following operations for each sample of multiple samples selected from the target data set according to the first preset rule: select a replacement image processing model from the model pool; input the current sample into the target image processing model to obtain the current output of the target image processing model, wherein the target image processing model is a pre-trained model isomorphic to the replacement image processing model, and the current output of the target image processing model includes the features of the current sample; input the features of the current sample and the prompt word for the current sample into the diffusion model to obtain a replacement sample, wherein the prompt word is used to instruct the diffusion model to generate a replacement sample based on the features of the current sample; input the replacement sample into the replacement image processing model to obtain the current output of the replacement image processing model; based on the current output of the replacement image processing model and the current output of the target image processing model, calculate the gradient matching loss between the target image processing model and the replacement image processing model; and train the diffusion model based on the gradient matching loss.

[0078] In response to the completion of the training, the data generation unit 304 may use the obtained set of all replacement samples as a replacement dataset after distillation of the target dataset.

[0079] According to an exemplary embodiment of the present disclosure, the dataset distillation device 300 based on cross-model generalization and diffusion models may also include but is not limited to a first model pool screening unit (not shown in the figure), which can screen the models in the model pool based on a second preset rule to obtain a first model pool, wherein the second preset rule is that for multiple image processing models with the same basic architecture, when the difference in the number of network layers is less than a first preset threshold, the image processing model with the least network layers is selected; wherein the data distillation unit 303 can select an alternative image processing model from the first model pool.

[0080] According to an exemplary embodiment of the present disclosure, the dataset distillation device 300 based on cross-model generalization and diffusion models may also include but is not limited to a second model pool screening unit (not shown in the figure), which can embed and map the sub-networks in each image processing model for each image processing model in the first model pool, and then calculate the similarity between each image processing model, wherein the similarity between each image processing model is obtained by calculating the similarity between the sub-networks of a preset number of layers between the two image processing models, and the preset number of layers is the number of network layers of the image processing model with fewer network layers between the two image processing models; the image processing model with the least number of network layers is selected from each group of image processing models in the first model pool to obtain a second model pool, wherein each group of image processing models contains at least two image processing models whose similarity is greater than a second preset threshold; wherein the data distillation unit 303 can select a replacement image processing model from the second model pool.

[0081] According to an exemplary embodiment of the present disclosure, the replacement image processing model is a plurality of image processing models selected from the second model pool, and the target image processing model is a plurality of pre-trained image processing models that are isomorphic to the replacement image processing model; wherein, the data distillation unit 303 can calculate the gradient matching loss between each group of isomorphic target image processing models and replacement image processing models in the multiple groups of isomorphic target image processing models and replacement image processing models composed of the multiple target image processing models and the multiple replacement image processing models; based on the gradient matching losses between the multiple groups of isomorphic target image processing models and replacement image processing models, the total gradient matching loss is calculated.

[0082] According to an exemplary embodiment of the present disclosure, the dataset distillation device 300 based on cross-model generalization and diffusion model may also include but is not limited to a model selection unit (not shown in the figure), which can replace the image processing model with multiple image processing models randomly selected from the second model pool in the first stage of training the diffusion model; in the second stage of training the diffusion model, the selection probability corresponding to each image processing model in the second model pool is determined based on the size of the gradient matching loss corresponding to each image processing model in the second model pool, and based on the selection probability corresponding to each image processing model in the second model pool, multiple image processing models are selected from the second model pool as replacement image processing models.

[0083] According to an exemplary embodiment of the present disclosure, the data distillation unit 303 may add the gradient matching losses between multiple groups of isomorphic target image processing models and replacement image processing models to obtain a total gradient matching loss.

[0084] According to an exemplary embodiment of the present disclosure, the data distillation unit 303 may perform weighted addition on the gradient matching losses between multiple groups of isomorphic target image processing models and replacement image processing models to obtain a total gradient matching loss.

[0085] It can be understood that in the exemplary embodiment of the dataset distillation device 300 based on cross-model generalization and diffusion model, the specific implementation process is roughly the same as the exemplary embodiment of the dataset distillation method based on cross-model generalization and diffusion model, and will not be repeated here. The dataset distillation device 300 based on cross-model generalization and diffusion model can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, these devices may correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. In addition, one or more functions implemented by these devices may also be uniformly executed by components in physical entity devices (for example, processors, clients or servers, etc.).

[0086] Figure 4 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

[0087] Reference Figure 4 The electronic device 400 includes at least one memory 401 and at least one processor 402, wherein the at least one memory 401 stores a set of computer-executable instructions. When the computer-executable instruction set is executed by the at least one processor 402, a data set distillation method based on cross-model generalization and diffusion model according to an exemplary embodiment of the present disclosure is executed.

[0088] As an example, electronic device 400 may be a PC, tablet device, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 400 is not necessarily a single electronic device, but may also be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction set) individually or in combination. Electronic device 400 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that interfaces with local or remote devices (e.g., via wireless transmission).

[0089] In electronic device 400, processor 402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0090] The processor 402 can execute instructions or codes stored in the memory 401, wherein the memory 401 can also store data. Instructions and data can also be sent and received over the network via the network interface device, wherein the network interface device can use any known transmission protocol.

[0091] Memory 401 may be integrated with processor 402, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, memory 401 may comprise a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. Memory 401 and processor 402 may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, or the like, such that processor 402 can access files stored in memory.

[0092] In addition, the electronic device 400 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 400 may be connected to each other via a bus and / or a network.

[0093] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein, when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the above-mentioned dataset distillation method based on cross-model generalization and diffusion model.

[0094] Examples of computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as a multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device configured to store a computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers. It should be noted that the instructions can also be used to perform additional steps in addition to the above steps or to perform more specific processing when executing the above steps. These additional steps and further processing contents have been mentioned in the description of the relevant methods, so they will not be repeated here to avoid repetition.

[0095] Another embodiment of the present disclosure relates to a system comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to perform the above-mentioned dataset distillation method based on cross-model generalization and diffusion model.

[0096] It should be noted that the system according to the exemplary embodiments of the present disclosure can completely rely on the execution of computer programs or instructions to realize the corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (for example, lib library) to realize the corresponding functions.

[0097] On the other hand, when the above-mentioned system is implemented in software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0098] According to exemplary embodiments of the present disclosure, the storage device may be integrated with the computing device, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Furthermore, the storage device may include a standalone device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The storage device and the computing device may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, or the like, such that the computing device can access instructions stored in the storage device.

[0099] Another embodiment of the present disclosure relates to a computer program product, including a computer program / instruction, which, when executed by a processor, implements the dataset distillation method based on cross-model generalization and diffusion model as described in any one of the above.

[0100] According to the dataset distillation method, device, electronic device and storage medium based on cross-model generalization and diffusion model provided by the present disclosure, by combining the gradient matching method and the diffusion model, an alternative dataset that is universally applicable to a large number of network structures can be obtained through one training, and the performance of the model trained based on the alternative dataset can achieve excellent performance; the general capability of this method is significantly improved, the training cost is greatly reduced, and the model training performance is greatly improved, and it can be deployed and promoted in practical applications.

[0101] In addition, by introducing multi-expert constraints, good generalization ability can be achieved for different models.

[0102] While various exemplary embodiments of the present disclosure have been described above, it should be understood that the foregoing description is merely illustrative and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present disclosure. Therefore, the scope of protection of the present disclosure should be determined by the scope of the claims.

Claims

1. A dataset distillation method based on cross-model generalization and diffusion model, characterized in that: The method comprises: Acquire a target data set, wherein the target data set is an image data set; Acquire multiple image processing models corresponding to the application scenarios of the target data set to obtain a model pool; For each sample of the plurality of samples selected from the target data set according to the first preset rule, the following operations are performed: Selecting a replacement image processing model from the model pool; Inputting the current sample into a target image processing model to obtain a current output of the target image processing model, wherein the target image processing model is a pre-trained model isomorphic to the alternative image processing model, and the current output of the target image processing model includes features of the current sample; Inputting the features of the current sample and a prompt word for the current sample into a diffusion model to obtain a replacement sample, wherein the prompt word is used to instruct the diffusion model to generate the replacement sample based on the features of the current sample; Inputting the replacement sample into the replacement image processing model to obtain a current output of the replacement image processing model; Calculating a gradient matching loss between the target image processing model and the substitute image processing model based on a current output of the substitute image processing model and a current output of the target image processing model; Training the diffusion model based on the gradient matching loss; In response to the completion of the training, a set of all obtained replacement samples is used as a replacement dataset after distillation of the target dataset.

2. The dataset distillation method based on cross-model generalization and diffusion model according to claim 1, characterized in that: The method further comprises: The models in the model pool are screened based on a second preset rule to obtain a first model pool, wherein the second preset rule is that, for multiple image processing models with the same basic architecture, when the difference in the number of network layers is less than a first preset threshold, the image processing model with the least number of network layers is selected; Wherein, the step of selecting a replacement image processing model from the model pool comprises: The replacement image processing model is selected from the first model pool.

3. The dataset distillation method based on cross-model generalization and diffusion model as claimed in claim 2, characterized in that: The method further comprises: For each image processing model in the first model pool, after embedding and mapping the subnetworks in each image processing model respectively, calculate the similarity between each image processing model, wherein the similarity between each image processing model is obtained by calculating the similarity between the subnetworks of a preset number of layers between the two image processing models, and the preset number of layers is the number of network layers of the image processing model with fewer network layers between the two image processing models; Selecting an image processing model with the least number of network layers from each group of image processing models in the first model pool to obtain a second model pool, wherein each group of image processing models includes at least two image processing models whose similarity is greater than a second preset threshold; Wherein, the step of selecting a replacement image processing model from the model pool comprises: The replacement image processing model is selected from the second model pool.

4. The dataset distillation method based on cross-model generalization and diffusion model according to claim 3, characterized in that: The replacement image processing models are multiple image processing models selected from the second model pool, and the target image processing models are multiple pre-trained image processing models that are isomorphic to the replacement image processing models; The step of calculating the gradient matching loss between the target image processing model and the alternative image processing model includes: For each set of isomorphic target image processing models and alternative image processing models in a plurality of sets of isomorphic target image processing models and alternative image processing models consisting of a plurality of target image processing models and a plurality of alternative image processing models, calculating a gradient matching loss between the set of isomorphic target image processing models and alternative image processing models; Based on the gradient matching losses between multiple groups of isomorphic target image processing models and alternative image processing models, the total gradient matching loss is calculated.

5. The dataset distillation method based on cross-model generalization and diffusion model according to claim 4, characterized in that: In the first stage of training the diffusion model, the replacement image processing model is a plurality of image processing models randomly selected from the second model pool; In the second stage of training the diffusion model, determining the selection probability corresponding to each image processing model in the second model pool based on the size of the gradient matching loss corresponding to each image processing model in the second model pool; Based on the selection probability corresponding to each image processing model in the second model pool, multiple image processing models are selected from the second model pool as the replacement image processing models.

6. The dataset distillation method based on cross-model generalization and diffusion model according to claim 4, characterized in that: The total gradient matching loss is calculated based on the gradient matching loss between the multiple groups of isomorphic target image processing models and the alternative image processing models, including: The gradient matching losses between multiple groups of isomorphic target image processing models and alternative image processing models are added to obtain the total gradient matching loss.

7. The dataset distillation method based on cross-model generalization and diffusion model according to claim 4, characterized in that: The total gradient matching loss is calculated based on the gradient matching loss between the multiple groups of isomorphic target image processing models and the alternative image processing models, including: The total gradient matching loss is obtained by weighted addition of the gradient matching losses between multiple groups of isomorphic target image processing models and alternative image processing models.

8. A dataset distillation device based on cross-model generalization and diffusion model, characterized in that: The device comprises: The target data set acquisition unit is configured to: acquire a target data set, wherein the target data set is an image data set; A model pool generating unit is configured to: obtain a plurality of image processing models corresponding to the application scenarios of the target data set to obtain a model pool; The data distillation unit is configured as follows: For each sample of the plurality of samples selected from the target data set according to the first preset rule, the following operations are performed: Selecting a replacement image processing model from the model pool; Inputting the current sample into a target image processing model to obtain a current output of the target image processing model, wherein the target image processing model is a pre-trained model isomorphic to the alternative image processing model, and the current output of the target image processing model includes features of the current sample; Inputting the features of the current sample and a prompt word for the current sample into a diffusion model to obtain a replacement sample, wherein the prompt word is used to instruct the diffusion model to generate the replacement sample based on the features of the current sample; Inputting the replacement sample into the replacement image processing model to obtain a current output of the replacement image processing model; Calculating a gradient matching loss between the target image processing model and the substitute image processing model based on a current output of the substitute image processing model and a current output of the target image processing model; Training the diffusion model based on the gradient matching loss; The data generation unit is configured to: in response to the completion of training, use the set of all replacement samples obtained as a replacement data set after distillation of the target data set.

9. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the dataset distillation method based on cross-model generalization and diffusion model as described in any one of claims 1-8.

10. A computer-readable storage medium storing instructions, characterized in that: When the instruction is executed by at least one processor, the at least one processor is prompted to perform the dataset distillation method based on cross-model generalization and diffusion model as described in any one of claims 1-8.

Images