Data distillation method, electronic equipment and medium

Abstract

The invention discloses a data distillation method, which comprises the following steps of: acquiring a regression model by adopting a kernel ridge regression method based on an initial synthetic data set, updating the synthetic data set by adopting a meta-learning method based on the regression model, updating the regression model by adopting the kernel ridge regression method based on the updated synthetic data set, and sequentially iterating until a preset requirement is met. The kernel ridge regression method is adopted to evaluate the synthetic data set, so that the model training process can be omitted, the data distillation efficiency is greatly improved, meanwhile, it is ensured that the distillation effect is not strongly bound with a certain specific network, and the application range of the synthetic data set is widened.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a data distillation method, electronic equipment and medium. Background Art

[0002] Deep learning is an important branch of artificial intelligence. It simulates human learning ability by building multi-layer neural networks, thereby realizing functions such as image recognition, natural language processing, and speech recognition. Neural Architecture Search (NAS) is a technology that uses search algorithms to find the optimal or nearly optimal deep learning model structure. It can automatically generate a suitable network architecture according to different task objectives and constraints, and perform training and evaluation, thereby saving the time and cost of manual design and debugging, and improving the performance and efficiency of the model. However, the search process requires the use of automatically generated neural networks to train and evaluate on the data set, which consumes a lot of time and resources.

[0003] To address this problem, currently, a small-scale dataset is usually constructed by selecting some samples from the original data to search for neural structures. However, this method cannot achieve both high compression ratio and high information retention of the dataset. To solve this problem, some studies have begun to use data distillation to generate a synthetic dataset containing only a small number of samples based on the deep learning dataset. The synthetic dataset can retain most of the information of the original dataset and remove duplicate data.

[0004] Most existing data distillation methods require training a neural network distillation model on a synthetic dataset and using the model to evaluate the distillation effect, which makes the process too time-consuming. In addition, the distillation dataset is strongly bound to the architecture of the distillation model, which seriously limits the efficiency of neural structure search. Summary of the invention

[0005] In view of some or all of the problems in the prior art, the first aspect of the present invention provides a method for data distillation, which performs data distillation based on a meta-learning method of kernel ridge regression, comprising:

[0006] Based on the initial synthetic dataset, a kernel ridge regression method is used to obtain a regression model; and

[0007] Based on the regression model, a meta-learning method is adopted to update the synthetic data set;

[0008] The regression model is updated based on the updated synthetic dataset and iterated sequentially until the preset requirements are met.

[0009] Furthermore, the acquisition of the initial synthetic data set includes:

[0010] Randomly generated; or

[0011] A specified number of images and their labels are randomly selected from the original dataset to form an initial synthetic dataset.

[0012] Furthermore, the kernel ridge regression method is used to obtain the regression model including:

[0013] Based on the synthetic dataset, the analytical solution of the regression model is calculated by the kernel ridge regression method.

[0014] Furthermore, the kernel ridge regression method may include multiple groups of mapping functions and corresponding kernel functions.

[0015] Furthermore, the kernel function includes a neural tangent kernel.

[0016] Furthermore, the kernel function includes a neural network Gaussian process kernel.

[0017] Further, updating the synthetic dataset using a meta-learning method includes:

[0018] Based on the regression model, calculating the loss function of the synthetic data set; and

[0019] The synthetic dataset is updated so as to minimize the loss function.

[0020] Furthermore, the preset requirement includes that the loss function of the synthetic data set is lower than a preset value.

[0021] Based on the data distillation method as described above, the second aspect of the present invention provides an electronic device for data distillation, which includes a memory and a processor, wherein the memory is configured to store a computer program, and the computer program executes the data distillation method as described above when the processor is running.

[0022] The third aspect of the present invention also provides a computer-readable storage medium for data distillation, which stores a computer program. When the computer program is run on a processor, it executes the data distillation method as described above.

[0023] This scheme proposes a data distillation method, which adopts the kernel ridge regression method to replace the inner loop in the existing data distillation method based on the meta-learning framework. On the one hand, it omits the model training process, thereby greatly improving the data distillation efficiency. On the other hand, due to the independence of the kernel method and the downstream tasks, it can ensure that the distillation effect is not strongly bound to a specific network, thereby improving the applicability of the obtained synthetic dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] To further illustrate the above and other advantages and features of various embodiments of the present invention, a more specific description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings only depict typical embodiments of the present invention and are therefore not to be considered as limiting the scope thereof. In the accompanying drawings, for clarity, identical or corresponding parts will be represented by identical or similar reference numerals.

[0025] Figure 1 A schematic flow chart showing a data distillation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In the following description, the present invention is described with reference to various embodiments. However, those skilled in the art will recognize that various embodiments may be implemented without one or more specific details or with other replacement and / or additional methods or components. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring the inventive point of the present invention. Similarly, for the purpose of explanation, specific numbers and configurations are set forth to provide a comprehensive understanding of embodiments of the present invention. However, the present invention is not limited to these specific details.

[0027] In this specification, reference to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present invention. The phrase "in one embodiment" appearing in various places in this specification does not necessarily all refer to the same embodiment.

[0028] It should be noted that the embodiments of the present invention describe the method steps in a specific order, but this is only for the purpose of illustrating the specific embodiment, rather than limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0029] The purpose of data distillation is to significantly compress the size of deep learning datasets and improve the efficiency of downstream neural structure search tasks while ensuring the effect. The data distillation task can be expressed as a two-layer optimization problem, in which the outer loop updates the synthetic dataset by minimizing the risk of the model on the target dataset, and the purpose of the inner loop is to optimize the model parameters according to the synthetic dataset. Through nested loops, the synthetic dataset gradually converges to one of the optimal values. In the existing common data distillation methods, the inner loop is usually implemented by the gradient descent method of the neural network, such as the back propagation through time (BPTT) method. The back propagation through time (BPTT) method requires training the neural network model, which is time-consuming and will cause the synthetic dataset to be strongly bound to the trained neural network model architecture. To address this problem, the present invention uses a method of using kernel ridge regression to calculate an analytical solution on the synthetic dataset to replace the inner loop, thereby solving the problems of low efficiency of the existing data distillation process and strong binding of the distilled dataset to a specific model architecture.

[0030] The scheme of the present invention is further described below in conjunction with the accompanying drawings of the embodiments.

[0031] like Figure 1 As shown, a method for data distillation includes:

[0032] First, in step 101, an initial synthetic data set is obtained. In one embodiment of the present invention, the initial synthetic data set S0 is obtained by a random initialization method. In another embodiment of the present invention, the initial synthetic data set S0 is extracted from the original data set. Specifically, m pictures and their corresponding labels are randomly selected from the original data set T to form the initial synthetic data set S0.

[0033] Next, in step 102, a regression model is determined. The regression model is determined by using a kernel ridge regression method. Specifically, in one embodiment of the present invention, an analytical solution of the regression model is calculated by a kernel ridge regression method based on a synthetic data set. Kernel ridge regression includes ridge regression and kernel mechanism, which first requires the definition of a nonlinear mapping function Through the nonlinear mapping function The low-dimensional space data x is mapped to a high-dimensional space, and then a linear regression method, i.e., the ridge regression method, is used in the high-dimensional space to obtain its analytical solution. The regression model can be expressed as The analytical solution of the regression model is the value of w, which can also be understood as the parameter of the regression model. In one embodiment of the present invention, the analytical solution is obtained by minimizing the objective function, and the objective function of the ridge regression method is the objective function of the least squares regression model plus the penalty term of the L2 norm:

[0034] J(β)=‖y-Xβ‖ 2 +λ‖β‖ 2 ;

[0035] where X∈R n×m is the input variable, y∈R n×1 is the output variable, β is the parameter of the regression model, and since λ is a non-negative number, the objective function can be further expressed as:

[0036] J(β)=(y-Xβ) T (y-Xβ)+λβ T β=y T yy T Xβ-β T X T y+β T X T Xβ+λβ T β,

[0037] To minimize J(β), its derivative can be set to 0, and the parameter β can be obtained:

[0038] β=(X T X+λI m×m ) -1 X T y,

[0039] In kernel ridge regression, replace X with in Finally, we can get the parameters of the regression model Further, in one embodiment of the present invention, the kernel function is defined as a Gram matrix between two variables, that is: Then the parameters

[0040] It should be understood that in order to improve the effect of the synthetic data set on the migration task, in one embodiment of the present invention, multiple sets of different nonlinear mapping functions and corresponding kernel functions can be used to perform kernel ridge regression operations. For example, in order to improve the effect of the synthetic data set on a complex data set, in one embodiment of the present invention, methods such as the neural tangent kernel can also be used to improve the fitting ability of the kernel function. For another example, in order to further reduce the computational overhead of the Gram matrix, in one embodiment of the present invention, methods such as the neural network Gaussian process kernel can also be used; and

[0041] Finally, in step 103, the synthetic data set is updated. Based on the analytical solution described above, in one embodiment of the present invention, the loss function of the synthetic data set is calculated as follows:

[0042]

[0043] Among them, y t For data set X t The corresponding label, and y s For data set X s The corresponding labels, where the dataset X t For the dataset X s Based on the synthetic data set obtained by distillation, the meta-learning method is used to update the synthetic data set based on the loss function. Specifically, in one embodiment of the present invention, the loss function is minimized by updating the data of the synthetic data set and its corresponding labels. In practical applications, multiple iterations are usually required, that is, steps 102 and 103 are repeated continuously. The analytical solution and the synthetic data set are iteratively updated to make the synthetic data set meet the preset requirements, such as making the loss function lower than the preset value, and obtaining a synthetic data set that is only a few hundredths of the size of the original data set. The synthetic data set can retain most of the original information, and replace the original data set for neural architecture search. Under the premise of ensuring the search effect, the time required for neural structure search can be greatly reduced, thereby improving the search efficiency. In addition, due to the independence of the kernel method and the downstream task, it can also ensure that the distillation effect is not strongly bound to a specific network.

[0044] Based on the data distillation method as described above, the present invention also provides an electronic device for data distillation, which includes a memory and a processor, wherein the memory is configured to store a computer program, and the computer program executes the data distillation method as described above when the processor is running.

[0045] The present invention further provides a computer-readable storage medium for data distillation, which stores a computer program, and the computer program executes the method of data distillation as described above when running on a processor. The computer-readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device, such as an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. Specifically, the computer-readable storage medium includes, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination of the above.

[0046] In an embodiment of the present invention, the computer program can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer program from the network and forwards it for storage in a computer-readable storage medium in each computing / processing device.

[0047] In an embodiment of the present invention, the computer program may be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-dependent instruction, a microcode, a firmware instruction, a state setting data, or a source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be executed by using the state information of the computer program to personalize the electronic circuit.

[0048] Although various embodiments of the present invention are described above, it should be understood that they are presented as examples only and not as limitations. It is obvious to those skilled in the relevant art that various combinations, modifications and changes can be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should only be defined according to the attached claims and their equivalents.

Claims

1. A method for data distillation, characterized in that: Includes steps: Based on the initial synthetic dataset, the kernel ridge regression method is used to obtain the regression model; Based on the regression model, a meta-learning method is adopted to update the synthetic data set; as well as Based on the updated synthetic dataset, the kernel ridge regression method is used to update the regression model, and it is iterated sequentially until the preset requirements are met.

2. The method according to claim 1, characterized in that The acquisition of the initial synthetic dataset includes the following steps: Get an initial synthetic dataset by random generation; or A specified number of images and their labels are randomly selected from the original dataset to form an initial synthetic dataset.

3. The method according to claim 1, characterized in that The kernel ridge regression method is used to obtain the regression model, which includes the following steps: Based on the synthetic data set, the analytical solution w of the regression model is calculated by the kernel ridge regression method, where the regression model is expressed as: in is a nonlinear mapping function.

4. The method according to claim 1, characterized in that The kernel ridge regression method includes at least one set of mapping functions and corresponding kernel functions.

5. The method according to claim 4, characterized in that The kernel function includes a neural tangent kernel.

6. The method according to claim 4, characterized in that The kernel function includes a neural network Gaussian process kernel.

7. The method according to claim 1, characterized in that Updating the synthetic dataset using a meta-learning method includes: Based on the regression model, calculating the loss function of the synthetic data set; and The synthetic dataset is updated so as to minimize the loss function.

8. The method according to claim 7, characterized in that The preset requirement includes: the loss function of the synthetic data set is lower than a preset value.

9. An electronic device for data distillation, characterized in that: It comprises a memory and a processor, wherein the memory is configured to store a computer program, and the computer program executes the data distillation method as claimed in any one of claims 1 to 8 when the processor is running.

10. A computer-readable storage medium for data distillation, characterized in that: A computer program is stored, and when the computer program is run on a processor, the method for data distillation according to any one of claims 1 to 8 is executed.

Images