Data-free knowledge distillation learning method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data-free knowledge distillation learning method and device, equipment and a storage medium, and belongs to the technical field of knowledge distillation. The method comprises the following steps: under a current iteration round, determining at least one generator loss weight based on a current iteration number, calculating to obtain a generator loss value, and adjusting generator parameters by using the generator loss value to obtain a trained updated generator under the current iteration round; determining at least one student model loss weight based on the current iteration times and calculating to obtain a student model loss value; adjusting student model parameters by using the student model loss value to obtain a trained updated student model in the current iteration round; wherein the loss weight of the generator and the loss weight of the student model are in an overall dynamic increase trend along with the increase of the number of iterations; and repeatedly performing training on the generator and the student model until the number of iterations reaches a preset value, and determining an updated student model obtained by the last iteration round as a target student model.

Description

Technical Field

[0001] The present application relates to the field of knowledge distillation technology, and in particular to a data-free knowledge distillation learning method, device, equipment and storage medium. Background Art

[0002] Knowledge Distillation (KD) is a model compression method that improves the performance of a lightweight student model by guiding it to imitate a teacher model with better performance and more complex structure. Among them, knowledge distillation without access to the original data is called data-free knowledge distillation.

[0003] In the related technology, a generator is used to generate synthetic samples with similar distribution to the teacher model training data to train the student model. However, the traditional data-free knowledge distillation method does not take into account the impact of synthetic samples of different difficulty levels on the learning effect of the student model, resulting in low efficiency in student model training. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a data-free knowledge distillation learning method, device, equipment and storage medium, aiming to improve the training efficiency of the student model in data-free knowledge distillation learning.

[0005] To achieve the above purpose, a first aspect of an embodiment of the present application proposes a data-free knowledge distillation learning method, the method comprising:

[0006] Step 101: Obtain the number of iterations of the current iteration round, and obtain the first synthetic sample generated by the initial generator under specified generation information;

[0007] Step 102: input the first synthetic sample into the initial student model and the pre-trained teacher model respectively, output the first student prediction result obtained after the initial student model processes the first synthetic sample, and process the first synthetic sample based on the teacher model provided with a random deactivation layer, and output the first teacher prediction result;

[0008] Step 103: determine at least one generator loss weight based on the current number of iterations, and calculate the generator loss value according to the generator loss weight, the first student prediction result, and the first teacher prediction result; use the generator loss value to adjust the generator parameters to obtain an updated generator trained in the current iteration round;

[0009] Step 104: in the current iteration round, obtain a second synthetic sample generated by the update generator under the specified generation information;

[0010] Step 105: input the second synthetic sample into the initial student model and the pre-trained teacher model respectively, output the second student prediction result obtained after the initial student model processes the second synthetic sample, and process the second synthetic sample based on the teacher model provided with a random deactivation layer, and output the second teacher prediction result;

[0011] Step 106: determine at least one student model loss weight based on the current number of iterations, and calculate the student model loss value according to the student model loss weight, the second student prediction result, and the second teacher prediction result; use the student model loss value to adjust the student model parameters to obtain an updated student model trained in the current iteration round; wherein the generator loss weight and the student model loss weight both show an overall dynamic increase trend with the increase in the number of iterations;

[0012] Step 107: If the number of iterations is less than the preset maximum iteration threshold, repeat steps 101 to 107; otherwise, determine that the updated student model obtained in the last iteration round is the target student model.

[0013] In some embodiments, the first teacher prediction result includes a first standard prediction result and a plurality of first random inactivation prediction results; the second teacher prediction result includes a plurality of second random inactivation prediction results;

[0014] The first synthetic sample is processed based on the teacher model provided with a random deactivation layer, and a first teacher prediction result is output, including:

[0015] Under the condition of closing the random inactivation layer, the output teacher model processes the first synthetic sample to obtain the first standard prediction result;

[0016] Under the condition of turning on the random dropout layer, outputting a plurality of first random dropout prediction results obtained by the teacher model processing the first synthetic sample multiple times;

[0017] The second synthetic sample is processed based on the teacher model provided with a random inactivation layer, and a second teacher prediction result is output, including:

[0018] Under the condition of turning on the random deactivation layer, the output teacher model processes the second synthetic sample multiple times and obtains multiple second random deactivation prediction results accordingly.

[0019] In some embodiments, the generator loss weight is determined according to the following steps, comprising:

[0020] When the number of iterations is within a preset first interval, determining the generator loss weight to be a preset first weight value;

[0021] When the number of iterations is within a preset second interval, determining the generator loss weight as the product of the preset second weight value and the number of iterations;

[0022] When the number of iterations is within a preset third interval, the generator loss weight is determined to be a preset third weight value.

[0023] In some embodiments, the generator loss weights include a first generator loss weight and a second generator loss weight;

[0024] According to the generator loss weight, the first student prediction result and the first teacher prediction result, the generator loss value is calculated, including:

[0025] Calculate a first difference value representing the degree of difference between the first student prediction result and the first standard prediction result, and use the product of the first generator loss weight and the first difference value as the first loss term;

[0026] Determining a second loss term based on the second generator loss weight and the plurality of first random dropout prediction results;

[0027] Based on the cross entropy loss between the first standard prediction results and the generated information, the third loss term is obtained;

[0028] The first loss term, the second loss term, and the third loss term are superimposed to obtain the generator loss value.

[0029] In some embodiments, the student model loss weight is determined according to the following steps, comprising:

[0030] Calculate a second difference value representing the degree of difference between the second student prediction result and the plurality of second random inactivation prediction results;

[0031] If the second difference value is less than the difference threshold that changes dynamically with the number of iterations, the preset fourth weight value is determined to be the student model loss weight; otherwise, the preset fifth weight value is determined to be the student model loss weight.

[0032] In some embodiments, the student model loss value is calculated according to the student model loss weight, the second student prediction result and the second teacher prediction result, including:

[0033] The product of the student model loss weight and the second difference value is used as the fourth loss term;

[0034] Update the student model loss weight based on the preset hyperparameters to obtain the fifth loss term;

[0035] Superimpose the fourth loss term and the fifth loss term to obtain the student model loss value.

[0036] In some embodiments, after adjusting the generator parameters using the generator loss value, the method further includes:

[0037] Get the first training step number of the initial generator in the current iteration round;

[0038] If the first training step number does not reach the preset first training threshold, repeat steps 101 to 103, otherwise obtain the updated generator trained in the current iteration round;

[0039] After adjusting the student model parameters using the student model loss value, it also includes:

[0040] Get the second training step number of the initial student model in the current iteration round;

[0041] If the second training step number does not reach the preset second training threshold, repeat steps 104 to 106, otherwise obtain the updated student model trained in the current iteration round.

[0042] In some embodiments, the generator loss value is used to adjust the generator parameters to obtain an updated generator trained in the current iteration round, including:

[0043] Based on the preset first learning rate and generator loss value, the generator parameters are gradient updated to obtain updated generator parameters;

[0044] Based on the updated generator parameters, the updated generator trained in the current iteration round is obtained;

[0045] The student model loss value is used to adjust the student model parameters to obtain the updated student model trained in the current iteration round, including:

[0046] Based on the preset second learning rate and the student model loss value, the student model parameters are gradient updated to obtain updated student model parameters;

[0047] Based on the updated student model parameters, the updated student model trained in the current iteration round is obtained.

[0048] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a data-free knowledge distillation learning device, the device comprising:

[0049] A first acquisition module, used to execute step 101, obtain the number of iterations of the current iteration round, and obtain the first synthetic sample generated by the initial generator under the specified generation information;

[0050] A first processing module is used to execute step 102, input the first synthetic sample into the initial student model and the pre-trained teacher model respectively, output a first student prediction result obtained after the initial student model processes the first synthetic sample, and process the first synthetic sample based on the teacher model provided with a random deactivation layer, and output a first teacher prediction result;

[0051] A generator training module is used to execute step 103, determine at least one generator loss weight based on the current number of iterations, calculate the generator loss value according to the generator loss weight, the first student prediction result and the first teacher prediction result; use the generator loss value to adjust the generator parameters to obtain an updated generator trained in the current iteration round;

[0052] A second acquisition module, configured to execute step 104, and acquire, in the current iteration round, a second synthetic sample generated by the update generator under the specified generation information;

[0053] A second processing module is used to execute step 105, input the second synthetic sample into the initial student model and the pre-trained teacher model respectively, output the second student prediction result obtained after the initial student model processes the second synthetic sample, and process the second synthetic sample based on the teacher model provided with a random deactivation layer, and output the second teacher prediction result;

[0054] The student model training module is used to execute step 106, determine at least one student model loss weight based on the current number of iterations, calculate the student model loss value according to the student model loss weight, the second student prediction result and the second teacher prediction result; adjust the student model parameters using the student model loss value to obtain an updated student model trained in the current iteration round;

[0055] The target generation module is used to repeat steps 101 to 107 if the number of iterations is less than a preset maximum iteration threshold; otherwise, determine that the updated student model obtained in the last iteration round is the target student model.

[0056] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method of the above-mentioned first aspect when executing the computer program.

[0057] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of the first aspect is implemented.

[0058] The data-free knowledge distillation learning method and device, electronic device and storage medium proposed in the present application, which, through step 101, obtains the number of iterations of the current iteration round, and obtains the first synthetic sample generated by the initial generator under the specified generation information; step 102, inputs the first synthetic sample into the initial student model and the pre-trained teacher model respectively, outputs the first student prediction result obtained after the initial student model processes the first synthetic sample, and processes the first synthetic sample based on the teacher model provided with a random deactivation layer, and outputs the first teacher prediction result; step 103, determines at least one generator loss weight based on the current number of iterations, and calculates the generator loss value according to the generator loss weight, the first student prediction result and the first teacher prediction result; uses the generator loss value to adjust the generator parameters to obtain the updated generator trained under the current iteration round; step 104, in the current iteration round, obtains the second Synthesize samples; Step 105, input the second synthesized samples into the initial student model and the pre-trained teacher model respectively, output the second student prediction result obtained after the initial student model processes the second synthesized samples, and process the second synthesized samples based on the teacher model with a random deactivation layer, and output the second teacher prediction result; Step 106, determine at least one student model loss weight based on the current number of iterations, and calculate the student model loss value according to the student model loss weight, the second student prediction result and the second teacher prediction result; use the student model loss value to adjust the student model parameters to obtain the updated student model trained in the current iteration round; wherein, the generator loss weight and the student model loss weight both show an overall dynamic increase trend with the increase in the number of iterations; Step 107, if the number of iterations is less than the preset maximum iteration threshold, repeat steps 101 to 107; otherwise, determine that the updated student model obtained in the last iteration round is the target student model. In this way, by adjusting the opening of the random inactivation layer to measure the uncertainty of the teacher model, the difficulty of the generator to generate synthetic samples and the learning progress of the student model are controlled, thereby helping the student model to converge faster and improving the training efficiency of the student model in data-free knowledge distillation learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is an optional flowchart of the data-free knowledge distillation learning method provided in the embodiment of the present application;

[0060] Figure 2 It is an optional distillation process schematic diagram of the data-free knowledge distillation learning method provided in the embodiment of the present application;

[0061] Figure 3 It is an optional overall model training schematic diagram of the data-free knowledge distillation learning method provided in the embodiment of the present application;

[0062] Figure 4 yes Figure 1 A flowchart of an implementation of step 102 in FIG.

[0063] Figure 5 yes Figure 1 An implementation process of step 103 in;

[0064] Figure 6 It is an optional generator loss weight determination flow chart of the data-free knowledge distillation learning method provided in the embodiment of the present application;

[0065] Figure 7 It is an optional generator parameter update adjustment flow chart of the data-free knowledge distillation learning method provided in the embodiment of the present application;

[0066] Figure 8 It is an optional generator training flow chart of the data-free knowledge distillation learning method provided in the embodiment of the present application;

[0067] Fig. 9 yes Figure 1 A flowchart of an implementation of step 106 in FIG.

[0068] Fig.10 It is an optional student model loss weight determination flow chart of the data-free knowledge distillation learning method provided in an embodiment of the present application;

[0069] Fig.11 It is an optional student model parameter update and adjustment flow chart of the data-free knowledge distillation learning method provided in the embodiment of the present application;

[0070] Fig.12 yes Figure 1 An implementation flow chart after step 106 in FIG. 1 ;

[0071] Fig.13 It is an optional student model training flow chart of the data-free knowledge distillation learning method provided in the embodiment of the present application;

[0072] Fig.14 This is an optional flow chart of the data-free knowledge distillation learning device provided in an embodiment of the present application;

[0073] Fig.15 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0075] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0077] Knowledge distillation aims to refine and "instill" the knowledge and experience of a complex and large model (usually called the teacher model) into a smaller and simpler model (usually called the student model). This process not only retains the predictive power and accuracy of the teacher model, but also significantly improves the operational efficiency and computing performance of the student model, enabling it to perform well even in resource-constrained environments.

[0078] Among them, data-free knowledge distillation is a special type of knowledge distillation. Its core feature is to use the trained model (teacher model) to discover and refine knowledge without accessing the original training data, and transfer this knowledge to another lightweight model (student model). In this way, it can protect data privacy while realizing knowledge migration and reuse and model compression.

[0079] In the related technology, a generator is used to generate synthetic samples with similar distribution to the teacher model training data to train the student model. However, the traditional data-free knowledge distillation method does not take into account the impact of synthetic samples of different difficulty levels on the learning effect of the student model, resulting in low efficiency in student model training.

[0080] Based on this, the embodiments of the present application propose a data-free knowledge distillation learning method, device, equipment and storage medium, aiming to improve the training efficiency of the student model in data-free knowledge distillation learning.

[0081] It should be noted that in the embodiments of the present application, when it comes to information related to user characteristics such as user basic information or user identity, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, although the synthetic samples are not generated directly from the original data, they may still contain features or patterns similar to the original data. If these features or patterns are obvious enough, malicious people may still be able to use them to infer certain information of the original data. Therefore, when the embodiments of the present application need to obtain information related to the original data, the user's separate permission or consent will be obtained first. After the user's separate permission or consent is clearly obtained, the necessary data for the normal operation of the embodiments of the present application will be obtained. For example, before obtaining the synthetic samples generated by the generator under the specified generation information, the embodiments of the present application will first obtain the user's authorization or consent, otherwise it cannot be used in the synthetic samples of the embodiments of the present application. In addition, the other relevant data obtained by the data-free knowledge distillation learning device of the present application are all legal data, which will not be repeated here.

[0082] In the embodiment of the present application, the description will be made from the dimension of a data-free knowledge distillation learning device (for the convenience of description, it may also be referred to as a "distillation learning device" or "distillation device" below), which can be integrated into a computer device, such as a server. Figure 1 As shown, Figure 1 is an optional flowchart of the data-free knowledge distillation learning method provided in an embodiment of the present application (for the convenience of description, it may also be referred to as “distillation learning method” or “distillation method” hereinafter). Figure 1 The method may include but is not limited to the following steps 101 to 107. When the distillation device performs the prediction method, the specific process is as follows. It should be noted that this embodiment Figure 1 The order of step 101 to step 107 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0083] Step 101, obtain the iteration number of the current iteration round, and obtain the first synthetic sample generated by the initial generator under the specified generation information.

[0084] The following is a detailed description of step 101.

[0085] Among them, the goal of the generator is to generate new data instances that are as close to the real data distribution as possible. It receives random noise as input and converts it into data points (such as images, audio, etc.), and the converted data points should be indistinguishable from real data. The generator in the embodiment of the present application can be a standard generative adversarial network (GAN) generator, a cyclic GAN generator, a deep convolutional GAN ​​generator, etc. In actual use, a suitable generator can be selected according to the specific circumstances. The embodiment of the present application does not limit the selected generator. In the embodiment of the present application, the generator that has not been trained in the current iteration round is called the initial generator, and the student model that has not been trained is called the initial student model; if there is no special explanation in the future, the generator refers to the initial generator, and the student model refers to the initial student model.

[0086] The first synthetic sample may be one or more. When the number of the first synthetic samples is more than one, the multiple first synthetic samples will form a synthetic sample set and be input into the student model and the teacher model respectively. That is, this is only for the convenience of explanation, and does not specifically refer to the specific number of the first synthetic samples.

[0087] like Figure 2 As shown, Figure 2 This is an optional distillation process diagram of the data-free knowledge distillation learning method provided in an embodiment of the present application, wherein the generator, the pre-trained teacher model, and the student model to be trained constitute the data-free knowledge distillation learning architecture (referred to as the total model). In scenarios where data is scarce or data privacy is sensitive, the generator can generate synthetic samples that meet the requirements of data-free knowledge distillation learning under the constraints of specified generation information, thereby avoiding the exposure of implicit data and helping the student model to perform efficient distillation learning by autonomously generating sample data when data is scarce. For example, Figure 2 The generator G shown is a category generator, which can generate synthetic samples based on preset generation information. Figure 2 The generation information in the example is used to instruct the generator to output a digital image.

[0088] Among them, the teacher model is a well-trained and well-performing model, which is usually a larger and more complex model to achieve a higher performance level on a specific task. Usually, the teacher model is trained based on a pre-set privacy data set, and the teacher model has the characteristics of high performance, high complexity, and rich knowledge. Exemplarily, the teacher model can be a deep convolutional neural network (CNNs), a recurrent neural network (RNNs), a neural network model based on a self-attention mechanism, etc.

[0089] Among them, the student model is a smaller, simpler model, and the student model is intended to learn and imitate its behavior or prediction results from another more complex and better performing model (i.e., the teacher model). Exemplarily, the student model can be a simplified CNNs model, a simplified RNNs model, etc. It should be noted that the teacher model and the student model used in the embodiments of the present application can be adaptively adjusted according to actual conditions, which are only examples and not mandatory constraints.

[0090] Furthermore, after completing the training architecture, initialize the generator parameters, such as Figure 2 As shown, the generator G includes two inputs: a d-dimensional vector z obeys a standard normal distribution z~N(0,I) and generation information Y, so that the generator can output synthetic samples (including a first synthetic sample and a second synthetic sample); initialize the relevant parameters of the teacher model and the student model, wherein the number of parameters of the student model is much smaller than the number of parameters of the teacher model, so as to achieve the purpose of reducing storage space and reducing the amount of inference calculation during data processing; initialize the number of iterations, and the current number of iterations after initialization c=0.

[0091] like Figure 3 As shown, Figure 3 This is an optional overall model training schematic diagram of the data-free knowledge distillation learning method provided in the embodiment of the present application. In order to obtain the final target student model, the training process of the overall model in the embodiment of the present application includes:

[0092] ① Overall model preparation and parameter initialization;

[0093] ② Train the initial generator to obtain the updated generator;

[0094] ③ Use the update generator to train the initial student model to obtain the updated student model;

[0095] ④When the number of iterative training reaches the preset maximum iteration threshold, the updated student model obtained in the last iteration round is determined as the target student model; otherwise, repeat steps ② and ③.

[0096] First, the initial generator is trained to obtain the updated generator.

[0097] Step 102: input the first synthetic sample into the initial student model and the pre-trained teacher model respectively, output the first student prediction result obtained after the initial student model processes the first synthetic sample, and process the first synthetic sample based on the teacher model provided with a random deactivation layer, and output the first teacher prediction result.

[0098] The following is a detailed description of step 102.

[0099] like Figure 2As shown, in the current iteration round, bs d-dimensional z vectors are randomly generated to construct the matrix Z = (z1; z2; ...; z bs ) and generate information Y = (y1; y2; ...; y bs ) is input into the generator G to obtain bs generated images (synthetic samples), where the dimension dim of each synthetic sample is h ×dim w The pixel matrix, dim h Represents the row dimension of the matrix, dim w Represents the column dimension of the matrix; the first synthetic sample

[0100] Furthermore, a random deactivation layer (Dropout layer) is provided in the teacher model proposed in the embodiment of the present application. The Dropout layer randomly closes some neurons during the training process to improve the generalization ability of the model. The related technology usually chooses to close the Dropout of the model during the reasoning process. In the example of the present application, the Dropout layer is turned on during the reasoning process, and the Monte Carlo Dropout technology is used to estimate the uncertainty of the model for the prediction results. Specifically, Monte Carlo Dropout is a method of estimating the uncertainty of the model using the Dropout technology. In the test phase, multiple predictions are performed by turning on the Dropout layer, and then the multiple prediction results are averaged to measure the uncertainty of the model.

[0101] Furthermore, the first synthetic sample generated by the generator will be input into the initial student model to be trained and the trained teacher model respectively, so that the student model and the teacher model can process the input first synthetic sample according to the given task and output the processing result. Figure 2 In this example, the given task is an image content recognition task. The student model S to be trained and the pre-trained teacher model T will process the first synthetic sample input respectively. Then, the student model S outputs the first student prediction result corresponding to the first synthetic sample. in, Including multiple components

[0102] In some embodiments, Figure 4 yes Figure 1 A flowchart for implementing step 102 in the embodiment of the present invention, processing a first synthetic sample based on a teacher model provided with a random deactivation layer, and outputting a first teacher prediction result, includes the following steps 201 to 202:

[0103] Step 201, under the condition of turning off the random deactivation layer, the output teacher model processes the first synthetic sample to obtain a first standard prediction result.

[0104] Step 202, under the condition of turning on the random dropout layer, output a plurality of first random dropout prediction results obtained by the teacher model performing multiple processing on the first synthetic sample.

[0105] In some embodiments, when the student model processes the synthetic sample, the teacher model will also synchronously process the synthetic sample, and the teacher model outputs the first teacher prediction result under the condition of turning on the random deactivation layer. Since the student model implements distillation learning based on the student prediction results output by the student model itself and the teacher prediction results output by the teacher model, the embodiment of the present application adjusts the opening of the random deactivation layer to measure the uncertainty of the teacher model, thereby controlling the difficulty of the generator to generate synthetic samples and the learning progress of the student model, so as to help the student model converge faster.

[0106] Specifically, under the condition of closing the Dropout layer of the teacher model, the standard prediction results obtained by the teacher model after multiple processing of the synthetic samples are output. in, Including multiple components i=1, 2...n; at the same time, under the condition of opening the Dropout layer of the teacher model, the output teacher model processes the synthetic samples multiple times and obtains multiple random dropout prediction results, such as Figure 2 In this example, the teacher model is set to process the synthetic sample K times when the Dropout layer of the teacher model is turned on, and K random dropout prediction results are obtained. Next, we average multiple random dropout prediction results to obtain It is used to measure the uncertainty of the teacher model. It should be noted that the embodiment of the present application does not impose any specific restrictions on the value of K, which can be adaptively adjusted according to actual conditions.

[0107] For example, when the given task is an image content recognition task, the standard prediction result and the random dropout prediction result both represent the output result of predicting the input synthetic sample, that is, the obtained image category probability Among them, x is a synthetic sample, The class that the teacher model predicts the image belongs to.

[0108] Step 103, determine at least one generator loss weight based on the current number of iterations, calculate the generator loss value according to the generator loss weight, the first student prediction result and the first teacher prediction result; use the generator loss value to adjust the generator parameters to obtain the updated generator trained in the current iteration round.

[0109] The following is a detailed description of step 103.

[0110] In some embodiments, the generator loss weight will gradually increase as the current number of iterations increases. In this way, the generator can gradually learn more about data distribution characteristics during the training process, so that the generated synthetic samples are closer to the real data; based on the increasing generator loss weight, the generator loss value can reflect the current learning progress and performance changes of the generator from easy to difficult in different iteration rounds, thereby achieving gradual optimization of the generator.

[0111] In some embodiments, Figure 5 yes Figure 1 A flowchart for implementing step 103 in FIG. 104 , wherein the generator loss value is calculated according to the generator loss weight, the first student prediction result and the first teacher prediction result, including the following steps 301 to 304:

[0112] Step 301, calculate a first difference value representing the degree of difference between the first student prediction result and the first standard prediction result, and use the product of the first generator loss weight and the first difference value as the first loss term.

[0113] Step 302: Determine a second loss term based on a second generator loss weight and a plurality of first random loss prediction results.

[0114] Step 303, obtaining a third loss term based on the cross entropy loss between the first standard prediction result and the generated information.

[0115] Step 304, superimpose the first loss term, the second loss term and the third loss term to obtain the generator loss value.

[0116] Steps 301 to 304 are described in detail below.

[0117] In some embodiments, the generator has an independent first training step in each iteration round. Specifically, each time an iteration round is entered, the first training step of the generator is initialized to obtain Stepg=0; then, through the following formula <1> Calculate the generator loss value:

[0118] Loss g =-α(c)Loss adv +β(c)Loss u +Loss c <1>

[0119] Among them, α(c) is the loss weight value of the first generator, Loss adv is the first difference value, -α(c)Loss adv is the first loss term; β(c) is the loss weight value of the second generator, β(c)Loss u is the second loss term;c The third loss item.

[0120] Among them, Loss adv It measures the difference between the prediction results of the student model and the teacher model. adv The smaller the loss is, the closer the prediction results of the teacher model and the student model for the same synthetic sample are. Since the synthetic sample generated by the generator has little effect on the parameter update of the student model in the next iteration round, when executing the distillation method, the embodiment of the present application expects that the difference between the synthetic sample generated by the generator and the prediction results output by the teacher model and the student model is as large as possible; Loss u It measures the uncertainty of the teacher model; Loss c It measures the degree of difference between the standard prediction results and the generated information.

[0121] Among them, α(c) is Loss adv The corresponding generator loss weight, α(c), can be determined by following steps 401 to 403:

[0122] In some embodiments, Figure 6 As shown, Figure 6 This is an optional generator loss weight determination flow chart of the data-free knowledge distillation learning method provided in the embodiment of the present application. For different items of the generator loss value, the corresponding generator loss weight can be determined according to the following steps 401 to 403:

[0123] Step 401, when the number of iterations is in a preset first interval, the generator loss weight is determined to be a preset first weight value.

[0124] Step 402, when the number of iterations is in a preset second interval, the generator loss weight is determined to be the product of the preset second weight value and the number of iterations.

[0125] Step 403, when the number of iterations is in a preset third interval, the generator loss weight is determined to be a preset third weight value.

[0126] Steps 401 to 403 are described in detail below.

[0127] In some embodiments, for the formula <1> The first generator loss weight α(c) in has the following update strategy:

[0128]

[0129] Where c is the current number of iterations; when c falls within the preset first interval (0, c α1 ), c is the preset first weight value, that is, c=0; when c falls within the preset second interval [c α1 , cα2 ], c is the preset second weight value, that is, c = αc; when c falls within the preset third interval (c α2 , c max ], c is the preset third weight value, that is, c = α max ;c α1 、c α2 , α, α max are all pre-set hyperparameters, c max The preset maximum number of iterations.

[0130] It is understandable that when 0 <c<c α1 When α(c) is set to 0, the student model has a low performance and may not be able to learn the knowledge of the teacher model. This allows the generator to focus on generating synthetic samples under the specified generation information, thereby preventing the generator from being disturbed by adversarial training and helping the generator to quickly learn basic image structures and category features. α1 ≤c≤c α2 When adv In Los g The weight of , so as to smoothly guide the generator from simple synthetic sample generation to more complex adversarial generation, thereby improving the quality of the generated synthetic samples; when c α2 <c≤c max , the generator already has a certain generation capability. At this time, by fixing Loss adv The weights ensure that the generator continues to optimize in the adversarial environment to generate more refined and realistic synthetic samples, which helps stabilize the training process and avoid performance fluctuations caused by over-adjustment.

[0131] In some embodiments, for the formula <1> The second generator loss weight β(c) in has the following update strategy:

[0132]

[0133] Where c is the current number of iterations; when c falls within the preset first interval (0, c β1 ), c is the preset first weight value, that is, c=0; when c falls within the preset second interval [c β1 , c β2 ], c is the preset second weight value, that is, c = βc; when c falls within the preset third interval (c β2 , c max ], c is the preset third weight value, that is, c = α max ;c β1 、c β2 , β, α max are all pre-set hyperparameters, c max The preset maximum number of iterations.

[0134] When 0 <c<c β1 When β(c) is low, the generator may not be able to generate high-quality synthetic samples. Setting β(c) to 0 can make the generator focus on generating synthetic samples under the specified generation information, thereby preventing the generator from being disturbed by adversarial training and helping the generator to quickly learn basic image structures and category features. β1 ≤c≤c β2 When u In Los g The weight of is used to smoothly guide the generator from generating simple synthetic samples to samples that are more difficult for the teacher model to distinguish, thereby improving the quality of the generated synthetic samples. α2 <c≤c max , the generator already has a certain generation capability. At this time, by fixing Loss u The weights ensure that the generator continues to optimize in the adversarial environment to generate more refined and realistic synthetic samples, which helps stabilize the training process and avoid performance fluctuations caused by over-adjustment.

[0135] Furthermore, both the first generator loss weight and the second generator loss weight show an overall dynamic increase trend with the increase in the number of iterations. And it should be noted that when the generator loss value includes more other loss items, and the loss item also includes the generator loss weight, the specific value of the generator loss weight changing with the number of iterations can also be determined according to steps 401 to 403, so as to achieve the training of the generator from easy to difficult.

[0136] Next, continue with the explanation <1> ,Mode <1> Loss adv Through the following formula <2> The calculation results are:

[0137]

[0138] in, Representative The i-th standard prediction result component in , Representative The i-th student prediction result component in ; bs represents the bs synthetic samples generated by the generator; KL is the information gain or relative entropy (Kullback-Leibler Divergence, KL divergence), which is an asymmetric measurement method used to measure the difference between two probability theoretical distributions and the true distribution.

[0139] Further, the formula <1> Loss u Through the following formula <3> The calculation results are:

[0140]

[0141] in, The mean is a measure of the central tendency of a group of values, which is the sum of all the values ​​added together divided by the number of values.

[0142] Further, the formula <1> Loss c Through the following formula <4> The calculation results are:

[0143]

[0144] Among them, CE(·) is used to characterize the cross entropy loss function, y ij is the jth element of the category one-hot encoding of the i-th synthetic sample, is the probability that the i-th synthetic sample belongs to the j-th generated information, that is, The j-th element of .

[0145] Furthermore, if Figure 7 As shown, Figure 7 This is an optional generator parameter update adjustment flow chart of the data-free knowledge distillation learning method provided in an embodiment of the present application. The generator parameters are adjusted using the generator loss value to obtain an updated generator trained in the current iteration round, including the following steps 701 to 702:

[0146] Step 701, based on the preset first learning rate and generator loss value, the generator parameters are gradient updated to obtain updated generator parameters.

[0147] Step 702, based on the updated generator parameters, obtain the updated generator trained in the current iteration round.

[0148] Step 701 to step 702 are described below.

[0149] In some embodiments, based on the following formula <5> Implement the generator parameter θ g Update:

[0150]

[0151] Among them, ε is the first learning rate, The generator loss value Loss calculated by the loss function g For the generator parameter θ g gradient.

[0152] In some embodiments, after the generator parameters are adjusted using the generator loss value, the following steps 801 to 802 are also included:

[0153] Step 801, obtain the first training step number of the initial generator in the current iteration round.

[0154] Step 802: If the first training step number does not reach the preset first training threshold, repeat steps 101 to 103; otherwise, obtain the updated generator trained in the current iteration round.

[0155] In some embodiments, in each round of iterative training, the initial generator is trained multiple times to obtain an updated generator trained in the current iterative round. Figure 8 As shown, Figure 8 This is an optional generator training flowchart of the data-free knowledge distillation learning method provided in the embodiment of the present application, which specifically includes the following training steps:

[0156] (1.1) Initialize the first training step of the generator at the current number of iterations Step g =0;

[0157] (1.2) generating a first synthetic sample based on preset generation information;

[0158] (1.3) Input the first synthetic sample into the initial student model to obtain the first student prediction result in, Including multiple components

[0159] (1.4) Close the Dropout layer of the teacher model, input the first synthetic sample into the teacher model, and obtain the first standard prediction result in, Including multiple components

[0160] (1.5) Turn on the Dropout layer of the teacher model, input the first synthetic sample into the teacher model, perform K predictions, and obtain K first random dropout prediction results.

[0161] (1.6) Based on the K first random dropout prediction results First standard prediction results And the first student prediction results Calculate the generator loss value Loss g ;

[0162] (1.7) Based on the generator loss value Loss g Update the generator parameters θ g ;

[0163] (1.8) Update the first training step Step g =Stepg +1;

[0164] (1.9) If Step g <Step gmax Then repeat steps (1.2) to (1.9), otherwise terminate the generator training in the current iteration round and obtain the updated generator; where Step gmax is the preset first training threshold, Step gmax Adaptive adjustments can be made according to actual conditions, and the embodiments of the present application do not limit this.

[0165] Next, we introduce how to train the initial student model based on the update generator to obtain the updated student model.

[0166] Step 104: in the current iteration round, obtain a second synthetic sample generated by the update generator under the specified generation information;

[0167] Step 105: input the second synthetic sample into the initial student model and the pre-trained teacher model respectively, output the second student prediction result obtained after the initial student model processes the second synthetic sample, and process the second synthetic sample based on the teacher model with a random deactivation layer, and output the second teacher prediction result.

[0168] Among them, the specific implementation methods of step 104 are similar to those of step 101, and the specific implementation methods of step 105 are basically the same as those of step 102. The difference is that, in order to obtain an updated student model, step 105 only outputs multiple second random deactivation prediction results obtained by the teacher model performing multiple processing on the second synthetic sample under the condition of turning on the random deactivation layer, and there is no need to output the prediction results when turning off the random deactivation layer.

[0169] Step 106: determine at least one student model loss weight based on the current number of iterations, and calculate the student model loss value according to the student model loss weight, the second student prediction result, and the second teacher prediction result; use the student model loss value to adjust the student model parameters to obtain an updated student model trained in the current iteration round; wherein the generator loss weight and the student model loss weight both show an overall dynamic increase trend with the increase in the number of iterations;

[0170] Step 106 is described in detail below.

[0171] In some embodiments, the student model loss weight will gradually increase as the current number of iterations increases. In this way, the student model can gradually learn more prediction result features during the training process, thereby making the generated student prediction results more accurate; based on the increasing student model loss weight, the student model loss value can reflect the current learning progress and performance changes of the student model from easy to difficult in different iteration rounds, thereby achieving gradual optimization of the student model.

[0172] In some embodiments, Fig. 9 As shown, Fig. 9 yes Figure 1 A flowchart for implementing step 106 in FIG. 100 , wherein the student model loss value is calculated according to the student model loss weight, the student prediction result and the teacher prediction result, including the following steps 501 to 503:

[0173] Step 501, taking the product of the student model loss weight and the second difference value as the fourth loss term;

[0174] Step 502, updating the student model loss weight based on the preset hyperparameters to obtain a fifth loss item;

[0175] Step 503, superimpose the fourth loss term and the fifth loss term to obtain the student model loss value.

[0176] Steps 501 to 503 are described in detail below.

[0177] In some embodiments, by the formula <6> Calculate the student model loss value:

[0178] Loss s =v(γ,L)Loss KL +g(γ,v) <6>

[0179] Among them, v(γ,L)Loss KL is the fourth loss term, g(γ,v) is the fifth loss term; v(γ,L) is the loss weight value of the student model. v(γ,L) is the loss KL The corresponding student model loss value is Fig.10 As shown, Fig.10 It is an optional student model loss weight determination flow chart of the data-free knowledge distillation learning method provided in the embodiment of the present application, that is, v(γ,L) can be determined by following steps 601 to 602:

[0180] Step 601, calculating a second difference value representing the degree of difference between the second student prediction result and a plurality of second random inactivation prediction results;

[0181] Step 602, if the second difference value is less than the difference threshold that changes dynamically with the number of iterations, determine the preset fourth weight value as the student model loss weight, otherwise, determine the preset fifth weight value as the student model loss weight.

[0182] Steps 601 to 602 are described in detail below.

[0183] In some embodiments, the formula <6> Loss KL Through the following formula <7> Calculate the second difference value Loss KL :

[0184]

[0185] in, yes For the prediction result of the i-th synthetic sample, by taking the average of multiple random inactivation prediction results of the teacher model as the learning target of the student model, the introduction of noise caused by the large uncertainty of the teacher model can be reduced, thereby controlling the learning progress of the student model; Formula <7> Explanation and formula of other parameters <2> The same, no further description here.

[0186] Furthermore, for the formula <6> The student model loss weight value v(γ,L) in has the following update strategy:

[0187]

[0188] Wherein, 1 is the preset fourth weight value, and 0 is the preset fifth weight value; the difference threshold γ is calculated by the following formula: <8> Update

[0189] γ(c)=γ0+ρc <8>

[0190] Among them, γ(c) is the linear model of the number of iterations; c represents the current iteration round; γ0 is the preset initial value; ρ is the preset growth rate; it should be noted that the preset parameters such as the fourth weight value, the fifth weight value, the growth rate, etc. can be adaptively adjusted according to actual conditions, and the embodiments of the present application do not limit the specific values ​​of these parameters.

[0191] Next, continue with the explanation <6> ,Mode <6> g(γ,v) in the above formula is the regularization term in course learning. One feasible implementation method is v i Indicates the corresponding student model loss weight for the i-th synthetic sample

[0192] In some embodiments, Fig.11 As shown, Fig.11This is an optional student model parameter update adjustment flow chart of the data-free knowledge distillation learning method provided in an embodiment of the present application. The student model parameters are adjusted using the student model loss value to obtain an updated student model trained in the current iteration round, including the following steps 703 to 704:

[0193] Step 703, based on the preset second learning rate and the student model loss value, gradient update the student model parameters to obtain updated student model parameters;

[0194] Step 704, based on the updated student model parameters, obtain the updated student model trained in the current iteration round.

[0195] Steps 703 to 704 are described in detail below.

[0196] In some embodiments, based on the following formula <9> Implement the student model parameters θ s Update:

[0197]

[0198] Among them, ε is the second learning rate, The student model loss value Loss calculated by the loss function s For the student model parameter θ s gradient.

[0199] In some embodiments, Fig.12 As shown, Fig.12 yes Figure 1 An implementation flow chart after step 106 in the embodiment further includes the following steps 803 to 804:

[0200] Step 803, obtaining the second training step number of the initial student model in the current iteration round;

[0201] Step 804, if the second training step number does not reach the preset second training threshold, repeat steps 104 to 106, otherwise obtain the updated student model trained in the current iteration round.

[0202] Steps 803 to 804 are described in detail below.

[0203] In some embodiments, in each round of iterative training, the initial student model is trained multiple times to obtain an updated student model trained in the current iterative round. Fig.13 As shown, Fig.13 This is an optional student model training flowchart of the data-free knowledge distillation learning method provided in the embodiment of the present application, which specifically includes the following training steps:

[0204] (2.1) At the current number of iterations, initialize the second training step of the student model Step s =0;

[0205] (2.2) generating a second synthetic sample based on preset generation information;

[0206] (2.3) Input the second synthetic sample into the initial student model to obtain the second student prediction result in, Including multiple components

[0207] (2.4) Turn on the Dropout layer of the teacher model, input the second synthetic sample into the teacher model, perform K predictions, and obtain K second random dropout prediction results.

[0208] (2.5) Based on the K second random dropout prediction results And the second student's prediction results Calculate the student model loss value Loss s ;

[0209] (2.6) Based on the student model loss value Loss s Update the student model parameters θ s ;

[0210] (2.7) Update the second training step Step s =Step s +1;

[0211] (2.8) If Step s <Step smax Then repeat steps (2.2) to (2.8), otherwise end the student model training in the current iteration round and obtain the updated student model; where Step smax is the preset second training threshold, Step smax Adaptive adjustments can be made according to actual conditions, and the embodiments of the present application do not limit this.

[0212] Step 107: If the number of iterations is less than the preset maximum iteration threshold, repeat steps 101 to 107; otherwise, determine that the updated student model obtained in the last iteration round is the target student model.

[0213] In some embodiments, after the generator completes Step gmax Training, and the student model completes Step smaxAfter training, end the training of the total model in the current iteration round; then, update the number of iterations c=c+1; determine whether the updated number of iterations reaches the maximum iteration threshold, if not, repeat steps 101 to 107; otherwise, determine that the updated student model obtained when the maximum iteration threshold is reached is the target student model.

[0214] Among them, the maximum iteration threshold can be adaptively adjusted according to actual conditions, and the embodiments of the present application do not limit this.

[0215] like Fig.14 As shown, Fig.14 901 is an optional flowchart of a data-free knowledge distillation learning device provided in an embodiment of the present application. The data-free knowledge distillation learning device includes the following modules 901 to 907:

[0216] The first acquisition module 901 is used to execute step 101, obtain the number of iterations of the current iteration round, and obtain the first synthetic sample generated by the initial generator under the specified generation information;

[0217] A first processing module 902 is used to execute step 102, input the first synthetic sample into the initial student model and the pre-trained teacher model respectively, output the first student prediction result obtained after the initial student model processes the first synthetic sample, and process the first synthetic sample based on the teacher model provided with a random deactivation layer, and output the first teacher prediction result;

[0218] The generator training determination module 903 is used to execute step 103, determine at least one generator loss weight based on the current number of iterations, calculate the generator loss value according to the generator loss weight, the first student prediction result and the first teacher prediction result; use the generator loss value to adjust the generator parameters to obtain the updated generator trained in the current iteration round;

[0219] A second acquisition module 904 is used to execute step 104, and acquire, in the current iteration round, a second synthetic sample generated by the update generator under the specified generation information;

[0220] The second processing module 905 is used to execute step 105, input the second synthetic sample into the initial student model and the pre-trained teacher model respectively, output the second student prediction result obtained after the initial student model processes the second synthetic sample, and process the second synthetic sample based on the teacher model provided with a random deactivation layer, and output the second teacher prediction result;

[0221] The student model training module 906 is used to execute step 106, determine at least one student model loss weight based on the current number of iterations, calculate the student model loss value according to the student model loss weight, the second student prediction result and the second teacher prediction result; adjust the student model parameters using the student model loss value to obtain an updated student model trained in the current iteration round;

[0222] The target generation module 907 is used to repeat steps 101 to 107 if the number of iterations is less than a preset maximum iteration threshold; otherwise, determine that the updated student model obtained in the last iteration round is the target student model.

[0223] The data-free knowledge distillation learning method, device, electronic device and storage medium proposed in this application measure the uncertainty of the teacher model by adjusting the opening of the random inactivation layer, thereby controlling the difficulty of the generator to generate synthetic samples and the learning progress of the student model, thereby helping the student model to converge faster and improving the training efficiency of the student model in data-free knowledge distillation learning.

[0224] The specific implementation of the distillation learning device is basically the same as the specific implementation example of the above-mentioned distillation learning method, and will not be repeated here.

[0225] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned data-free knowledge distillation learning method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0226] like Fig.15 As shown, Fig.15 : is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, the electronic device includes:

[0227] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0228] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the data-free knowledge distillation learning method of the embodiment of this application;

[0229] Input / output interface 1003, used to implement information input and output;

[0230] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0231] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0232] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0233] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned data-free knowledge distillation learning method.

[0234] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0235] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0236] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0237] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0238] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0239] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0240] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0241] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0242] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0243] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0244] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0245] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data-free knowledge distillation learning method, characterized in that: The method comprises: Step 101: Obtain the number of iterations of the current iteration round, and obtain the first synthetic sample generated by the initial generator under specified generation information; Step 102: input the first synthetic sample into an initial student model and a pre-trained teacher model respectively, output a first student prediction result obtained after the initial student model processes the first synthetic sample, and process the first synthetic sample based on the teacher model provided with a random deactivation layer, and output a first teacher prediction result; Step 103: determine at least one generator loss weight based on the current number of iterations, and calculate a generator loss value according to the generator loss weight, the first student prediction result, and the first teacher prediction result; use the generator loss value to adjust the generator parameters to obtain an updated generator trained in the current iteration round; Step 104: in the current iteration round, obtain a second synthetic sample generated by the update generator under the specified generation information; Step 105: input the second synthetic sample into the initial student model and the pre-trained teacher model respectively, output the second student prediction result obtained after the initial student model processes the second synthetic sample, and process the second synthetic sample based on the teacher model provided with a random deactivation layer, and output the second teacher prediction result; Step 106: determining at least one student model loss weight based on the current number of iterations, and calculating the student model loss value according to the student model loss weight, the second student prediction result, and the second teacher prediction result; adjusting the student model parameters using the student model loss value to obtain an updated student model trained in the current iteration round; wherein the generator loss weight and the student model loss weight both show an overall dynamic increasing trend with the increase in the number of iterations; Step 107: If the number of iterations is less than a preset maximum iteration threshold, repeat steps 101 to 107; otherwise, determine that the updated student model obtained in the last iteration round is the target student model.

2. The method according to claim 1, characterized in that: The first teacher prediction result includes a first standard prediction result and a plurality of first random inactivation prediction results; the second teacher prediction result includes a plurality of second random inactivation prediction results; The processing of the first synthetic sample based on the teacher model provided with a random deactivation layer and outputting a first teacher prediction result includes: Under the condition of closing the random deactivation layer, outputting the teacher model to process the first synthetic sample to obtain the first standard prediction result; Under the condition that the random deactivation layer is turned on, outputting a plurality of the first random deactivation prediction results obtained by the teacher model processing the first synthetic sample multiple times; The processing of the second synthetic sample based on the teacher model provided with a random deactivation layer and outputting a second teacher prediction result includes: Under the condition that the random deactivation layer is turned on, a plurality of second random deactivation prediction results corresponding to the teacher model performing multiple processing on the second synthetic sample are output.

3. The method according to claim 2, characterized in that The generator loss weight is determined according to the following steps, comprising: When the number of iterations is within a preset first interval, determining the generator loss weight to be a preset first weight value; When the number of iterations is within a preset second interval, determining the generator loss weight as the product of a preset second weight value and the number of iterations; When the number of iterations is within a preset third interval, the generator loss weight is determined to be a preset third weight value.

4. The method according to claim 2, characterized in that: The generator loss weight includes a first generator loss weight and a second generator loss weight; The generator loss value is calculated according to the generator loss weight, the first student prediction result and the first teacher prediction result, including: Calculate a first difference value representing the degree of difference between the first student prediction result and the first standard prediction result, and use the product of the first generator loss weight and the first difference value as a first loss term; determining a second loss term based on the second generator loss weight and a plurality of the first random dropout prediction results; Obtaining a third loss term based on a cross entropy loss between the first standard prediction result and the generated information; The first loss term, the second loss term and the third loss term are superimposed to obtain the generator loss value.

5. The method according to claim 2, characterized in that: The student model loss weight is determined according to the following steps, comprising: Calculate a second difference value representing the degree of difference between the second student prediction result and the plurality of second random inactivation prediction results; If the second difference value is less than the difference threshold that changes dynamically with the number of iterations, the preset fourth weight value is determined to be the student model loss weight; otherwise, the preset fifth weight value is determined to be the student model loss weight.

6. The method according to claim 5, characterized in that The calculating the student model loss value according to the student model loss weight, the second student prediction result and the second teacher prediction result includes: Taking the product of the student model loss weight and the second difference value as a fourth loss term; Update the student model loss weight based on the preset hyperparameter to obtain a fifth loss item; The fourth loss term and the fifth loss term are superimposed to obtain the student model loss value.

7. The method according to claim 2, characterized in that After adjusting the generator parameters using the generator loss value, the method further includes: Obtain the first training step number of the initial generator in the current iteration round; If the first training step number does not reach the preset first training threshold, repeat step 101 to step 103, otherwise obtain the updated generator trained in the current iteration round; After adjusting the student model parameters by using the student model loss value, the method further includes: Obtaining the second training step number of the initial student model in the current iteration round; If the second training step number does not reach the preset second training threshold, repeat step 104 to step 106, otherwise obtain the updated student model trained in the current iteration round.

8. The method according to claim 1, characterized in that The generator loss value is used to adjust the generator parameters to obtain an updated generator trained in the current iteration round, including: Based on a preset first learning rate and the generator loss value, gradient updating is performed on the generator parameters to obtain updated generator parameters; Based on the updated generator parameters, obtaining the updated generator trained in the current iteration round; The method of adjusting the student model parameters by using the student model loss value to obtain an updated student model trained in the current iteration round includes: Based on a preset second learning rate and the student model loss value, gradient updating is performed on the student model parameters to obtain updated student model parameters; Based on the updated student model parameters, the updated student model trained in the current iteration round is obtained.

9. A data-free knowledge distillation learning device, characterized in that: The device comprises: A first acquisition module, used to execute step 101, obtain the number of iterations of the current iteration round, and obtain the first synthetic sample generated by the initial generator under the specified generation information; A first processing module, configured to execute step 102, input the first synthetic sample into an initial student model and a pre-trained teacher model respectively, output a first student prediction result obtained after the initial student model processes the first synthetic sample, and process the first synthetic sample based on the teacher model provided with a random deactivation layer, and output a first teacher prediction result; A generator training module is used to execute step 103, determine at least one generator loss weight based on the current number of iterations, calculate the generator loss value according to the generator loss weight, the first student prediction result and the first teacher prediction result; use the generator loss value to adjust the generator parameters to obtain an updated generator trained in the current iteration round; A second acquisition module, configured to execute step 104, and acquire, in a current iteration round, a second synthetic sample generated by the update generator under specified generation information; A second processing module is used to execute step 105, input the second synthetic sample into the initial student model and the pre-trained teacher model respectively, output the second student prediction result obtained after the initial student model processes the second synthetic sample, and process the second synthetic sample based on the teacher model provided with a random deactivation layer, and output the second teacher prediction result; The student model training module is used to execute step 106, determine at least one student model loss weight based on the current number of iterations, calculate the student model loss value according to the student model loss weight, the second student prediction result and the second teacher prediction result; adjust the student model parameters using the student model loss value to obtain an updated student model trained in the current iteration round; The target generation module is used to repeat the steps 101 to 107 if the number of iterations is less than a preset maximum iteration threshold; otherwise, determine that the updated student model obtained in the last iteration round is the target student model.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images