Data-free knowledge distillation learning method and device, equipment and storage medium

Abstract

The embodiment of the application provides a kind of data-free knowledge distillation learning method, device, equipment and storage medium, belong to knowledge distillation technical field.The method comprises: in current iteration round, at least one generator loss weight is determined based on current iteration number and generator loss value is calculated, generator parameter is adjusted using generator loss value, and updated generator trained in current iteration round is obtained;At least one student model loss weight is determined based on current iteration number and student model loss value is calculated;Student model parameter is adjusted using student model loss value, and updated student model trained in current iteration round is obtained;Wherein, generator loss weight and student model loss weight all increase with the increase of iteration number and show the trend of overall dynamic increase;The training of generator and student model is repeatedly executed until the iteration number reaches the preset determined last iteration round, and the updated student model obtained is target student model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge distillation, and particularly relates to a data-free knowledge distillation learning method and device, equipment and a storage medium. BACKGROUND

[0002] Knowledge distillation (KD) is a model compression method, which guides a lightweight student model to imitate a teacher model with better performance and more complex structure, so as to improve the performance of the student model. The knowledge distillation performed without accessing original data is called data-free knowledge distillation.

[0003] In the related art, a generator is used to generate synthetic samples with similar distribution as the training data of the teacher model, and the student model is trained. However, the traditional data-free knowledge distillation method does not consider the influence of synthetic samples with different difficulties on the learning effect of the student model, thereby causing low training efficiency of the student model. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a data-free knowledge distillation learning method, device, equipment and storage medium, which aims to improve the training efficiency of the student model in data-free knowledge distillation learning.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application provides a data-free knowledge distillation learning method, which comprises the following steps:

[0006] Step 101, obtaining the number of iterations of the current iteration round and obtaining the first synthetic sample generated by the initial generator under the specified generation information;

[0007] Step 102, inputting the first synthetic sample into the initial student model and the pre-trained teacher model respectively, outputting the first student prediction result obtained by processing the first synthetic sample by the initial student model, and processing the first synthetic sample based on the teacher model with the random inactivation layer, and outputting the first teacher prediction result;

[0008] Step 103, determining at least one generator loss weight based on the current number of iterations, calculating the generator loss value according to the generator loss weight, the first student prediction result and the first teacher prediction result, adjusting the generator parameter by using the generator loss value, and obtaining the updated generator trained in the current iteration round;

[0009] Step 104, obtaining the second synthetic sample generated by the updated generator under the specified generation information in the current iteration round;

[0010] Step 105, input the second synthetic sample into the initial student model and the pre-trained teacher model respectively, output the second student prediction result obtained by processing the second synthetic sample by the initial student model, and process the second synthetic sample based on the teacher model with the random inactivation layer, and output the second teacher prediction result;

[0011] Step 106, determine at least one student model loss weight based on the current iteration number, calculate the student model loss value according to the student model loss weight, the second student prediction result and the second teacher prediction result, adjust the student model parameter by using the student model loss value, and obtain the updated student model trained in the current iteration round; wherein the generator loss weight and the student model loss weight both increase dynamically as the iteration number increases;

[0012] Step 107, if the iteration number is less than the preset maximum iteration threshold, repeat steps 101 to 107; otherwise, determine the updated student model obtained in the last iteration round as the target student model.

[0013] In some embodiments, the first teacher prediction result includes a first standard prediction result and a plurality of first random inactivation prediction results; and the second teacher prediction result includes a plurality of second random inactivation prediction results.

[0014] Processing the first synthetic sample based on the teacher model with the random inactivation layer to output the first teacher prediction result includes:

[0015] In the condition of closing the random inactivation layer, processing the first synthetic sample by the teacher model to obtain the first standard prediction result;

[0016] In the condition of opening the random inactivation layer, processing the first synthetic sample by the teacher model to obtain a plurality of first random inactivation prediction results;

[0017] Processing the second synthetic sample based on the teacher model with the random inactivation layer to output the second teacher prediction result includes:

[0018] In the condition of opening the random inactivation layer, processing the second synthetic sample by the teacher model to obtain a plurality of second random inactivation prediction results.

[0019] In some embodiments, the generator loss weight is determined according to the following steps, which include:

[0020] When the iteration number is in a preset first interval, the generator loss weight is determined as a preset first weight value;

[0021] When the iteration number is in a preset second interval, the generator loss weight is determined as the product of a preset second weight value and the iteration number.

[0022] When the iteration number is in a preset third interval, the generator loss weight is determined as a preset third weight value.

[0023] In some embodiments, the generator loss weight comprises a first generator loss weight and a second generator loss weight.

[0024] According to the generator loss weight, the first student prediction result and the first teacher prediction result, a generator loss value is calculated, comprising:

[0025] A first difference value representing the difference between the first student prediction result and the first standard prediction result is calculated, and the product of the first generator loss weight and the first difference value is taken as a first loss term.

[0026] A second loss term is determined based on the second generator loss weight and the plurality of first randomly inactivated prediction results.

[0027] A third loss term is obtained based on the cross-entropy loss between the first standard prediction result and the generation information.

[0028] The first loss term, the second loss term and the third loss term are superimposed to obtain the generator loss value.

[0029] In some embodiments, the student model loss weight is determined according to the following steps, comprising:

[0030] A second difference value representing the difference between the second student prediction result and the plurality of second randomly inactivated prediction results is calculated.

[0031] If the second difference value is less than a difference threshold that dynamically changes with the iteration number, a preset fourth weight value is determined as the student model loss weight, otherwise, a preset fifth weight value is determined as the student model loss weight.

[0032] In some embodiments, according to the student model loss weight, the second student prediction result and the second teacher prediction result, a student model loss value is calculated, comprising:

[0033] The product of the student model loss weight and the second difference value is taken as a fourth loss term.

[0034] The student model loss weight is updated based on a preset hyperparameter to obtain a fifth loss term.

[0035] The fourth loss term and the fifth loss term are superimposed to obtain the student model loss value.

[0036] In some embodiments, after adjusting the generator parameters using the generator loss value, further comprising:

[0037] obtaining a first training step number at which the initial generator is located in the current iteration round;

[0038] if the first training step number does not reach a preset first training threshold, repeating steps 101 to 103, otherwise obtaining the updated generator trained in the current iteration round;

[0039] after adjusting the student model parameters by using the student model loss value, further comprising:

[0040] obtaining a second training step number at which the initial student model is located in the current iteration round;

[0041] if the second training step number does not reach a preset second training threshold, repeating steps 104 to 106, otherwise obtaining the updated student model trained in the current iteration round.

[0042] In some embodiments, adjusting the generator parameters by using the generator loss value to obtain the updated generator trained in the current iteration round comprises:

[0043] updating the generator parameters by gradient based on the preset first learning rate and the generator loss value to obtain updated generator parameters;

[0044] obtaining the updated generator trained in the current iteration round based on the updated generator parameters;

[0045] adjusting the student model parameters by using the student model loss value to obtain the updated student model trained in the current iteration round comprises:

[0046] updating the student model parameters by gradient based on the preset second learning rate and the student model loss value to obtain updated student model parameters;

[0047] obtaining the updated student model trained in the current iteration round based on the updated student model parameters.

[0048] To achieve the above object, a second aspect of the embodiments of the present application proposes a data-free knowledge distillation learning device, which comprises:

[0049] a first obtaining module for performing step 101, obtaining the iteration number of the current iteration round, and obtaining the first synthetic sample generated by the initial generator under the specified generation information;

[0050] a first processing module for performing step 102, inputting the first synthetic sample into the initial student model and the pre-trained teacher model respectively, outputting the first student prediction result obtained by processing the first synthetic sample by the initial student model, and processing the first synthetic sample based on the teacher model provided with the random inactivation layer, and outputting the first teacher prediction result;

[0051] The generator training module is configured to perform step 103, determine at least one generator loss weight based on the current iteration number, calculate a generator loss value according to the generator loss weight, the first student prediction result and the first teacher prediction result, and adjust the generator parameters by using the generator loss value to obtain an updated generator trained in the current iteration round.

[0052] The second obtaining module is configured to perform step 104, and obtain a second synthetic sample generated by the updated generator under specified generation information in the current iteration round.

[0053] The second processing module is configured to perform step 105, input the second synthetic sample into the initial student model and the pre-trained teacher model respectively, output a second student prediction result obtained by processing the second synthetic sample by the initial student model, and process the second synthetic sample based on the teacher model with the random inactivation layer, and output a second teacher prediction result.

[0054] The student model training module is configured to perform step 106, determine at least one student model loss weight based on the current iteration number, calculate a student model loss value according to the student model loss weight, the second student prediction result and the second teacher prediction result, and adjust the student model parameters by using the student model loss value to obtain an updated student model trained in the current iteration round.

[0055] The target generation module is configured to repeat steps 101 to 107 if the iteration number is less than the preset maximum iteration threshold; otherwise, determine the updated student model obtained in the last iteration round as a target student model.

[0056] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0057] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0058] The data-free knowledge distillation learning method and device, electronic equipment and storage medium provided by the present application, through steps 101, the iteration number of the current iteration round is obtained, and the first synthetic sample generated by the initial generator under the specified generation information is obtained; step 102, the first synthetic sample is input into the initial student model and the pre-trained teacher model respectively, the first student prediction result obtained after the initial student model processes the first synthetic sample is output, and the first teacher prediction result is output by processing the first synthetic sample based on the teacher model with a random inactivation layer; step 103, at least one generator loss weight is determined based on the current iteration number, and the generator loss value is calculated according to the generator loss weight, the first student prediction result and the first teacher prediction result; the generator parameter is adjusted by using the generator loss value, and the updated generator trained under the current iteration round is obtained; step 104, under the current iteration round, the second synthetic sample generated by the updated generator under the specified generation information is obtained; step 105, the second synthetic sample is input into the initial student model and the pre-trained teacher model respectively, the second student prediction result obtained after the initial student model processes the second synthetic sample is output, and the second teacher prediction result is output by processing the second synthetic sample based on the teacher model with a random inactivation layer; step 106, at least one student model loss weight is determined based on the current iteration number, and the student model loss value is calculated according to the student model loss weight, the second student prediction result and the second teacher prediction result; the student model parameter is adjusted by using the student model loss value, and the updated student model trained under the current iteration round is obtained; wherein the generator loss weight and the student model loss weight both show a trend of overall dynamic increase with the increase of the iteration number; step 107, if the iteration number is less than the preset maximum iteration threshold, steps 101 to 107 are repeated; otherwise, the updated student model obtained in the last iteration round is determined as the target student model. In this way, by adjusting the opening of the random inactivation layer, the uncertainty of the teacher model is measured, and then the difficulty of the generator to generate the synthetic sample and the learning progress of the student model are controlled, so as to help the student model converge faster and improve the training efficiency of the student model in the data-free knowledge distillation learning. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is an optional flowchart of the data-free knowledge distillation learning method provided by the embodiments of the present application;

[0060] Figure 2 is an optional distillation process schematic diagram of the data-free knowledge distillation learning method provided by the embodiments of the present application;

[0061] Figure 3 is an optional total model training schematic diagram of the data-free knowledge distillation learning method provided by the embodiments of the present application;

[0062] Figure 4 is Figure 1 an implementation flowchart of step 102 in

[0063] Figure 5 is Figure 1 an implementation flowchart of step 103 in

[0064] Figure 6 is an optional generator loss weight determination flowchart of the data-free knowledge distillation learning method provided by the embodiments of the present application;

[0065] Figure 7 is an optional generator parameter update adjustment flowchart of the data-free knowledge distillation learning method provided by the embodiments of the present application;

[0066] Figure 8 is an optional generator training flowchart of the data-free knowledge distillation learning method provided by the embodiments of the present application;

[0067] Figure 9 is Figure 1 an implementation flowchart of step 106 in

[0068] Figure 10 is an optional student model loss weight determination flowchart of the data-free knowledge distillation learning method provided by the embodiments of the present application;

[0069] Figure 11 is an optional student model parameter update adjustment flowchart of the data-free knowledge distillation learning method provided by the embodiments of the present application;

[0070] Figure 12 is Figure 1 an implementation flowchart after step 106 in

[0071] Figure 13 is an optional student model training flowchart of the data-free knowledge distillation learning method provided by the embodiments of the present application;

[0072] Figure 14 is an optional flowchart of the data-free knowledge distillation learning device provided by the embodiments of the present application;

[0073] Figure 15 is a hardware structure schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0074] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application.

[0075] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the sequence in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification is for the purpose of describing the embodiments of the present application only and is not intended to be limiting of the present application.

[0077] Knowledge distillation aims to extract and "infuse" the knowledge and experience of a complex and large model (usually referred to as a teacher model) into a smaller and simpler model (usually referred to as a student model). This process not only preserves the prediction ability and accuracy of the teacher model, but also significantly improves the running efficiency and computing performance of the student model, so that it can perform well in a resource-limited environment.

[0078] Among them, data-free knowledge distillation is a special knowledge distillation, the core feature of which is to use a well-trained model (teacher model) to extract and refine knowledge without accessing the original training data, and then migrate these knowledge to another lightweight model (student model). In this way, knowledge migration and reuse, model compression can be achieved while protecting data privacy.

[0079] In the related art, a generator is used to generate synthetic samples with similar distribution as the training data of the teacher model, and the student model is trained. However, the traditional data-free knowledge distillation method does not consider the influence of synthetic samples of different difficulties on the learning effect of the student model, resulting in low training efficiency of the student model.

[0080] Based on this, the present embodiment proposes a data-free knowledge distillation learning method, device, equipment and storage medium, which aims to improve the training efficiency of the student model in data-free knowledge distillation learning.

[0081] It should be noted that in the embodiments of the present application, when it is necessary to obtain information related to the characteristics of the user, such as user basic information or information related to the identity of the user, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, although the synthetic samples are not directly generated from the original data, they may still contain similar features or patterns as the original data, and if these features or patterns are sufficiently obvious, malicious persons may still be able to use them to infer some information of the original data, therefore, when the embodiments of the present application need to obtain information related to the original data, the user's separate permission or separate consent will be obtained first, and after obtaining the user's separate permission or separate consent, the necessary data for enabling the embodiments of the present application to operate normally will be obtained, for example, the embodiments of the present application will first obtain the authorization or consent of the user before obtaining the synthetic samples generated by the generator under the specified generated information, otherwise the synthetic samples cannot be used in the embodiments of the present application. In addition, the other related data obtained by the data-free knowledge distillation learning device of the present application are all legal data, which will not be described here.

[0082] In the embodiments of the present application, the dimensions of the data-free knowledge distillation learning device (for the sake of description, hereinafter can also be referred to as "distillation learning device" or "distillation device") will be described, which can be integrated in a computer device, such as a server. Figure 1 As shown in Figure 1 is an optional flowchart of the data-free knowledge distillation learning method (for the sake of description, hereinafter can also be referred to as "distillation learning method" or "distillation method") provided by the embodiments of the present application, Figure 1 The method in the above can include but not limited to the following steps 101 to 107, and the specific process is as follows when the distillation device executes the prediction method. It should be first noted that the order of steps 101 to 107 in the above is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or increased according to actual needs. Figure 1

[0083] Step 101, obtaining the iteration number of the current iteration round and obtaining the first synthetic sample generated by the initial generator under the specified generated information.

[0084] The step 101 will be described in detail below.

[0085] ​The generator aims to generate new data instances as close as possible to the real data distribution, which receives random noise as input and converts it into data points (such as images, audio, etc.), and the converted data points should be indistinguishable from real data. The generator in the embodiments of the present application can be a standard generative adversarial network (GAN) generator, a recurrent GAN generator, a deep convolutional GAN generator, etc. In actual application, a suitable generator can be selected according to specific circumstances, and the selected generator is not limited in the embodiments of the present application. In the embodiments of the present application, the generator that is not trained completely at the current iteration round is referred to as an initial generator, and the student model that is not trained completely is referred to as an initial student model. In the subsequent description, the generator refers to the initial generator, and the student model refers to the initial student model unless otherwise specified.

[0086] The first synthetic sample can be one or multiple. When the number of first synthetic samples is multiple, the multiple first synthetic samples form a synthetic sample set and are respectively input into the student model and the teacher model. That is, the specific number of first synthetic samples is only for illustration and does not specifically refer to the specific number of first synthetic samples.

[0087] As shown in FIG. 1, Figure 2 Figure 2 is an optional distillation process schematic diagram of the data-free knowledge distillation learning method provided by the embodiments of the present application. The generator, the pre-trained teacher model, and the student model to be trained constitute a data-free knowledge distillation learning architecture (referred to as a total model). In the data-scarce or data-privacy-sensitive scenario, the generator can generate synthetic samples that meet the requirements of data-free knowledge distillation learning under the constraint of specified generation information. In this way, the exposure of sensitive data is avoided, and the student model can be helped to perform efficient distillation learning through self-generated sample data in the data-scarce scenario. Exemplarily, Figure 2 The generator G shown in FIG. 2 is a category generator, which can generate synthetic samples based on preset generation information, Figure 2 The generation information in the example is used to indicate that the generator outputs a digital image.

[0088] The teacher model is a well-trained and well-performing model, which is usually a larger and more complex model to achieve a high performance level on a specific task. Generally, the teacher model is trained based on a pre-set private data set, and the teacher model has the characteristics of high performance, high complexity, and rich knowledge. Exemplarily, the teacher model can be a deep convolutional neural network (Deep Convolutional Neural Networks, CNNs), a recurrent neural network (Recurrent Neural Networks, RNNs), a neural network model based on a self-attention mechanism, etc.

[0089] ​The student model is a smaller and simpler model, and the student model aims to learn and imitate the behavior or prediction result of another more complex and better-performing model (i.e., the teacher model). Exemplarily, the student model can be a simplified CNNs model, a simplified RNNs model, or the like. It should be noted that the teacher model and the student model used in the embodiments of the present application can be adaptively adjusted according to actual conditions, which are only examples and are not mandatory constraints.

[0090] Further, after completing the training architecture building, the generator parameters are initialized, such as Figure 2 As shown in the figure, the generator G includes two inputs: a d-dimensional vector z subject to a standard normal distribution z ~ N(0, I) and generation information Y, so that the generator can output a synthetic sample (including a first synthetic sample and a second synthetic sample); the related parameters of the teacher model and the student model are initialized, wherein the parameter amount of the student model is much smaller than that of the teacher model, so that the purpose of reducing the storage space and reducing the inference calculation amount during data processing can be achieved; the iteration number is initialized, and the current iteration number c after initialization is 0.

[0091] As shown in the figure, Figure 3 Figure 3 is an optional total model training schematic diagram of the data-free knowledge distillation learning method provided by the embodiments of the present application, in order to obtain the final target student model, the training process of the total model includes:

[0092] ① Total model preparation and parameter initialization;

[0093] ② Training the initial generator to obtain an updated generator;

[0094] ③ Training the initial student model using the updated generator to obtain an updated student model;

[0095] ④ When the iteration training number reaches a preset maximum iteration threshold, the updated student model obtained in the last iteration round is determined as the target student model; otherwise, steps ② and ③ are repeated.

[0096] First, the initial generator is trained to obtain an updated generator.

[0097] In step 102, the first synthetic sample is input into the initial student model and the pre-trained teacher model respectively, and the first student prediction result obtained by the initial student model processing the first synthetic sample is output, and the first teacher prediction result is output by processing the first synthetic sample based on the teacher model with a random inactivation layer.

[0098] Step 102 is described in detail as follows.

[0099] As shown in the figure, Figure 2 ​As shown, under the current iteration, a bs-dimensional z vector construction matrix Z = (z1; z2;...; z bs ) and a generation information Y = (y1; y2;...; y bs ) are randomly generated and input into the generator G to obtain bs generated images (synthetic samples), wherein each synthetic sample has a dimension of dim h x dim w pixel matrix, dim h represents the row dimension of the matrix, and dim w represents the column dimension of the matrix; the first synthetic sample

[0100] Further, the teacher model proposed in the embodiments of the present application is provided with a random inactivation layer (Dropout layer). The Dropout layer randomly closes part of the neurons in the training process to improve the generalization ability of the model. In the inference process, the related technology usually selects to close the Dropout of the model. In the inference process in the present application, the Dropout layer is opened, and the Monte Carlo Dropout technology is used to estimate the uncertainty of the prediction result of the model. Specifically, Monte Carlo Dropout is a method for estimating the uncertainty of the model using Dropout technology. In the test stage, multiple predictions are performed by opening the Dropout layer, and then the average of the multiple prediction results is calculated to measure the uncertainty of the model.

[0101] Further, the first synthetic sample generated by the generator will be input into the initial student model to be trained and the teacher model that has been trained, so that the student model and the teacher model perform data processing on the input first synthetic sample according to the predetermined task, and output the processing result. In the Figure 2 example, the predetermined task is an image content recognition task, and the student model S to be trained and the pre-trained teacher model T will process the input first synthetic sample, and then the student model S outputs the first student prediction result wherein, includes a plurality of components

[0102] In some embodiments, Figure 4 is Figure 1 an implementation flowchart of step 102 in

[0103] Step 201, under the condition of closing the random inactivation layer, the first standard prediction result obtained by processing the first synthetic sample by the teacher model is output.

[0104] In step 202, the teacher model is used to process the first synthetic sample for multiple times to obtain multiple first dropout prediction results under the condition that the dropout layer is turned on.

[0105] In some embodiments, when the student model processes the synthetic sample, the teacher model also processes the synthetic sample synchronously, and the teacher model outputs the first teacher prediction result under the condition that the dropout layer is turned on. Since the student model is distilled based on the student prediction result output by the student model itself and the teacher prediction result output by the teacher model, the embodiments of the present application measure the uncertainty of the teacher model by adjusting the turning on of the dropout layer, thereby controlling the difficulty of the synthetic sample generated by the generator and the learning progress of the student model, and achieving the purpose of helping the student model converge faster.

[0106] Specifically, under the condition that the Dropout layer of the teacher model is turned off, the teacher model is used to process the synthetic sample for multiple times to obtain standard prediction results Wherein, comprises multiple components i = 1, 2, …, n; and under the condition that the Dropout layer of the teacher model is turned on, the teacher model is used to process the synthetic sample for multiple times to obtain multiple dropout prediction results, such as Figure 2 In the example, it is assumed that the teacher model processes the synthetic sample K times when the Dropout layer of the teacher model is turned on, and K dropout prediction results are obtained Then, the multiple dropout prediction results are averaged to obtain for measuring the uncertainty of the teacher model. It should be noted that the embodiments of the present application do not specifically limit the value of K, which can be adaptively adjusted according to actual conditions.

[0107] Exemplarily, when the predetermined task is an image content recognition task, the standard prediction result and the dropout prediction result both represent the output result of predicting the input synthetic sample, i.e., the obtained image category probability Wherein, x is the synthetic sample, is the category of the image predicted by the teacher model.

[0108] In step 103, at least one generator loss weight is determined based on the current iteration number, and a generator loss value is calculated according to the generator loss weight, the first student prediction result and the first teacher prediction result; the generator loss value is used to adjust the generator parameter to obtain an updated generator trained in the current iteration round.

[0109] The step 103 is described in detail below.

[0110] In some embodiments, the generator loss weight will gradually increase as the current iteration number increases, so that the generator can gradually learn more about the characteristics of the data distribution during the training process, so that the generated synthetic samples are closer to the real data; based on the increasing generator loss weight, the generator loss value can reflect the current learning progress and performance change of the generator from easy to difficult in different iteration rounds, thereby realizing the step-by-step optimization of the generator.

[0111] In some embodiments, Figure 5 Figure 1 is an implementation flowchart of step 103 in

[0112] Step 301, calculate a first difference value representing the difference between the first student prediction result and the first standard prediction result, and take the product of the first generator loss weight and the first difference value as a first loss term.

[0113] Step 302, determine a second loss term based on the second generator loss weight and the plurality of first randomly inactivated prediction results.

[0114] Step 303, based on the cross-entropy loss between the first standard prediction result and the generation information, obtain a third loss term.

[0115] Step 304, superimpose the first loss term, the second loss term and the third loss term to obtain the generator loss value.

[0116] The steps 301 to 304 are described in detail as follows.

[0117] In some embodiments, the generator has an independent first training step number at each iteration round, specifically, the first training step number of the generator is initialized to Stepg=0 every time an iteration round is entered; then the generator loss value is calculated by the following formula<1>:

[0118] Loss g = -α(c)Loss adv +β(c)Loss u +Loss c <1>

[0119] Wherein, α(c) is the first generator loss weight value, Loss adv is the first difference value, -α(c)Loss adv is the first loss term; β(c) is the second generator loss weight value, β(c)Loss u is the second loss term; Loss​c is a third loss term.

[0120] wherein, Loss adv measures the difference between the prediction results of the student model and the teacher model, Loss adv is smaller, the prediction results of the teacher model and the student model on the same synthetic sample are closer, and since the synthetic sample generated by the generator has less influence on the parameter update of the student model in the next iteration round, the present embodiment expects that the difference between the prediction results of the synthetic sample generated by the generator and the teacher model and the student model is larger when the distillation method is performed; Loss u measures the uncertainty of the teacher model; Loss c measures the difference between the standard prediction result and the generated information.

[0121] wherein, α(c) is a generator loss weight corresponding to the Loss adv term, and α(c) can be determined through the following steps 401 to 403:

[0122] In some embodiments, as shown in Figure 6 , Figure 6 is an optional generator loss weight determination flowchart of the data-free knowledge distillation learning method provided by the present embodiment, and for different terms of the generator loss value, the corresponding generator loss weight can be determined according to the following steps 401 to 403:

[0123] Step 401, when the iteration number is in a preset first interval, the generator loss weight is determined as a preset first weight value.

[0124] Step 402, when the iteration number is in a preset second interval, the generator loss weight is determined as a product of a preset second weight value and the iteration number.

[0125] Step 403, when the iteration number is in a preset third interval, the generator loss weight is determined as a preset third weight value.

[0126] The steps 401 to 403 are described in detail as follows.

[0127] In some embodiments, for the first generator loss weight α(c) in formula <1>, the following update strategy is adopted:

[0128]

[0129] wherein, c is the current iteration number; when c falls in a preset first interval (0, c α1 ), c is a preset first weight value, i.e., c=0; when c falls in a preset second interval [c α1 , cα2 ], c is the preset second weight value, that is, c = αc; when c falls within the preset third interval (c α2 , c max ], c is the preset third weight value, that is, c = α max ;c α1 、c α2 , α, α max are all pre-set hyperparameters, c max The preset maximum number of iterations.

[0130] It is understandable that when 0 <c<c α1 When the performance of the student model is low, it may not be able to learn the knowledge of the teacher model. Setting α(c) to 0 can enable the generator to focus on generating synthetic samples under the specified generation information, thereby preventing the generator from being disturbed by adversarial training and helping the generator to quickly learn basic image structures and category features. α1 ≤c≤c α2 When gradually increasing Loss adv In Los g The weight of , so as to smoothly guide the generator from simple synthetic sample generation to more complex adversarial generation, thereby improving the quality of the generated synthetic samples; when c α2 <c≤c max When the generator has a certain generation capability, by fixing the Loss adv The weights ensure that the generator continues to optimize in the adversarial environment to produce more refined and realistic synthetic samples, which helps stabilize the training process and avoid performance fluctuations caused by over-adjustment.

[0131] In some embodiments, for the formula <1> The second generator loss weight β(c) in has the following update strategy:

[0132]

[0133] Wherein, c is the current number of iterations; when c falls within the preset first interval (0, c β1 ), c is the preset first weight value, that is, c=0; when c falls within the preset second interval [c β1 , c β2 ], c is the preset second weight value, that is, c = βc; when c falls within the preset third interval (c β2 , c max ], c is the preset third weight value, that is, c = α max ;c β1 、c β2 , β, α max are all pre-set hyperparameters, c max The preset maximum number of iterations.

[0134] When 0 <c<c β1 When the generator has low performance and may not be able to generate high-quality synthetic samples, setting β(c) to 0 can enable the generator to focus on generating synthetic samples under the specified generation information, thereby preventing the generator from being disturbed by adversarial training and helping the generator to quickly learn basic image structures and category features; when c β1 ≤c≤c β2 When gradually increasing Loss u In Los g The weight of , so that the generator can be smoothly guided to transition from simple synthetic sample generation to samples that are more difficult for the teacher model to distinguish, thereby improving the quality of the generated synthetic samples; when c α2 <c≤c max When the generator has a certain generation capability, by fixing the Loss u The weights ensure that the generator continues to optimize in the adversarial environment to produce more refined and realistic synthetic samples, which helps stabilize the training process and avoid performance fluctuations caused by over-adjustment.

[0135] Furthermore, both the first generator loss weight and the second generator loss weight show an overall dynamic increasing trend as the number of iterations increases. It should be noted that when the generator loss value includes more loss terms, and the loss terms also include the generator loss weight, the specific value of the generator loss weight that changes with the number of iterations can also be determined according to steps 401 to 403, thereby achieving training of the generator from easy to difficult.

[0136] Next, continue with the explanation <1> ,Mode <1> Loss adv Through the following formula <2> Calculation yields:

[0137]

[0138] in, Representative The i-th standard prediction result component in, Representative The i-th student prediction result component in ; bs represents the bs synthetic samples generated by the generator; KL is the information gain or relative entropy (Kullback-Leibler Divergence, KL divergence), which is an asymmetric measurement method used to measure the difference between two probability theoretical distributions and the true distribution.

[0139] Furthermore, <1> Loss u Through the following formula <3> Calculation yields:

[0140]

[0141] wherein, The mean is a measure of central tendency of a distribution of numerical values that is calculated by dividing the sum of all the numerical values by the number of values.

[0142] Further, the Loss in formula <1> is calculated by c

[0143]

[0144] wherein, CE(·) is used to represent the cross-entropy loss function, y ij is the jth element of the class one-hot encoding of the ith synthetic sample, is the probability of the model that the ith synthetic sample belongs to the jth generated information, i.e. is the jth element of the class one-hot encoding of the ith synthetic sample.

[0145] Further, as shown in Figure 7 Figure 7 is an optional generator parameter update adjustment flowchart of the data-free knowledge distillation learning method provided by the embodiments of the present application, which adjusts the generator parameters by using the generator loss value to obtain an updated generator trained in the current iteration round, including the following steps 701 to step 702:

[0146] Step 701, based on the preset first learning rate and the generator loss value, gradient update is performed on the generator parameters to obtain the updated generator parameters.

[0147] Step 702, based on the updated generator parameters, an updated generator trained in the current iteration round is obtained.

[0148] The steps 701 to step 702 are described below.

[0149] In some embodiments, the update of the generator parameters θ g is implemented based on the following formula <5>:

[0150]

[0151] wherein, ε is the first learning rate, is the generator loss value Loss calculated by the loss function, g is the gradient of the generator parameters θ g .

[0152] In some embodiments, after adjusting the generator parameters by using the generator loss value, the following steps 801 to 802 are further included:

[0153] ​​Step 801, obtaining a first training step number at which the initial generator is currently located in the current iteration round.

[0154] Step 802, if the first training step number does not reach a preset first training threshold, repeating steps 101 to 103, otherwise obtaining the updated generator trained in the current iteration round.

[0155] In some embodiments, in the iterative training of each round, the initial generator is trained multiple times to obtain the updated generator trained in the current iteration round. For example, as shown in Figure 8 Figure 8 is an optional generator training flowchart of the data-free knowledge distillation learning method provided by the embodiments of the present application, and specifically includes the following training steps:

[0156] (1.1) initializing a first training step number Step g = 0 of the generator in the current iteration number.

[0157] (1.2) generating a first synthetic sample based on preset generation information;

[0158] (1.3) inputting the first synthetic sample into the initial student model to obtain a first student prediction result wherein, comprises multiple components

[0159] (1.4) closing the Dropout layer of the teacher model, inputting the first synthetic sample into the teacher model to obtain a first standard prediction result wherein, comprises multiple components

[0160] (1.5) opening the Dropout layer of the teacher model, inputting the first synthetic sample into the teacher model to perform K times of prediction to obtain K first random inactivation prediction results

[0161] (1.6) calculating a generator loss value Loss based on the K first random inactivation prediction results the first standard prediction result and the first student prediction result g ;

[0162] (1.7) updating the generator parameter θ g based on the generator loss value Loss g ;

[0163] (1.8) updating the first training step number Step g = Step​g +1;

[0164] (1.9) If Step g <Step gmax then repeat steps (1.2) to (1.9), otherwise end the generator training in the current iteration round, and obtain the updated generator; wherein Step gmax is a preset first training threshold, Step gmax can be adaptively adjusted according to actual conditions, and the embodiments of the present application do not limit this.

[0165] Next, how to train the initial student model based on the updated generator to obtain the updated student model is introduced.

[0166] Step 104, in the current iteration round, obtain the second synthetic sample generated by the updated generator under the specified generation information;

[0167] Step 105, input the second synthetic sample into the initial student model and the pre-trained teacher model respectively, output the second student prediction result obtained by the initial student model processing the second synthetic sample, and process the second synthetic sample based on the teacher model with the random inactivation layer, output the second teacher prediction result.

[0168] Among them, the specific implementation of step 104 is similar to that of step 101, and the specific implementation of step 105 is basically the same as that of step 102, the difference is that in order to obtain the updated student model, step 105 only outputs a plurality of second random inactivation prediction results corresponding to the plurality of times of processing the second synthetic sample by the teacher model under the condition of opening the random inactivation layer, without outputting the prediction result under the condition of closing the random inactivation layer.

[0169] Step 106, determine at least one student model loss weight based on the current iteration number, calculate the student model loss value according to the student model loss weight, the second student prediction result and the second teacher prediction result; adjust the student model parameters using the student model loss value to obtain the updated student model trained in the current iteration round; wherein the generator loss weight and the student model loss weight both increase in a whole dynamic increasing trend with the increase of the iteration number;

[0170] Step 106 is described in detail below.

[0171] In some embodiments, the student model loss weight will gradually increase as the number of current iterations increases, so that the student model can gradually learn more prediction result features during the training process, so that the generated student prediction result is more accurate; based on the increasing student model loss weight, the student model loss value can reflect the current learning progress and performance change of the student model from easy to difficult in different iteration rounds, thereby realizing the step-by-step optimization of the student model.

[0172] In some embodiments, as Figure 9 shown, Figure 9 is Figure 1 an implementation flowchart of step 106 in the method, and the student model loss value is calculated based on the student model loss weight, the student prediction result and the teacher prediction result, including the following steps 501 to 503:

[0173] Step 501, multiplying the student model loss weight and the second difference value as a fourth loss term;

[0174] Step 502, updating the student model loss weight based on a preset hyperparameter to obtain a fifth loss term;

[0175] Step 503, superimposing the fourth loss term and the fifth loss term to obtain the student model loss value.

[0176] The following describes steps 501 to 503 in detail.

[0177] In some embodiments, the student model loss value is calculated by the following formula <6>:

[0178] Loss s = v(γ, L) Loss KL + g(γ, v) <6>

[0179] Wherein, v(γ, L) Loss KL is the fourth loss term, g(γ, v) is the fifth loss term; v(γ, L) is the student model loss weight value. v(γ, L) is the corresponding student model loss value of the Loss KL term, as Figure 10 shown, Figure 10 is an optional student model loss weight determination flowchart of the data-free knowledge distillation learning method provided by the embodiments of the present application, that is, v(γ, L) can be determined by the following steps 601 to 602:

[0180] Step 601, calculating a second difference value representing the difference between the second student prediction result and a plurality of second randomly inactivated prediction results;

[0181] Step 602, if the second difference value is less than the difference threshold dynamically changed with the iteration number, determining a preset fourth weight value as the student model loss weight, otherwise, determining a preset fifth weight value as the student model loss weight.

[0182] The steps 601 to 602 are described in detail below.

[0183] In some embodiments, Loss in formula <6> is KL The second difference value Loss is calculated by formula <7> below KL :

[0184]

[0185] Wherein, is The prediction result of the i th synthetic sample, by taking the average of the multiple random inactivation prediction results of the teacher model as the learning target of the student model, can reduce the introduction of noise due to the large uncertainty of the teacher model, and further control the learning progress of the student model; the explanations of other parameters in formula <7> are the same as those in formula <2>, which will not be repeated here.

[0186] Further, for the student model loss weight value v (γ, L) in formula <6>, the following updating strategy is used:

[0187]

[0188] Wherein, 1 is a preset fourth weight value, and 0 is a preset fifth weight value; the difference threshold γ is updated by formula <8> below

[0189] γ (c) = γ 0 + ρc <8>

[0190] Wherein, γ (c) is a linear model of the iteration number; c represents the current iteration round; γ 0 is a preset initial value; ρ is a preset growth rate; it should be noted that the preset parameters such as the fourth weight value, the fifth weight value, and the growth rate can be adaptively adjusted according to actual conditions, and the specific values of these parameters are not limited in the embodiments of the present application.

[0191] Next, the formula <6> is continued to be described, g (γ, v) in formula <6> is a regularization term in curriculum learning, and a feasible implementation manner is v i Indicates the corresponding student model loss weight of the i th synthetic sample

[0192] In some embodiments, as shown in Figure 11 Figure 11 ​This is an optional student model parameter update and adjustment flow chart of the data-free knowledge distillation learning method provided in an embodiment of the present application. The student model parameters are adjusted using the student model loss value to obtain an updated student model trained in the current iteration round, including the following steps 703 to 704:

[0193] Step 703: performing gradient update on the student model parameters based on a preset second learning rate and the student model loss value to obtain updated student model parameters;

[0194] Step 704: Based on the updated student model parameters, obtain the updated student model trained in the current iteration round.

[0195] Steps 703 to 704 are described in detail below.

[0196] In some embodiments, based on the following formula <9> Implement the student model parameters θ s Update:

[0197]

[0198] Among them, ε is the second learning rate, The student model loss value Loss calculated by the loss function s For the student model parameters θ s gradient.

[0199] In some embodiments, as Figure 12 As shown, Figure 12 yes Figure 1 An implementation flow chart after step 106 further includes the following steps 803 to 804:

[0200] Step 803: Obtain the second training step number of the initial student model in the current iteration round;

[0201] Step 804: If the second training step number does not reach the preset second training threshold, repeat steps 104 to 106; otherwise, obtain the updated student model trained in the current iteration round.

[0202] Steps 803 to 804 are described in detail below.

[0203] In some embodiments, in each round of iterative training, the initial student model is trained multiple times to obtain an updated student model trained in the current iterative round. Figure 13 As shown, Figure 13 This is an optional student model training flowchart of the data-free knowledge distillation learning method provided in the embodiment of the present application, which specifically includes the following training steps:

[0204] (2.1) initializing a second training step number Step s of the student model at the current iteration number

[0205] (2.2) generating a second synthetic sample based on preset generation information

[0206] (2.3) inputting the second synthetic sample into the initial student model to obtain a second student prediction result wherein, comprising a plurality of components

[0207] (2.4) starting a Dropout layer of the teacher model, inputting the second synthetic sample into the teacher model, and performing K times of prediction to obtain K second randomly inactivated prediction results

[0208] (2.5) calculating a student model loss value Loss based on the K second randomly inactivated prediction results and the second student prediction result s ;

[0209] (2.6) updating student model parameters θ s based on the student model loss value Loss s ;

[0210] (2.7) updating the second training step number Step s = Step s + 1

[0211] (2.8) if Step s < Step smax , repeating steps (2.2) to (2.8), otherwise ending the student model training at the current iteration round to obtain an updated student model; wherein Step smax is a preset second training threshold, and Step smax may be adaptively adjusted according to actual conditions, and the embodiments of the present application do not limit this.

[0212] Step 107, if the iteration number is less than a preset maximum iteration threshold, repeating steps 101 to 107, otherwise determining the updated student model obtained in the last iteration round as a target student model.

[0213] In some embodiments, after the generator completes Step gmax times of training, and the student model completes Step smaxAfter the secondary training, the training of the total model under the current iteration round is ended; then, the iteration number c is updated as c=c+1; it is judged whether the updated iteration number reaches the maximum iteration threshold or not, if not, steps 101 to 107 are repeated; otherwise, the updated student model obtained when the maximum iteration threshold is reached is determined as the target student model.

[0214] The maximum iteration threshold can be adaptively adjusted according to actual conditions, and the embodiments of the present application do not limit this.

[0215] As shown in Figure 14 , Figure 14 is an optional flowchart of the data-free knowledge distillation learning device provided by the embodiments of the present application, and the data-free knowledge distillation learning device includes the following modules 901 to 907:

[0216] The first acquisition module 901 is configured to execute step 101, acquire the iteration number of the current iteration round, and acquire the first synthetic sample generated by the initial generator under the specified generation information;

[0217] The first processing module 902 is configured to execute step 102, input the first synthetic sample into the initial student model and the pre-trained teacher model respectively, output the first student prediction result obtained by the initial student model processing the first synthetic sample, and process the first synthetic sample based on the teacher model with the random inactivation layer, and output the first teacher prediction result;

[0218] The generator training determination module 903 is configured to execute step 103, determine at least one generator loss weight based on the current iteration number, calculate the generator loss value according to the generator loss weight, the first student prediction result and the first teacher prediction result, and adjust the generator parameter by using the generator loss value to obtain the updated generator trained under the current iteration round;

[0219] The second acquisition module 904 is configured to execute step 104, acquire the second synthetic sample generated by the updated generator under the specified generation information under the current iteration round;

[0220] The second processing module 905 is configured to execute step 105, input the second synthetic sample into the initial student model and the pre-trained teacher model respectively, output the second student prediction result obtained by the initial student model processing the second synthetic sample, and process the second synthetic sample based on the teacher model with the random inactivation layer, and output the second teacher prediction result;

[0221] The student model training module 906 is configured to perform step 106, determine at least one student model loss weight based on the current iteration number, calculate a student model loss value according to the student model loss weight, the second student prediction result and the second teacher prediction result, and adjust the student model parameter by using the student model loss value to obtain an updated student model trained in the current iteration round.

[0222] The target generation module 907 is configured to repeat the steps 101 to 107 if the iteration number is less than the preset maximum iteration threshold, or determine the updated student model obtained in the last iteration round as the target student model.

[0223] The data-free knowledge distillation learning method, device, electronic device and storage medium provided in the present application can measure the uncertainty of the teacher model by adjusting the opening of the random inactivation layer, and then control the difficulty of the generator to generate the synthetic sample and the learning progress of the student model, so as to help the student model converge faster and improve the training efficiency of the student model in the data-free knowledge distillation learning.

[0224] The specific implementation of the distillation learning device is basically the same as the specific embodiment of the distillation learning method described above, and will not be repeated here.

[0225] The present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-mentioned data-free knowledge distillation learning method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0226] As shown in Figure 15 , the hardware structure of the electronic device provided in the present application is shown in Figure 15 , which includes:

[0227] The processor 1001 can be implemented in the form of a general CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided in the present application.

[0228] The memory 1002 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1002 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1002 and are called and executed by the processor 1001 to implement the data-free knowledge distillation learning method of the embodiments of the present application;

[0229] The input / output interface 1003 is configured to realize information input and output.

[0230] The communication interface 1004 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0231] The bus 1005 is configured to transmit information between various components (for example, the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004) of the device.

[0232] The processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 are connected to each other through the bus 1005 to realize the communication connection between the devices.

[0233] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned data-free knowledge distillation learning method.

[0234] The memory is a non-transitory computer readable storage medium, which can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0235] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0236] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation to the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0237] The device embodiments described above are merely illustrative, and units described as separate components can or can not be physically separated, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0238] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0239] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0240] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0241] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0242] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0243] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0244] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0245] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the rights of the embodiments of the present application. Any modification, equivalent replacement and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A data-free knowledge distillation learning method, characterized in that: The method comprises: Step 101: Obtain the number of iterations of the current iteration round, and obtain a first synthetic sample generated by an initial generator under specified generation information, wherein the first synthetic sample is an image; Step 102: Input the first synthetic sample into an initial student model and a pre-trained teacher model respectively, output a first student prediction result obtained after the initial student model processes the first synthetic sample, and process the first synthetic sample based on the teacher model provided with a random dropout layer, and output a first teacher prediction result; Step 103: determining at least one generator loss weight based on the current number of iterations, and calculating a generator loss value based on the generator loss weight, the first student prediction result, and the first teacher prediction result; adjusting the generator parameters using the generator loss value to obtain an updated generator trained in the current iteration round; Step 104: In the current iteration round, obtain a second synthetic sample generated by the update generator under the specified generation information, wherein the second synthetic sample is an image; Step 105: Input the second synthetic sample into the initial student model and the pre-trained teacher model respectively, output a second student prediction result obtained after the initial student model processes the second synthetic sample, and process the second synthetic sample based on the teacher model provided with a random dropout layer, and output a second teacher prediction result; Step 106: Determine at least one student model loss weight based on the current number of iterations, and calculate a student model loss value based on the student model loss weight, the second student prediction result, and the second teacher prediction result; use the student model loss value to adjust the student model parameters to obtain an updated student model trained in the current iteration round; wherein the generator loss weight and the student model loss weight both show an overall dynamic increasing trend with the increase in the number of iterations; Step 107: If the number of iterations is less than the preset maximum iteration threshold, repeat steps 101 to 107; otherwise, determine that the updated student model obtained in the last iteration round is the target student model.

2. The method according to claim 1, characterized in that The first teacher prediction result includes a first standard prediction result and a plurality of first random loss prediction results; the second teacher prediction result includes a plurality of second random loss prediction results; The processing of the first synthetic sample based on the teacher model provided with the random deactivation layer and outputting a first teacher prediction result includes: outputting the first standard prediction result obtained by processing the first synthetic sample by the teacher model under the condition that the random dropout layer is closed; Under the condition that the random dropout layer is turned on, outputting a plurality of first random dropout prediction results corresponding to the teacher model processing the first synthetic sample multiple times; The processing of the second synthetic sample based on the teacher model provided with the random deactivation layer and outputting a second teacher prediction result includes: Under the condition that the random deactivation layer is turned on, multiple second random deactivation prediction results obtained by the teacher model performing multiple processing on the second synthetic sample are output.

3. The method according to claim 2, characterized in that The generator loss weight is determined according to the following steps, comprising: When the number of iterations is within a preset first interval, determining the generator loss weight to be a preset first weight value; When the number of iterations is within a preset second interval, determining the generator loss weight as a product of a preset second weight value and the number of iterations; When the number of iterations is within a preset third interval, the generator loss weight is determined to be a preset third weight value.

4. The method according to claim 2, characterized in that The generator loss weight includes a first generator loss weight and a second generator loss weight; The generator loss value is calculated according to the generator loss weight, the first student prediction result, and the first teacher prediction result, including: Calculating a first difference value representing the degree of difference between the first student prediction result and the first standard prediction result, and taking the product of the first generator loss weight and the first difference value as a first loss term; determining a second loss term based on the second generator loss weight and a plurality of the first random dropout prediction results; Obtaining a third loss term based on a cross entropy loss between the first standard prediction result and the generated information; The first loss term, the second loss term, and the third loss term are superimposed to obtain the generator loss value.

5. The method according to claim 2, characterized in that The student model loss weight is determined according to the following steps, comprising: Calculating a second difference value representing the degree of difference between the second student prediction result and the plurality of second random inactivation prediction results; If the second difference value is less than the difference threshold that changes dynamically with the number of iterations, the preset fourth weight value is determined to be the student model loss weight; otherwise, the preset fifth weight value is determined to be the student model loss weight.

6. The method according to claim 5, characterized in that The calculating the student model loss value according to the student model loss weight, the second student prediction result, and the second teacher prediction result includes: multiplying the student model loss weight by the second difference value as a fourth loss term; Updating the student model loss weight based on preset hyperparameters to obtain a fifth loss term; The fourth loss term and the fifth loss term are superimposed to obtain the student model loss value.

7. The method according to claim 2, characterized in that After adjusting the generator parameters using the generator loss value, the method further includes: Obtain the first training step number of the initial generator in the current iteration round; If the first training step number does not reach the preset first training threshold, repeat steps 101 to 103, otherwise obtain the updated generator trained in the current iteration round; After adjusting the student model parameters using the student model loss value, the method further includes: Obtaining the second training step number of the initial student model in the current iteration round; If the second training step number does not reach the preset second training threshold, repeat steps 104 to 106, otherwise obtain the updated student model trained in the current iteration round.

8. The method according to claim 1, characterized in that The generator loss value is used to adjust the generator parameters to obtain an updated generator trained in the current iteration round, including: Based on a preset first learning rate and the generator loss value, performing a gradient update on the generator parameters to obtain the updated generator parameters; Based on the updated generator parameters, obtaining the updated generator trained in the current iteration round; The method of adjusting the student model parameters by using the student model loss value to obtain an updated student model trained in the current iteration round includes: Based on a preset second learning rate and the student model loss value, gradient updating the student model parameters to obtain updated student model parameters; Based on the updated student model parameters, the updated student model trained in the current iteration round is obtained.

9. A data-free knowledge distillation learning device, characterized in that: The device comprises: A first acquisition module, configured to execute step 101, obtain the number of iterations of the current iteration round, and obtain a first synthetic sample generated by the initial generator under specified generation information, wherein the first synthetic sample is an image; a first processing module, configured to execute step 102, input the first synthetic sample into an initial student model and a pre-trained teacher model, respectively, output a first student prediction result obtained after the initial student model processes the first synthetic sample, and process the first synthetic sample based on the teacher model provided with a random dropout layer, and output a first teacher prediction result; A generator training module is configured to execute step 103, determine at least one generator loss weight based on the current number of iterations, calculate a generator loss value based on the generator loss weight, the first student prediction result, and the first teacher prediction result; and use the generator loss value to adjust generator parameters to obtain an updated generator trained for the current iteration round. A second acquisition module is configured to execute step 104, and acquire, in a current iteration round, a second synthetic sample generated by the update generator under specified generation information, wherein the second synthetic sample is an image; a second processing module, configured to execute step 105, input the second synthetic sample into the initial student model and the pre-trained teacher model, respectively, output a second student prediction result obtained after the initial student model processes the second synthetic sample, and process the second synthetic sample based on the teacher model provided with a random dropout layer, and output a second teacher prediction result; The student model training module is configured to execute step 106, determine at least one student model loss weight based on the current number of iterations, calculate a student model loss value based on the student model loss weight, the second student prediction result, and the second teacher prediction result; adjust the student model parameters using the student model loss value to obtain an updated student model trained in the current iteration round; wherein both the generator loss weight and the student model loss weight show an overall dynamic increasing trend as the number of iterations increases; The target generation module is used to execute step 107. If the number of iterations is less than the preset maximum iteration threshold, repeat steps 101 to 107; otherwise, determine that the updated student model obtained in the last iteration round is the target student model.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images