Model training method and device based on knowledge distillation, equipment and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and provides a model training method and device based on knowledge distillation, equipment and a storage medium, wherein the method comprises the following steps: obtaining a first model meeting a target condition and N second initial models not meeting the target condition; obtaining N second label probability sets according to a training sample set; processing the second label set to obtain a third label probability set corresponding to the second initial model; obtaining an average loss function corresponding to the N second initial models according to a first label probability set obtained in advance by the base and the third label probability set; obtaining a target loss function corresponding to each second initial model according to the loss function corresponding to the second initial model; obtaining N second models meeting the target condition according to the target loss function; and performing information fusion on the N models in the second model when the second model is trained, so that the information obtained by each second model is more comprehensive, thereby improving the performance of the second model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, device, and storage medium based on knowledge distillation. Background Technology

[0002] With the development of artificial intelligence recognition, models are widely used for data processing and image recognition. Generally, when there are customized model requirements for different application scenarios, technicians choose one of two training methods: 1. Use a pre-trained general model on a common dataset, modify the number of categories in the final output layer, and then readjust the model parameters using their own dataset; 2. Design a simple model and train the model parameters from scratch using their own dataset. The first method, using pre-trained model parameters, can accelerate training convergence and ensure model accuracy, but it requires a long training time for larger models with many parameters. The second method can customize a small model with a simple structure and fewer parameters, but the model parameters need to be trained from scratch, which inevitably slows down the convergence speed and cannot guarantee model accuracy. Moreover, with a small training dataset, it is prone to overfitting.

[0003] Currently, knowledge distillation is used to transfer knowledge learned from the teacher model to the student model, thereby enabling the student model to generalize with the teacher model. However, current knowledge distillation methods generally transfer knowledge learned from the teacher model to a student model. Due to the limited parameters of the student model, it cannot fully learn the knowledge in the teacher model, resulting in poor performance of the generated student model. Therefore, how to improve the training process of knowledge distillation to improve the performance of the student model has become an urgent problem to be solved. Summary of the Invention

[0004] Therefore, it is necessary to provide a model training method, apparatus, device, and storage medium based on knowledge distillation to address the aforementioned technical problems and solve the issue of low model performance during training.

[0005] A first aspect of this application provides a model training method based on knowledge distillation, the method comprising:

[0006] Obtain the first model that satisfies the target conditions and N second models that do not satisfy the target conditions, where N is an integer greater than 1;

[0007] N different initial parameters are obtained through initialization. Each second model is assigned an initial parameter, resulting in N second initial models. The training sample set is input into the second initial model, and the second label probability set is output, resulting in N second label probability sets. The second initial model and the second label probability set are in one-to-one correspondence.

[0008] The second label probability set corresponding to the second initial model is preprocessed, and the preprocessing result is input into the normalization layer to output the third label probability set corresponding to the second initial model.

[0009] Based on the first label probability set obtained from the first model and the training sample set, and the third label probability set corresponding to each second initial model, the loss function corresponding to each second initial model is obtained, and the average loss function corresponding to N second initial models is constructed.

[0010] Based on the average loss function and the loss function corresponding to each second initial model, the target loss function corresponding to each second initial model is obtained. According to the target loss function, the corresponding second initial model is trained, and the initial parameters corresponding to each second initial model are adjusted to obtain N second models that satisfy the target conditions.

[0011] A second aspect of this application provides a model training apparatus based on knowledge distillation, the apparatus comprising:

[0012] The acquisition module is used to acquire the first model that meets the target conditions and N second models that do not meet the target conditions, where N is an integer greater than 1;

[0013] The second label probability set determination module is used to initialize and obtain N different initial parameters, assign an initial parameter to each second model to obtain N second initial models, input the training sample set into the second initial model, and output the second label probability set to obtain N second label probability sets. The second initial model and the second label probability set correspond one-to-one.

[0014] The third label probability set determination module is used to preprocess the second label probability set corresponding to the second initial model, input the preprocessing result into the normalization layer, and output the third label probability set corresponding to the second initial model.

[0015] The average loss function construction module is used to obtain the loss function corresponding to each second initial model based on the first label probability set obtained based on the first model and the training sample set, and the third label probability set corresponding to each second initial model, and to construct the average loss function corresponding to N second initial models.

[0016] The second model determination module is used to obtain the target loss function corresponding to each second initial model based on the average loss function and the loss function corresponding to each second initial model, train the corresponding second initial model according to the target loss function, adjust the initial parameters corresponding to each initial model, and obtain N second models that satisfy the target conditions.

[0017] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the knowledge distillation-based model training method as described in the first aspect.

[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the knowledge distillation-based model training method as described in the first aspect.

[0019] The advantages of this invention compared to the prior art are:

[0020] Obtain a first model that meets the target conditions and N second models that do not, where N is an integer greater than 1. Initialize N different initial parameters, assigning one initial parameter to each second model to obtain N second initial models. Input the training sample set into the second initial models and output a second label probability set, resulting in N second label probability sets. There is a one-to-one correspondence between the second initial models and the second label probability sets. Preprocess the second label probability sets corresponding to the second initial models and input the preprocessing results into a normalization layer to output a third label probability set corresponding to the second initial models. Based on the first label probability set obtained from the first model and the training sample set... We obtain the third label probability set corresponding to each second initial model, the loss function corresponding to each second initial model, and construct the average loss function corresponding to N second initial models. Based on the average loss function and the loss function corresponding to each second initial model, we obtain the target loss function corresponding to each second initial model. According to the target loss function, we train the corresponding second initial model and adjust the initial parameters corresponding to each second initial model to obtain N second models that meet the target conditions. When training the second models, we fuse the information of the N models in the second model to make the information obtained by each second model more comprehensive, thereby improving the performance of the second model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of an application environment for a model training method based on knowledge distillation provided in an embodiment of the present invention;

[0023] Figure 2 This is a schematic flowchart of a model training method based on knowledge distillation provided in an embodiment of the present invention;

[0024] Figure 3 This is a schematic flowchart of a model training method based on knowledge distillation provided in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of a model training device based on knowledge distillation according to an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0029] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0031] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0033] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0034] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0035] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0036] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0037] An embodiment of the present invention provides a model training method based on knowledge distillation, which can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0038] See Figure 2This is a flowchart illustrating a model training method based on knowledge distillation according to an embodiment of the present invention. The above-described model training based on knowledge distillation can be applied to... Figure 1 The server in the example connects to the corresponding client, providing model training services to the client. For example... Figure 2 As shown, the knowledge distillation-based model training method may include the following steps.

[0039] S201: Obtain the first model that meets the target conditions and N second models that do not meet the target conditions.

[0040] In step S201, the first model that meets the target conditions and the N second models that do not meet the target conditions are deep learning convolutional neural network models. The first model that meets the target conditions is obtained by training the first model using a training set, where N is a positive integer. The first model that meets the target conditions is a trained deep learning model, and the second models that do not meet the target conditions are untrained deep learning models. The second models are used to learn the parameters from the first model to save training time.

[0041] In this embodiment, the first model is the teacher model in knowledge distillation, and the second model is the student model in knowledge distillation. To obtain a first model that meets the conditions, the sample data in the training set can first be manually labeled, and then the labeled sample data can be used to train the first model. This training process can be understood as a pre-training process. During training, the loss function of the first model can be used to calculate the loss value between the actual output of the first model and the labeled result. Backpropagation training is then performed based on the loss value until the loss converges, resulting in a first model that meets the conditions. For example, if the above first model is used for voiceprint recognition, the actual output of the first model that meets the conditions can be a voiceprint feature vector.

[0042] It should be noted that the first and second models can be the same type of neural network model, meaning they have the same network layer structure, or they can be different types of neural network models, meaning their network layer structures differ. However, the first model has fewer network layers than the second model, meaning the first model has a smaller model size and fewer parameters. Because the second model has more parameters, outputting prediction results takes longer in practical applications. To improve the speed of prediction output and conserve computing resources, knowledge distillation can be performed on the large model to obtain a lightweight smaller model. In practical applications, model training based on knowledge distillation can extract the feature representations already learned by the complex, high-learning-capability first network model and pass them to the second model, which has fewer parameters and weaker learning capabilities.

[0043] For example, the first model can be built based on the ResNet34 residual network, and the second model can be built based on ResNet10. Larger networks often suffer from the problem of being large and redundant, making it difficult to meet real-time recognition requirements. Smaller networks, on the other hand, are prone to low accuracy due to insufficient parameter count and feature representation capabilities. The problem is that while they may meet the real-time requirements of online applications, they may fail to meet accuracy requirements. Therefore, training the second model with the knowledge from the first model allows the larger network to positively influence the smaller one, enabling the second model to obtain better fitting parameters and thus improving its accuracy.

[0044] S202: Initialize to obtain N different initial parameters, assign an initial parameter to each second model to obtain N second initial models, input the training sample set into the second initial models, output the second label probability set, and obtain N second label probability sets.

[0045] In step S202, initialization parameters are set for each of the N second models, resulting in N second initial models. The initialization parameters are different for each second model, and the number of initialization parameters in each second model is determined by the kernel size, input channel size, output channel size, and bias term size of the corresponding model. For example, if the kernel size of a convolutional neural network is k×k, the input channel is i, the output channel is o, and the bias term is y, then the number of parameters required for each convolution operation is: k×k×i×o+y. Based on the obtained initialized second models, the training sample set is input into the second initial models, and the second label probability sets are output, resulting in N second label probability sets.

[0046] In this embodiment, different initialization parameters can be set for different second models according to different strategies, so that the second models can learn more comprehensive knowledge from the first model when interacting with each other. For example, when the activation function is the saturating activation function tanh, the Xavier initialization method is used to initialize the parameters in the second model. The Xavier initialization method helps to speed up convergence and reduce overfitting. When the activation function is ReLU or its variants, the Kaiming initialization method is used. LeCun initialization is suitable for neural networks with the sigmoid activation function. Its main design idea is to assume that the network input is Gaussian distributed, and during the forward propagation of activation, the expected value and variance of the sampling distribution of the weight parameters during initialization are controlled so that the expected value of the activation value of each neuron is 0 and the variance is 1.

[0047] It should be noted that the purpose of initialization is to avoid gradient explosion or gradient vanishing. A higher requirement is to ensure the stability of both activation forward propagation and gradient backward propagation. That is, to control the mean and variance of neuron activation signals during forward propagation and error signals during backward propagation to be stable. Therefore, when choosing different methods for initialization, we should try to make the mean and variance of neuron activation signals during forward propagation and error signals during backward propagation as stable as possible.

[0048] In this embodiment, after initializing each second model, a second initial model is obtained. The training sample set is input into the second initial model, and a second label probability set is output, resulting in N second label probability sets. Each second label probability set includes the second label probability corresponding to each second initial model. For the same training set, inputting it into different second initial models yields different label probability values.

[0049] Optionally, different initial parameters are set for each second model to obtain N second initial models, including:

[0050] The parameter weights in each second model are calculated using a preset algorithm.

[0051] Based on the parameter weights, an initial parameter is assigned to each second model, resulting in N second initial models.

[0052] In this embodiment, the second model is a neural network model, which includes multiple layers of neurons. Each neuron represents a feature in the training sample. The parameter weights of each neuron are calculated based on the dimension and importance of the features in the training sample. For example, if the dimension of the positive feature vector in the training sample is 5, then the second model has 5 neurons. When calculating the parameter weights of each neuron, the parameter weights are determined based on the features corresponding to each neuron. For example, if the feature vector of the training sample has 5 features, and the third dimension feature has a significant impact on the training sample result, then the largest value from 5 randomly obtained values ​​can be selected as the parameter weight of the neuron corresponding to the third dimension feature. Values ​​are assigned to the parameters of each second model sequentially to obtain N initial second models.

[0053] It should be noted that the parameter values ​​can be obtained from a random function. When assigning parameter values ​​to each neuron, if the corresponding features in each neuron have the same impact on the training results, the same parameter values ​​will be assigned to each neuron.

[0054] S203: Preprocess the second label probability set corresponding to the second initial model, input the preprocessing result into the normalization layer, and output the third label probability set corresponding to the second initial model.

[0055] In step S203, the second label probability set corresponding to the second initial model is preprocessed to ensure that the values ​​in the second label probability set are on a similar scale, which helps to speed up the gradient descent algorithm. The preprocessing result is input into the normalization layer, and the third label probability set corresponding to the second initial model is output.

[0056] In this embodiment, the second label probability set corresponding to the second initial model is preprocessed. This preprocessing involves appropriately scaling the second label probability values ​​to ensure they are at a similar scale. Scaling parameters are set based on the magnitude of the values ​​in the second label probability set. After scaling, the processed label probability values ​​are input to the normalization layer, outputting the third label probability set corresponding to the second initial model. The values ​​in the third label probability set will be on the same order of magnitude, allowing different features from the training samples to be preserved, preventing the features of the corresponding training samples from being ignored when the label probability is low.

[0057] Optionally, the second label probability set corresponding to the second initial model is preprocessed, and the preprocessing result is input into the normalization layer to output the third label probability set corresponding to the second initial model, including:

[0058] Set preprocessing parameters and perform distillation on the second label probability set corresponding to the second initial model to obtain the distilled label probability set.

[0059] The distillation label probability set is input into the normalization layer, and the third label probability set corresponding to the second initial model is output.

[0060] In this embodiment, preprocessing parameters are set to distill the second label probability set corresponding to the second initial model to obtain the distilled label probability set. For example, the parameters can be set according to the difference between the maximum and minimum values ​​in the second label probability set to obtain the order of magnitude of the maximum and minimum values ​​in the second label probability set. Different parameters are set according to the difference in order of magnitude. For example, when the maximum value in the second label probability set is 20 and the minimum value is 1, the parameter can be set to 10 so that the values ​​in the second label probability set can be at a similar scale to obtain the distilled label probability set.

[0061] It should be noted that when setting preprocessing parameters, they can also be determined based on the order of magnitude of the most frequent values ​​within the same order of magnitude in the second label probability set. For example, if the most frequent value within the same order of magnitude in the second label probability set is a two-digit number, the preprocessing parameter can be set to 10; if the most frequent value within the same order of magnitude in the second label probability set is a three-digit number, the preprocessing parameter can be set to 100, and so on. The corresponding distilled label probability set is obtained and input into the normalization layer. The output is the third label probability set corresponding to the second initial model. The normalization layer contains a pre-set normalization function, which can be selected from current normalization or norm normalization, etc.

[0062] S204: Based on the first label probability set obtained from the first model and the training sample set, and the third label probability set corresponding to each second initial model, obtain the loss function corresponding to each second initial model, and construct the average loss function corresponding to N second initial models.

[0063] In step S204, a first label probability set is obtained based on the first model and the training sample set, wherein the first label probability set is a label probability set obtained based on the softening process. A loss function corresponding to each second initial model is constructed based on the first label probability set and the third label probability set. The loss functions corresponding to each second initial model are summed and the mean is calculated to obtain the average loss function corresponding to N second initial models.

[0064] In this embodiment, training samples are input into the first model to obtain the corresponding first label probability set. Based on the first label probability set and the third probability set corresponding to each second initial model, a loss function corresponding to each second initial model is constructed. When constructing the loss function corresponding to each second initial model, the difference between the first label probability set and the third probability set corresponding to the second initial model can be used to construct the loss function corresponding to each second initial model. Based on the loss function corresponding to each second initial model, the average value of the parameters of the N loss functions is calculated to obtain the average loss function.

[0065] Optionally, based on the first label probability set obtained from the first model and the training sample set, and the third label probability set corresponding to each second initial model, a loss function is obtained for each second initial model, including:

[0066] The training sample set is input into the first model, and the first label probability set corresponding to the training sample set is output.

[0067] Based on the first label probability set and the third label probability set corresponding to each second initial model, the loss function corresponding to each second initial model is obtained through the relative entropy algorithm.

[0068] In this embodiment, the training sample set is input into the first model, and the output is the first label probability set corresponding to the training sample set. Based on the first label probability set and the third label probability set corresponding to each second initial model, the loss function corresponding to each second initial model is obtained through the relative entropy algorithm. The relative entropy algorithm realizes the quantification of the matching degree between each predicted value and the true value. Here, the label in the first label probability set is taken as the true value, and the label in the third label set corresponding to each second initial model is taken as the predicted value. The value of relative entropy should be between 0 and infinity, so it can be used as a measure of the distance between two distributions. It can be considered as a measure of the distance between probability distributions. Especially when the difference between the two distributions is very small, it can be considered as the distance between local distributions. The smaller the relative entropy, the better the matching effect between the true value and the predicted value. The loss function corresponding to each second initial model is obtained through the relative entropy algorithm.

[0069] S205: Based on the average loss function and the loss function corresponding to each second initial model, obtain the target loss function corresponding to each second initial model. According to the target loss function, train the corresponding second initial model, adjust the initial parameters corresponding to each second initial model, and obtain N second models that satisfy the target conditions.

[0070] In step S205, the target loss function for each second initial model is the sum of the average loss function and the loss function for each second initial model. When training each second model using the training set, supervised training can be performed. A corresponding threshold is set. When the difference between the training result and the supervision label is less than the threshold, the target loss function is considered to have converged, resulting in N second models that meet the target conditions.

[0071] In this embodiment, when training each initial second model using the training sample set, the training result obtained from the target loss function in each initial second model is used to calculate the loss value with the labels in the training sample set. It is then determined whether the loss value meets the preset conditions. If the preset conditions are not met, backpropagation is performed on each initial second model based on the loss value to obtain a second initial model with updated model parameters. The second initial model with updated model parameters is then trained again based on the training set until the loss value meets the preset conditions. This process is repeated to obtain N second models that meet the target conditions.

[0072] Optionally, based on the average loss function and the loss function corresponding to each second initial model, the target loss function corresponding to each second initial model is obtained, including:

[0073] Based on the training sample set and the probability set of each second label, a loss function corresponding to the second initial model is constructed using a preset algorithm;

[0074] Based on the average loss function and the loss function corresponding to each second initial model, the target loss function corresponding to each second initial model is obtained.

[0075] In this embodiment, the training sample set is input into each second initial model, and the predicted value corresponding to the training sample set is output. Based on the training sample set and the predicted value, the loss function of each second initial model is constructed using the cross-entropy algorithm. The average loss function is summed with the loss function corresponding to each second initial model to obtain the target loss function of each second initial model.

[0076] It should be noted that the predicted values ​​corresponding to the output training samples are the label probability values ​​with a scaling parameter of 1, that is, the label probability values ​​obtained by each second initial model are the true label probability values ​​of the output.

[0077] Optionally, based on the average loss function and the loss function corresponding to each second initial model, the target loss function corresponding to each second initial model is obtained, including:

[0078] Obtain the ratio of the average loss function to the loss function corresponding to each second initial model;

[0079] The target loss function for each second initial model is obtained by weighting and summing the average loss function and the loss function corresponding to each second initial model based on the proportion coefficient.

[0080] In this embodiment, the proportion coefficients of the average loss function and the loss function corresponding to each second initial model are obtained. This is done by selecting the corresponding proportion coefficients from different combinations. For example, during the training of the second initial model, the first model contributes more in the early stages of training, and the average loss function includes the loss function of the first model. Therefore, the proportion coefficient in the average loss function is larger. To obtain the proportion coefficients, a combination of different proportion coefficients is selected, with the larger value used as the proportion coefficient of the average loss function and the smaller value used as the proportion coefficient of the loss function corresponding to the second initial model. Based on the proportion coefficients, a weighted sum is performed on the average loss function and the loss function corresponding to each second initial model to obtain the target loss function corresponding to each second initial model. In the later stages of training, the second initial model contributes more, so the proportion coefficient of the loss function corresponding to the second initial model is larger. When obtaining the proportion coefficient, it is necessary to obtain a combination from different combinations of proportion coefficients, take the larger value as the proportion coefficient of the loss function corresponding to the second initial model, and take the smaller value as the proportion coefficient of the average loss function. Based on the proportion coefficient, the average loss function and the loss function corresponding to each second initial model are weighted and summed to obtain the target loss function corresponding to each second initial model.

[0081] Obtain a first model that meets the target conditions and N second models that do not, where N is an integer greater than 1. Initialize N different initial parameters, assigning one initial parameter to each second model to obtain N second initial models. Input the training sample set into the second initial models and output a second label probability set, resulting in N second label probability sets. There is a one-to-one correspondence between the second initial models and the second label probability sets. Preprocess the second label probability sets corresponding to the second initial models and input the preprocessing results into a normalization layer to output a third label probability set corresponding to the second initial models. Based on the first label probability set obtained from the first model and the training sample set... We obtain the third label probability set corresponding to each second initial model, the loss function corresponding to each second initial model, and construct the average loss function corresponding to N second initial models. Based on the average loss function and the loss function corresponding to each second initial model, we obtain the target loss function corresponding to each second initial model. According to the target loss function, we train the corresponding second initial model and adjust the initial parameters corresponding to each second initial model to obtain N second models that meet the target conditions. When training the second models, we fuse the information of the N models in the second model to make the information obtained by each second model more comprehensive, thereby improving the performance of the second model.

[0082] See Figure 3 This is a flowchart illustrating a model training method based on knowledge distillation according to an embodiment of the present invention, as shown below. Figure 3 The knowledge distillation-based model training method may include the following steps:

[0083] S301: Train the pre-built first model based on the training sample set, adjust the parameters in the first model, and obtain a first model that meets the target conditions.

[0084] In this embodiment, when training the pre-built first model using the training sample set, sample pairs can be used for training to accelerate convergence. Each sample pair selects the same positive and negative samples. Since the magnitude of the backpropagation gradient is determined by the cumulative difference between each positive and negative sample pair, it becomes difficult to find a training boundary in the feature space that satisfies all sample pairs when the number of samples in a sample pair is too large, leading to poorer training convergence. Furthermore, repeatedly calculating the distance difference between each pair of samples as the backpropagation gradient results in training redundancy and increased training time. Because the comparison of the magnitude of a pair of positive and negative samples in different sample pairs may occur multiple times, excessively increasing the number of positive and negative samples in a sample pair provides little benefit to training. Therefore, one positive and one negative sample can be selected from each sample pair. Using the selected sample pairs, the pre-built first model is trained, and the parameters in the first model are adjusted to obtain a first model that satisfies the target conditions.

[0085] The purpose of obtaining a first model that meets the target conditions is to pass the information from the first model to N second models that do not meet the target conditions, thereby accelerating the training of the second models.

[0086] S302: Obtain the first model that satisfies the target condition and N second models that do not satisfy the target condition, where N is an integer greater than 1;

[0087] S303: Initialize to obtain N different initial parameters, assign an initial parameter to each second model to obtain N second initial models, input the training sample set into the second initial models, output the second label probability set to obtain N second label probability sets, and the second initial models correspond one-to-one with the second label probability sets;

[0088] S304: Preprocess the second label probability set corresponding to the second initial model, input the preprocessing result into the normalization layer, and output the third label probability set corresponding to the second initial model;

[0089] S305: Based on the first label probability set obtained from the first model and the training sample set, and the third label probability set corresponding to each second initial model, obtain the loss function corresponding to each second initial model, and construct the average loss function corresponding to N second initial models.

[0090] S306: Based on the average loss function and the loss function corresponding to each second initial model, obtain the target loss function corresponding to each second initial model. According to the target loss function, train the corresponding second initial model, adjust the initial parameters corresponding to each second initial model, and obtain N second models that satisfy the target conditions.

[0091] The steps S302 to S306 are the same as the steps S201 to S205, and can be referred to the description of steps S201 to S205, which will not be repeated here.

[0092] Please see Figure 4 , Figure 4 This is a schematic diagram of a model training device based on knowledge distillation provided in an embodiment of the present invention. In this embodiment, the terminal includes units used for execution... Figures 2 to 3 The steps in the corresponding embodiments. Please refer to the details. Figures 2 to 3 as well as Figures 2 to 3 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 4 The training device 40 includes: an acquisition module 41, a second label probability set determination module 42, a third label probability set determination module 43, an average loss function construction module 44, and a second model determination module 45.

[0093] The acquisition module 41 is used to acquire the first model that meets the target conditions and N second models that do not meet the target conditions, where N is an integer greater than 1.

[0094] The second label probability set determination module 42 is used to initialize and obtain N different initial parameters, assign an initial parameter to each second model, obtain N second initial models, input the training sample set into the second initial model, output the second label probability set, and obtain N second label probability sets. The second initial model and the second label probability set correspond one-to-one.

[0095] The third label probability set determination module 43 is used to preprocess the second label probability set corresponding to the second initial model, input the preprocessing result into the normalization layer, and output the third label probability set corresponding to the second initial model.

[0096] The average loss function construction module 44 is used to obtain the loss function corresponding to each second initial model based on the first label probability set obtained from the first model and the training sample set, and the third label probability set corresponding to each second initial model, and to construct the average loss function corresponding to N second initial models.

[0097] The second model determination module 45 is used to obtain the target loss function corresponding to each second initial model based on the average loss function and the loss function corresponding to each second initial model, train the corresponding second initial model according to the target loss function, adjust the initial parameters corresponding to each initial model, and obtain N second models that meet the target conditions.

[0098] Optionally, the second label probability set determination module 42 mentioned above includes:

[0099] The calculation unit is used to calculate the parameter weights in each second model using a preset algorithm.

[0100] The assignment unit is used to assign an initial parameter to each second model according to the parameter weights, resulting in N second initial models.

[0101] Optionally, the third label probability set determination module 43 mentioned above includes:

[0102] The preprocessing unit is used to set preprocessing parameters and perform distillation processing on the second label probability set corresponding to the second initial model to obtain the distilled label probability set.

[0103] The output unit is used to input the distillation label probability set into the normalization layer and output the third label probability set corresponding to the second initial model.

[0104] Optionally, the above-mentioned average loss function construction module 44 includes:

[0105] The first label probability set determination unit is used to input the training sample set into the first model and output the first label probability set corresponding to the training sample set.

[0106] The loss function determination unit corresponding to the second initial model is used to obtain the loss function corresponding to each second initial model by using the relative entropy algorithm based on the first label probability set and the third label probability set corresponding to each second initial model.

[0107] Optionally, the second model determination module 45 mentioned above includes:

[0108] The building unit is used to construct the loss function corresponding to the second initial model based on the training sample set and each second label probability set using a preset algorithm.

[0109] The target loss function determination unit is used to obtain the target loss function corresponding to each second initial model based on the average loss function and the loss function corresponding to each second initial model.

[0110] Optionally, the target loss function determination unit mentioned above includes:

[0111] Obtain sub-units to obtain the proportion coefficients of the average loss function and the loss function corresponding to each second initial model.

[0112] The weighted summation subunit is used to perform a weighted summation of the average loss function and the loss function corresponding to each second initial model based on the proportion coefficient, so as to obtain the target loss function corresponding to each second initial model.

[0113] Optionally, the above-mentioned training device also includes:

[0114] The training module is used to train a pre-built first model based on a training sample set, adjust the parameters in the first model, and obtain a first model that meets the target conditions.

[0115] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0116] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 5 As shown, the computer device of this embodiment includes: at least one processor ( Figure 5 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, wherein the processor executes the computer program to implement the steps in any of the above embodiments of the knowledge distillation-based model training methods.

[0117] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0118] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0119] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0120] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0121] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0122] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0123] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0124] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0126] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A model training method based on knowledge distillation, characterized in that, The training method includes: Obtain a first model that meets the target conditions and N second models that do not meet the target conditions, where N is an integer greater than 1; the first model and the second models are used for voiceprint recognition; N different initial parameters are obtained through initialization. Each second model is assigned an initial parameter, resulting in N second initial models. The training sample set is input into the second initial model, and the second label probability set is output, resulting in N second label probability sets. The second initial model and the second label probability set are in one-to-one correspondence. The second label probability set is the probability set of voiceprint feature vectors. The second label probability set corresponding to the second initial model is preprocessed, and the preprocessing result is input into the normalization layer to output the third label probability set corresponding to the second initial model; the preprocessing is to scale the second label probability set; the third label probability set is a voiceprint feature vector probability set. Based on the difference between the first label probability set obtained from the first model and the training sample set, and the third probability set corresponding to the second initial model, a loss function corresponding to each second initial model is obtained, and N average loss functions corresponding to the second initial models are constructed; the first label probability set is a voiceprint feature vector probability set; Based on the average loss function and the loss function corresponding to each second initial model, the target loss function corresponding to each second initial model is obtained. According to the target loss function, the corresponding second initial model is trained, and the initial parameters corresponding to each second initial model are adjusted to obtain N second models that satisfy the target conditions. The step of obtaining the target loss function for each second initial model based on the average loss function and the loss function corresponding to each second initial model includes: Based on the training sample set and each second label probability set, a loss function corresponding to the second initial model is constructed using a preset algorithm; The average loss function is summed with the loss function corresponding to each second initial model to obtain the target loss function corresponding to each second initial model; The average loss function is summed with the loss function corresponding to each second initial model to obtain the target loss function corresponding to each second initial model, including: Obtain the ratio of the average loss function to the loss function corresponding to each second initial model; The average loss function and the loss function corresponding to each second initial model are weighted and summed according to the proportion coefficient to obtain the target loss function corresponding to each second initial model.

2. The model training method based on knowledge distillation as described in claim 1, characterized in that, The initialization process yields N different initial parameters. Each second model is assigned an initial parameter, resulting in N second initial models, including: The parameter weights in each second model are calculated using a preset algorithm. Based on the parameter weights, an initial parameter is assigned to each second model, resulting in N second initial models.

3. The model training method based on knowledge distillation as described in claim 1, characterized in that, The step of preprocessing the second label probability set corresponding to the second initial model, inputting the preprocessing result into the normalization layer, and outputting the third label probability set corresponding to the second initial model includes: Set preprocessing parameters and perform distillation on the second label probability set corresponding to the second initial model to obtain the distilled label probability set. The distillation label probability set is input into the normalization layer, and the third label probability set corresponding to the second initial model is output.

4. The model training method based on knowledge distillation as described in claim 1, characterized in that, The step of obtaining the loss function corresponding to each second initial model based on the first label probability set obtained from the first model and the training sample set, and the third label probability set corresponding to each second initial model, includes: The training sample set is input into the first model, and the first label probability set corresponding to the training sample set is output. Based on the first label probability set and the third label probability set corresponding to each second initial model, the loss function corresponding to each second initial model is obtained through the relative entropy algorithm.

5. The model training method based on knowledge distillation as described in claim 1, characterized in that, Before obtaining the first model that satisfies the target conditions, the process also includes: The first model is trained based on the training sample set, and the parameters in the first model are adjusted to obtain a first model that meets the target conditions.

6. A model training device based on knowledge distillation, characterized in that, The device includes: The acquisition module is used to acquire a first model that meets the target conditions and N second models that do not meet the target conditions, where N is an integer greater than 1; the first model and the second model are used for voiceprint recognition; The second label probability set determination module is used to initialize N different initial parameters, assign an initial parameter to each second model to obtain N second initial models, input the training sample set into the second initial models, and output the second label probability set to obtain N second label probability sets. The second initial models and the second label probability sets correspond one-to-one. The second label probability set is a voiceprint feature vector probability set. The third label probability set determination module is used to preprocess the second label probability set corresponding to the second initial model, input the preprocessing result into the normalization layer, and output the third label probability set corresponding to the second initial model; the preprocessing is to scale the second label probability set; the third label probability set is a voiceprint feature vector probability set. The average loss function construction module is used to obtain the loss function corresponding to each second initial model based on the difference between the first label probability set obtained from the first model and the training sample set and the third probability set corresponding to the second initial model, and to construct N average loss functions corresponding to the second initial models; the first label probability set is the voiceprint feature vector probability set; The second model determination module is used to obtain the target loss function corresponding to each second initial model based on the average loss function and the loss function corresponding to each second initial model, train the corresponding second initial model according to the target loss function, adjust the initial parameters corresponding to each initial model, and obtain N second models that meet the target conditions. The step of obtaining the target loss function for each second initial model based on the average loss function and the loss function corresponding to each second initial model includes: Based on the training sample set and each second label probability set, a loss function corresponding to the second initial model is constructed using a preset algorithm; The average loss function is summed with the loss function corresponding to each second initial model to obtain the target loss function corresponding to each second initial model; The average loss function is summed with the loss function corresponding to each second initial model to obtain the target loss function corresponding to each second initial model, including: Obtain the ratio of the average loss function to the loss function corresponding to each second initial model; The average loss function and the loss function corresponding to each second initial model are weighted and summed according to the proportion coefficient to obtain the target loss function corresponding to each second initial model.

7. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the knowledge distillation-based model training method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the knowledge distillation-based model training method as described in any one of claims 1 to 5.

Images