Model distillation method, device and medium

Abstract

The invention discloses a model distillation method and device and a medium. The method comprises the steps that a teacher similarity matrix of a teacher model, a student similarity matrix of a student model and a sorting difference matrix representing the similarity sorting difference of the teacher model and the student model are acquired; calculation precision of the teacher model is higher than that of the student model. Constructing a contrast learning sample pair according to the teacher similarity matrix, and constructing a sorting difference sample pair according to the sorting difference matrix; constructing a distillation loss function based on the contrast learning sample pair and the sorting difference sample pair; and performing iterative training on the student model through the distillation loss function to obtain a target student model. Therefore, the contrast learning sample pair and the sorting difference sample pair are constructed based on the teacher model, the distillation loss function is constructed, and the student model is iteratively trained through the distillation loss function, so that the student model can learn the knowledge of the teacher model, the accuracy of the student model is improved, and the calculation efficiency and the calculation accuracy are considered at the same time.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model distillation method, device and medium. Background Art

[0002] With the rapid development of deep learning technology, neural network models are widely used in various fields such as autonomous driving, image processing, natural language processing, computer vision, etc. In the actual application of neural network models, the higher the calculation accuracy and complexity of the model, the slower the model will run, and it may even fail to run on mobile or embedded devices.

[0003] Therefore, how to balance the model calculation accuracy and running speed and expand the scope of model application is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0004] In view of this, one aspect of the present application provides a model distillation method, the method comprising: Obtaining a teacher similarity matrix of the teacher model, a student similarity matrix of the student model, and a ranking difference matrix representing the ranking difference of similarity between the teacher model and the student model; the calculation accuracy of the teacher model is higher than the calculation accuracy of the student model; According to the teacher similarity matrix, constructing comparative learning sample pairs; and according to the sorting difference matrix, constructing sorting difference sample pairs; Constructing a distillation loss function based on the contrastive learning sample pair and the sorting difference sample pair; The student model is iteratively trained using the distillation loss function to obtain a target student model.

[0005] Optionally, constructing a distillation loss function based on the contrastive learning sample pair and the sorting difference sample pair includes: Obtaining the task type of the student model; Determining a label loss function of the student model according to the task type; Determine a learning contrast loss function based on the contrast learning sample pair; and determine a sorting difference loss function based on the sorting difference sample pair; The distillation loss function is determined according to the label loss function, the learning contrast loss function and the sorting difference loss function.

[0006] Optionally, determining a sorting difference loss function according to the sorting difference sample pairs includes: Determine a teacher ranking matrix, a student ranking matrix and a similarity difference matrix according to the teacher similarity matrix and the student similarity matrix; Determining a ranking difference weight according to the teacher ranking matrix, the student ranking matrix and the ranking difference sample pairs; Determining a similarity difference weight according to the similarity difference matrix and the ranked difference sample pairs; Determining a priori weights of a teacher model according to the teacher similarity matrix and the ranked difference sample pairs; Determine a difference loss weight by using the ranking difference weight, the similarity difference weight and the teacher model prior weight; The sorting difference loss function is determined according to the difference loss weight and the sorting difference sample pair.

[0007] Optionally, when the ranking difference between the teacher ranking matrix and the student ranking matrix between the same ranking difference sample pairs is greater, the ranking difference weight is greater; When the similarity difference between the teacher similarity matrix and the student similarity matrix in the same sorted difference sample pair is greater, the similarity difference weight is greater; The smaller the similarity of the sorted difference sample pair in the teacher similarity matrix, the greater the prior weight of the teacher model.

[0008] Optionally, determining a difference loss weight by using the ranking difference weight, the similarity difference weight and the teacher model prior weight includes: Assigning a first weight to the ranking difference weight; assigning a second weight to the similarity difference weight; and assigning a third weight to the teacher model prior weight; The difference loss weight is determined according to the first weight, the second weight, the third weight, the sorting difference weight, the similarity difference weight and the teacher model prior weight; wherein, as the current iteration number of the iterative training continues to increase, the larger the first weight is, the smaller the second weight and the third weight are.

[0009] Optionally, constructing a comparative learning sample pair according to the teacher similarity matrix includes: The samples corresponding to the similarity greater than the first threshold in the teacher similarity matrix are taken as positive sample pairs, and the samples corresponding to the similarity less than the second threshold are taken as negative sample pairs; wherein the first threshold is greater than the second threshold, and as the current iteration number of the iterative training increases, the greater the first threshold, the smaller the second threshold; The positive sample and the negative sample constitute the contrastive learning sample pair.

[0010] Optionally, constructing a sorting difference sample pair according to the sorting difference matrix includes: Determine the target object that meets the preset conditions in the sorting difference matrix; wherein the preset conditions include the similarity sorting difference being greater than a preset value, the first preset quantile in the sorting result of the similarity sorting difference from large to small, and any one of the first preset quantiles of the sorting result.

[0011] The samples corresponding to the target object are used as the sorting difference sample pairs.

[0012] Another aspect of the present application provides a model distillation device, the device comprising: A matrix acquisition module, used to acquire a teacher similarity matrix of a teacher model, a student similarity matrix of a student model, and a ranking difference matrix representing the ranking difference of similarity between the teacher model and the student model; the calculation accuracy of the teacher model is higher than the calculation accuracy of the student model; A sample pair construction module, used to construct comparative learning sample pairs according to the teacher similarity matrix; and to construct sorting difference sample pairs according to the sorting difference matrix; A distillation loss function construction module, used to construct a distillation loss function based on the contrastive learning sample pair and the sorting difference sample pair; The student model training module is used to iteratively train the student model through the distillation loss function to obtain a target student model.

[0013] Another aspect of the present application provides a model distillation device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and the processor implements the steps of the model distillation method when executing the program.

[0014] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the model distillation method when the program is executed by a processor.

[0015] The present application provides a model distillation method, device and medium, which have the following beneficial effects: constructing comparative learning sample pairs and sorting difference sample pairs based on a teacher model, building a distillation loss function, and iteratively training the student model through the distillation loss function, so that the student model with lower accuracy can learn the knowledge of the high-precision teacher model and obtain performance close to the teacher model at a lower computing cost, thereby achieving the purpose of improving the accuracy of the student model, achieving both computational efficiency and computational accuracy of the student model, and expanding the scope of application of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a model distillation method provided in an embodiment of the present application; Figure 2A schematic diagram of the principle of a model distillation method provided in an embodiment of the present application; Figure 3 A schematic flow chart of a model distillation method provided in another embodiment of the present application; Figure 4 A schematic diagram of the structure of a model distillation device provided in an embodiment of the present application; Figure 5 A schematic structural diagram of a model distillation apparatus provided in another embodiment of the present application.

[0017] The reference numerals are as follows: 50 is a memory, 51 is a processor, 52 is a display screen, 53 is an input / output interface, 54 is a communication interface, 55 is a power supply, 56 is a communication bus, 501 is a computer program, 502 is an operating system, and 503 is data. DETAILED DESCRIPTION

[0018] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0019] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0020] Figure 1 A schematic diagram of a model distillation method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes: S10: obtaining a teacher similarity matrix of the teacher model, a student similarity matrix of the student model, and a ranking difference matrix representing the ranking difference of similarity between the teacher model and the student model; the calculation accuracy of the teacher model is higher than the calculation accuracy of the student model; In a specific embodiment, the computational accuracy of the teacher model is higher than that of the student model, that is, the teacher model has a higher model complexity and its performance is significantly better than that of the student model. Therefore, in practical applications, the computing resources required by the teacher model are usually relatively high, and it is usually difficult to deploy directly on resource-constrained devices, while the computing resources required by the student model are relatively low and can be deployed directly. In order to enable the student model to have high running speed and high precision at the same time, the student model can learn the knowledge of the teacher model, so as to approach the performance of the teacher model.

[0021] Figure 2 A schematic diagram of the principle of a model distillation method provided in an embodiment of the present application, specifically, the number of samples is Input sample Input the teacher model and student model respectively to obtain the teacher output result of the teacher model And the student output results of the student model .in, , , It is used to represent the number of input samples and the number of output results.

[0022] Furthermore, according to the teacher's output results The teacher similarity matrix of the teacher model can be generated , according to the student output results The student similarity matrix of the student model can be generated ,in, It should be noted that the similarity matrix is ​​composed of the similarity values ​​between the output results, which is used to characterize the degree of difference between the output samples. For example, in the teacher similarity matrix In the The input sample and The difference between the input samples is , it means that the similarity between the first input sample and the third input sample is 0.7, which means the similarity is high. Figure 2 As shown, after the input samples are input into the teacher model and the student model, the teacher similarity matrix and the student similarity matrix can be constructed respectively.

[0023] In an optional embodiment, obtaining the teacher similarity matrix Similarity matrix with students , specifically, enter , take the output result corresponding to the current input sample as the basis vector, take the output results of other samples as the target vector, calculate the similarity values ​​between the basis vector and the target vector in turn, and after traversing all input samples, construct the similarity matrix through all similarity values. In fact, it can be understood as calculating the similarity values ​​between the output results of each pair to form a similarity matrix.

[0024] It should be noted that when calculating the similarity value between the basis vector and the target vector, the calculation can be performed using, but not limited to, metric functions such as cosine similarity and Euclidean distance.

[0025] Furthermore, the teacher similarity matrix Similarity matrix with students By subtracting, we can get the similarity matrix used to describe teachers Similarity matrix with students The similarity difference matrix of the similarity difference .in, , the similarity matrix is ​​symmetric, , the diagonal elements are 0. It should be noted that in the similarity difference matrix In the paper, the relationship between the teacher model and the student model input samples at the same position (i.e., in The input sample and That is, it describes the degree of similarity difference between the input samples at the same position in the teacher similarity matrix and the student similarity matrix.

[0026] In addition, if Figure 2 As shown, the teacher ranking matrix can also be obtained based on the teacher similarity matrix , and obtain the student ranking matrix based on the student similarity matrix In the teacher ranking matrix In the The input sample is the reference sample, The input sample is relative to the The similarity ranking position of the input samples. For example, =1, which means that the third sample is closest to the first sample, that is, the similarity ranking position is the first. =3, which describes that the first sample is the reference sample, and the third sample is ranked third in similarity, that is, the third most similar. Student ranking matrix Similarly, it can be understood that since the sorting matrix is ​​used to describe the similarity ranking between two input samples, all of them are integers representing the ranking positions in the sorting matrix.

[0027] Furthermore, the teacher ranking matrix and the student ranking matrix After the difference, we can get the ranking difference matrix used to characterize the ranking difference of similarity between the teacher model and the student model. .in, , the diagonal elements of the batch sample sorting matrix are 0. In the sorting difference matrix In the paper, the relationship between the teacher model and the student model input samples at the same position (i.e., in The input sample and That is, it describes the degree of difference in similarity ranking between the input samples at the same position in the teacher ranking matrix and the student ranking matrix.

[0028] S11: construct comparative learning sample pairs according to the teacher similarity matrix; and construct sorting difference sample pairs according to the sorting difference matrix; S12: Construct a distillation loss function based on contrastive learning sample pairs and sorting difference sample pairs; S13: Iteratively train the student model through the distillation loss function to obtain the target student model.

[0029] In order to learn the performance of the teacher model, a comparative learning sample pair is constructed according to the teacher similarity matrix. Specifically, the samples corresponding to the preset similarity conditions in the teacher similarity matrix are used as comparative learning sample pairs. In addition, in an optional embodiment, the sorted difference sample pairs are constructed by sorting the samples corresponding to the preset conditions in the sorted difference matrix.

[0030] Furthermore, a distillation loss function is constructed based on the constructed contrast learning sample pairs and sorting difference sample pairs. It can be understood that model distillation is a technique that transfers the knowledge of a complex model (ie, the teacher model) to a small simple model (ie, the student model). In this way, the student model can achieve performance close to that of the teacher model while maintaining a low computational cost. Therefore, it is necessary to construct a distillation loss function to iteratively train the student model to obtain a target student model with a high operating rate and high precision. It should be noted that the iterative training of the student model can be trained to a preset number of iterations, and this application does not limit the iteration termination conditions.

[0031] That is, the target student model trained by the constructed distillation loss function has a good running speed and performance close to that of the teacher model. It also has the characteristics of high precision and high rate, so that a large number of mobile terminals and embedded devices can run the target student model, thereby achieving the purpose of expanding the scope of model use.

[0032] Therefore, the model distillation method provided in the embodiment of the present application constructs a distillation loss function based on the teacher model to construct comparative learning sample pairs and sorted difference sample pairs, and iteratively trains the student model through the distillation loss function, so that the student model with lower precision can learn the knowledge of the high-precision teacher model and obtain performance close to the teacher model at a lower computational cost, thereby achieving the purpose of improving the accuracy of the student model, realizing that the student model takes into account both computational efficiency and computational accuracy, and expanding the scope of application of the model.

[0033] In an optional embodiment, a distillation loss function is constructed based on contrastive learning sample pairs and sorting difference sample pairs, including: Get the task type of the student model; Determine the label loss function of the student model based on the task type; Determine a learning contrast loss function based on the contrast learning sample pairs; and determine a sorting difference loss function based on the sorting difference sample pairs; The distillation loss function is determined based on the label loss function, the learning contrast loss function and the sorting difference loss function.

[0034] In a specific embodiment, Figure 2 As shown, the distillation loss function It is a comprehensive loss function, including the learning contrast loss function determined based on the contrastive learning sample pairs , and the sorting difference loss function determined based on the sorting difference sample pairs .

[0035] Among them, learning contrast loss function It is mainly used for contrastive learning and is a self-supervised learning method. This function encourages the representation of similar samples to be closer, while the representation of dissimilar samples to be farther away. By introducing contrastive learning in the model distillation process, the student model can not only imitate the prediction results of the teacher model, but also imitate the internal representation or feature space structure of the teacher model, so that the student model is close to the performance of the teacher model. The ranking difference loss function It can be used to ensure that the student model maintains a similarity ranking to the teacher model, thereby further ensuring that the student model is as close to the teacher model as possible.

[0036] It should be noted that the learning contrast loss function It may include but is not limited to normalized temperature scale cross entropy loss and conventional contrastive learning loss. In an optional embodiment, the learning contrast loss function You can choose the normalized temperature scale cross entropy loss, which is used to maximize the similarity of positive sample pairs in the contrastive learning sample pairs and minimize the similarity of negative sample pairs. According to the contrastive learning sample pairs, the specific calculation formula for determining the learning contrast loss function is: (1) in, To learn the contrastive loss function, To compare the positive sample pairs in the learning sample pairs, To compare the negative sample pairs in the learning sample pairs, is the number of positive sample pairs, , and The student model , and The output result corresponding to the input samples. The similarity function is, in an optional embodiment, it can be expressed by cosine similarity, which is directly obtained from the student similarity matrix. Used to calculate the similarity of positive sample pairs, Used to calculate the similarity between positive sample pairs and negative sample pairs. is the temperature parameter, which is used to control the smoothness of the distribution.

[0037] It should be noted that the positive sample pairs in the comparative learning sample pairs refer to the input samples corresponding to the teacher similarity matrix whose similarity is greater than the first threshold, and the positive sample pairs refer to the input samples corresponding to the teacher similarity matrix whose similarity is less than the second threshold, and the first threshold is greater than the second threshold.

[0038] In an optional embodiment, in order to ensure that the student model does not deviate too far from the basic information provided by the true label when learning the knowledge of the teacher model, Figure 2 As shown, the label loss function is introduced . The label loss function can be It is understood as a task loss function, that is, different loss functions are selected according to different task types. In a specific embodiment, a distillation loss function is constructed. When , get the task type of the student model and select the label loss function for the task type .

[0039] It is understandable that the label loss function It mainly depends on the specific task type of the student model. In an optional embodiment, when the task type is a classification task, the cross-entropy loss (Cross-EntropyLoss) can be used as the label loss function , the specific calculation formula is formula (2): (2) in, is the number of categories for classification, is the true label (if One-hot encoding is used, the target category is 1 and other categories are 0), is the predicted probability of the student model.

[0040] Further, such as Figure 2 As shown, according to the sorting difference sample pairs, the sorting difference loss function is determined In fact, the distillation loss function can be It is understood to include two parts, one of which is to describe the loss between the student model and the true label, that is, the supervision loss of the student model itself, that is, the label loss function The other part is the imitation loss function, that is, the loss function that imitates the output of the teacher model, which specifically includes the learning contrast loss function And the ranking difference loss function Thus, the distillation loss function is obtained The formula is: (3) In a specific embodiment of a CIFAR100 image classification task, a 26-layer residual network is used as the teacher model and a 14-layer residual network is used as the student model. The accuracy of the teacher model is 61.72%, and the accuracy of the student model using conventional training is 58.15%. After training the student model, the accuracy of the target student model can reach 61.08%, which is only 0.64% lower than that of the teacher model. That is, while maintaining a high running speed, the accuracy is greatly improved by learning the performance of the teacher model.

[0041] Figure 3 A schematic flow chart of a model distillation method provided in another embodiment of the present application is as follows: Figure 3 As shown, in an optional embodiment, determining the sorting difference loss function according to the sorting difference sample pairs includes: S30: Determine a teacher ranking matrix, a student ranking matrix and a similarity difference matrix according to the teacher similarity matrix and the student similarity matrix; S31: determining the ranking difference weight according to the teacher ranking matrix, the student ranking matrix and the ranking difference sample pairs; In a specific embodiment, a teacher ranking matrix describing the similarity ranking relationship between different input samples can be generated according to the teacher similarity matrix, that is, the similarity ranking relationship between the input samples is recorded in the teacher ranking matrix. At the same time, a student similarity matrix can be generated according to the student similarity matrix.

[0042] Further, according to the teacher ranking matrix and the student ranking matrix, the similarity ranking difference between the input samples at the same position of the teacher ranking matrix and the student ranking matrix can be obtained. The specific acquisition method can refer to the description of the above embodiment, which will not be repeated here.

[0043] In an optional embodiment, the sorting difference loss function Adopt adaptive weights. Specifically, the sorting difference loss function The difference loss weight Include sort difference weights , similarity difference weight and the teacher model prior weights .

[0044] Among them, the ranking difference weight is determined by step S31 Specifically, according to the teacher ranking matrix, the student ranking matrix and the ranking difference sample pairs, the calculation formula for determining the ranking difference weight is: (4) in, is the sorting difference weight. The teacher ranking matrix is The input sample is the reference sample, The input sample is relative to the The similarity ranking position of the input samples. Sort the students in the matrix by The input sample is the reference sample, The input sample is relative to the The similarity ranking position of the input samples. The input samples and The input samples constitute the sorted difference sample pairs. is the input sample size.

[0045] S32: determining a similarity difference weight according to the similarity difference matrix and the sorted difference sample pairs; In an optional embodiment, according to the similarity difference matrix and the sorted difference sample pairs, a specific calculation formula for determining the similarity difference weight is: (5) in, is the similarity difference weight. is the similarity difference matrix composed of The input samples and The similarity value corresponding to the sorted difference sample pair composed of input samples is used to describe the first similarity matrix in the teacher The input samples and The similarity between the input sample and the student similarity matrix The input samples and The difference in similarity between the input samples. is the maximum similarity value in the similarity difference matrix.

[0046] S33: determining the teacher model prior weight according to the teacher similarity matrix and the sorted difference sample pairs; In an optional embodiment, the specific calculation formula for determining the prior weight of the teacher model is as follows based on the teacher similarity matrix and the sorted difference sample pairs: (6) in, is the teacher model prior weight, is the teacher similarity matrix composed of The input samples and The similarity value corresponding to the sorted difference sample pair composed of input samples, is the maximum similarity value in the teacher similarity matrix.

[0047] S34: determining a difference loss weight by sorting the difference weight, the similarity difference weight, and the teacher model prior weight; S35: Determine a sorting difference loss function according to the difference loss weight and the sorting difference sample pair.

[0048] Furthermore, the sorting difference weight is determined based on the sorting difference sample pairs , similarity difference weight and the teacher model prior weights After that, the sorting difference loss function can be calculated The difference loss weight , and then the sorting difference loss function can be determined according to formula (7) : (7) in, The student model is the first The input samples and The difference between the output results corresponding to the input samples.

[0049] Based on the above embodiment, as an optional embodiment, when the ranking difference between the teacher ranking matrix and the student ranking matrix for the same ranking difference sample pair is greater, the ranking difference weight is greater.

[0050] In a specific embodiment, the ranking difference weight is used to measure the ranking difference between the student model and the teacher model at the same position input sample pair. The input samples and input samples, then the teacher ranking matrix and the student ranking matrix are The input samples and The larger the ranking difference between the input samples, the greater the student model needs to be. The input samples and input samples, so a larger sorting difference weight should be set.

[0051] In another optional embodiment, when the similarity difference between the teacher similarity matrix and the student similarity matrix for the same sorted difference sample pairs is greater, the similarity difference weight is greater.

[0052] In a specific embodiment, the similarity difference weight is used to measure the similarity difference between the input sample pairs of the student model and the teacher model at the same position. When the similarity difference is larger, it indicates that the student model has learned the corresponding knowledge, and a higher similarity difference weight should be set at this time.

[0053] In an optional embodiment, the smaller the similarity between the sorted difference sample pairs in the teacher similarity matrix, the larger the prior weight of the teacher model.

[0054] In a specific embodiment, the teacher model prior weight is determined according to the teacher model similarity matrix. The smaller the similarity in the teacher similarity matrix, that is, the greater the difference in the representation sample pairs, the higher the teacher model prior weight should be assigned.

[0055] In an optional embodiment, the difference loss weight is determined by sorting the difference weight, the similarity difference weight and the teacher model prior weight, including: Assign a first weight to the ranking difference weight; assign a second weight to the similarity difference weight; and assign a third weight to the teacher model prior weight; The difference loss weight is determined according to the first weight, the second weight, the third weight, the sorting difference weight, the similarity difference weight and the teacher model prior weight; wherein, as the current iteration number of iterative training continues to increase, the larger the first weight is, the smaller the second weight and the third weight are.

[0056] In order to further improve the training effect of the target student model, in a specific embodiment, the ranking difference weight, the similarity difference weight and the teacher model prior weight are linearly fused to obtain the difference loss weight. Give first weight ; is the similarity difference weight Give the second weight ; and are the prior weights for the teacher model Give third weight Therefore, the specific calculation formula for determining the difference loss weight is: (8) In an optional embodiment, the sorting difference weight is dynamically adjusted , similarity difference weight and the teacher model prior weights , you can also adjust the first weight , second weight and the third weight Specifically, as the number of iterations of the iterative training increases, the first weight The larger the second weight and the third weight In a specific embodiment, increasing the first weight , can strengthen the learning of difficult samples. The second weight and the third weight , which can make the student model dependent on its own learning results.

[0057] In an optional embodiment, constructing a comparative learning sample pair according to the teacher similarity matrix includes: The samples corresponding to the similarity greater than the first threshold in the teacher similarity matrix are regarded as positive sample pairs, and the samples corresponding to the similarity less than the second threshold are regarded as negative sample pairs; the first threshold is greater than the second threshold, and as the current number of iterations of iterative training increases, the larger the first threshold, the smaller the second threshold; Positive samples and negative samples constitute contrastive learning sample pairs.

[0058] In a specific embodiment, Figure 2 As shown, when constructing comparative learning sample pairs, they are constructed based on the teacher similarity matrix and the preset first screening rule. Among them, the comparative learning sample pairs include positive sample pairs and negative sample pairs. The positive sample pairs refer to the sample pairs corresponding to the teacher similarity matrix with a similarity greater than the first threshold, and the negative sample pairs refer to the samples corresponding to the teacher similarity matrix with a similarity less than the second threshold as negative sample pairs.

[0059] It is understandable that in the teacher similarity matrix In the The input sample and The degree of difference between the input samples. Therefore, when When it is greater than the first threshold, the The input sample and input samples as positive sample pairs, when When the value is less than the second threshold, the The input sample and The input samples are taken as negative sample pairs.

[0060] In an optional embodiment, the first threshold is greater than the second threshold, and as the distillation loss function continues to iterate the student model, the first threshold and the second threshold can be dynamically adjusted. Specifically, when the current number of iterations continues to increase, the larger the first threshold, the smaller the second threshold. In a specific embodiment, in the early stage of iterative training, a looser threshold can be set to ensure that the student model has sufficient learning signals. As the training progresses, the first threshold is gradually increased and the second threshold is lowered, thereby increasing the learning difficulty of the student model so as to improve the performance of the student model.

[0061] In an optional embodiment, a rule for adjusting the first threshold and the second threshold may be pre-set. Specifically, the initial first threshold, the adjustment range of the first threshold, the initial second threshold and the adjustment range of the second threshold are pre-set, the current iteration number of the iterative training of the student model by the distillation loss function is obtained in real time, and the target first threshold and the target second threshold for the next round of training are calculated according to the current iteration number. The specific calculation formulas are formula (9) and formula (10): (9) (10) in, For the The first threshold of the round of iterative training, For the The second threshold for round iterative training, is the initial first threshold, is the initial second threshold, is the total number of iterations, is the adjustment amplitude of the first threshold, is the adjustment amplitude of the second threshold.

[0062] For example, in an optional embodiment, the initial first threshold value may be set to 0.7, and the initial second threshold value may be set to 0.3. In each iterative training, different positive sample pairs and negative sample pairs may be obtained according to different first threshold values ​​and second threshold values, that is, different comparative learning sample pairs including positive sample pairs and negative sample pairs may be obtained.

[0063] In an optional embodiment, constructing a sorting difference sample pair according to the sorting difference matrix includes: Determine the target object that meets the preset conditions in the sorting difference matrix; wherein the preset conditions include any one of the similarity sorting difference being greater than a preset value, the first preset quantile in the sorting result of the similarity sorting difference from large to small, and the first preset number of the sorting result.

[0064] The samples corresponding to the target object are taken as sorted difference sample pairs.

[0065] In a specific embodiment, Figure 2 As shown, when constructing the sorting difference sample pair, it is necessary to first determine the sorting difference matrix based on the teacher sorting matrix and the student sorting matrix. Further, the target object that meets the preset conditions is screened out from the sorting difference matrix. Specifically, it can be understood that the sorting difference matrix describes the degree of similarity ranking difference between the input samples at the same position of the teacher sorting matrix and the student sorting matrix. Therefore, the objects whose similarity ranking difference in the sorting difference matrix is ​​greater than the preset value can be used as target objects.

[0066] In another optional embodiment, the objects corresponding to the first preset quantiles in the sorting results of similarity sorting difference from large to small can also be used as target objects, for example, the first 30% in the sorting results are used as target objects. Of course, the first preset number in the sorting results can also be used as target objects, for example, the first 40 are used as target objects.

[0067] Furthermore, the samples corresponding to the selected target objects can be used as the sorting difference samples. It should be noted that any one of the above preset conditions can be selected.

[0068] In the above embodiments, the model distillation method is described in detail. The present application also provides a corresponding embodiment of a model distillation device.

[0069] Figure 4 A schematic diagram of the structure of a model distillation device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the device comprises: The matrix acquisition module 40 is used to obtain the teacher similarity matrix of the teacher model, the student similarity matrix of the student model, and the ranking difference matrix representing the ranking difference of the similarity between the teacher model and the student model; the calculation accuracy of the teacher model is higher than the calculation accuracy of the student model; The sample pair construction module 41 is used to construct comparative learning sample pairs according to the teacher similarity matrix; and to construct sorting difference sample pairs according to the sorting difference matrix; A distillation loss function construction module 42, used to construct a distillation loss function based on the contrastive learning sample pairs and the sorting difference sample pairs; The student model training module 43 is used to iteratively train the student model through the distillation loss function to obtain a target student model.

[0070] In addition, the model distillation apparatus provided in the embodiment of the present application also includes: The task type acquisition module is used to obtain the task type of the student model; The label loss function determination module is used to determine the label loss function of the student model according to the task type; A first determination module is used to determine a learning contrast loss function based on a contrast learning sample pair; and to determine a sorting difference loss function based on a sorting difference sample pair; The second determination module is used to determine the distillation loss function according to the label loss function, the learning contrast loss function and the sorting difference loss function.

[0071] A ranking matrix determination module is used to determine a teacher ranking matrix, a student ranking matrix and a similarity difference matrix according to a teacher similarity matrix and a student similarity matrix; A ranking difference weight determination module is used to determine the ranking difference weight according to the teacher ranking matrix, the student ranking matrix and the ranking difference sample pair; A similarity difference weight determination module is used to determine the similarity difference weight according to the similarity difference matrix and the sorted difference sample pairs; A teacher model prior weight determination module is used to determine the teacher model prior weight according to the teacher similarity matrix and the sorted difference sample pairs; A difference loss weight determination module, used to determine the difference loss weight by sorting difference weight, similarity difference weight and teacher model prior weight; The second determination module is further used to determine the sorting difference loss function according to the difference loss weight and the sorting difference sample pair.

[0072] A weight assignment model is used to assign a first weight to the ranking difference weight; assign a second weight to the similarity difference weight; and assign a third weight to the teacher model prior weight; The difference loss weight determination module is also used to determine the difference loss weight based on the first weight, the second weight, the third weight, the sorting difference weight, the similarity difference weight and the teacher model prior weight; wherein, as the current number of iterations of iterative training continues to increase, the larger the first weight, the smaller the second weight and the third weight.

[0073] The comparative learning sample pair acquisition module is used to take the samples corresponding to the similarity greater than the first threshold in the teacher similarity matrix as positive sample pairs, and the samples corresponding to the similarity less than the second threshold as negative sample pairs; the first threshold is greater than the second threshold, and as the current iteration number of iterative training increases, the larger the first threshold, the smaller the second threshold; the positive samples and negative samples constitute the comparative learning sample pairs.

[0074] The sorting difference sample pair acquisition module is used to determine the target object that meets the preset conditions in the sorting difference matrix; wherein the preset conditions include the similarity sorting difference being greater than the preset value, the first preset quantile in the sorting result of the similarity sorting difference from large to small, and any one of the first preset numbers of the sorting result.

[0075] The samples corresponding to the target object are taken as sorted difference sample pairs.

[0076] Figure 5 A schematic diagram of the structure of a model distillation device provided in another embodiment of the present application is shown in FIG. Figure 5 As shown, the model distillation apparatus comprises: a memory 50 for storing a computer program; The processor 51 is used to implement the steps of the model distillation method mentioned in the above embodiment when executing the computer program.

[0077] The model distillation apparatus provided in this embodiment may include, but is not limited to, a laptop computer or a desktop computer.

[0078] Among them, the processor 51 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 51 can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processor, referred to as DSP), a field programmable gate array (Field-Programmable Gate Array, referred to as FPGA), and a programmable logic array (Programmable Logic Array, referred to as PLA). The processor 51 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (Central Processing Unit, referred to as CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 51 may be integrated with a graphics processing unit (Graphics Processing Unit, referred to as GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 51 may also include an artificial intelligence (Artificial Intelligence, referred to as AI) processor, which is used to process computing operations related to machine learning.

[0079] The memory 50 may include one or more computer-readable storage media, which may be non-transitory. The memory 50 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 50 is at least used to store the following computer program 501, wherein the computer program, after being loaded and executed by the processor 51, can implement the relevant steps of the model distillation method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 50 may also include an operating system 502 and data 503, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 502 may include Windows, Unix, Linux, etc. Data 503 may include, but is not limited to, relevant data involved in the model distillation method, etc.

[0080] In some embodiments, the model distillation device may further include a display screen 52 , an input / output interface 53 , a communication interface 54 , a power supply 55 , and a communication bus 56 .

[0081] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation of the model distillation apparatus, which may include more or fewer components than shown.

[0082] The model distillation device provided in the embodiment of the present application includes a memory and a processor. When the processor executes the program stored in the memory, the model distillation device in the above embodiment can be implemented.

[0083] It should be noted that, although the operations are depicted in a specific order in the accompanying drawings, this should not be understood as requiring these operations to be performed in the specific order shown or to be performed sequentially, or requiring all illustrated operations to be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

Claims

1. A model distillation method, characterized in that: The method comprises: Obtaining a teacher similarity matrix of the teacher model, a student similarity matrix of the student model, and a ranking difference matrix representing the ranking difference of similarity between the teacher model and the student model; the calculation accuracy of the teacher model is higher than the calculation accuracy of the student model; According to the teacher similarity matrix, constructing comparative learning sample pairs; and according to the sorting difference matrix, constructing sorting difference sample pairs; Constructing a distillation loss function based on the contrastive learning sample pair and the sorting difference sample pair; The student model is iteratively trained using the distillation loss function to obtain a target student model.

2. The model distillation method according to claim 1, characterized in that Based on the contrastive learning sample pair and the sorting difference sample pair, a distillation loss function is constructed, including: Obtaining the task type of the student model; Determining a label loss function of the student model according to the task type; Determine a learning contrast loss function based on the contrast learning sample pair; and determine a sorting difference loss function based on the sorting difference sample pair; The distillation loss function is determined according to the label loss function, the learning contrast loss function and the sorting difference loss function.

3. The model distillation method according to claim 2, characterized in that: Determining a sorting difference loss function according to the sorting difference sample pair includes: Determine a teacher ranking matrix, a student ranking matrix and a similarity difference matrix according to the teacher similarity matrix and the student similarity matrix; Determining a ranking difference weight according to the teacher ranking matrix, the student ranking matrix and the ranking difference sample pairs; Determining a similarity difference weight according to the similarity difference matrix and the ranked difference sample pairs; Determining a priori weights of a teacher model according to the teacher similarity matrix and the ranked difference sample pairs; Determine a difference loss weight by using the ranking difference weight, the similarity difference weight and the teacher model prior weight; The sorting difference loss function is determined according to the difference loss weight and the sorting difference sample pair.

4. The model distillation method according to claim 3, characterized in that: The greater the ranking difference between the teacher ranking matrix and the student ranking matrix between the same ranking difference sample pairs, the greater the ranking difference weight; When the similarity difference between the teacher similarity matrix and the student similarity matrix in the same sorted difference sample pair is greater, the similarity difference weight is greater; The smaller the similarity of the sorted difference sample pair in the teacher similarity matrix, the greater the prior weight of the teacher model.

5. The model distillation method according to claim 3, characterized in that: Determining a difference loss weight by using the ranking difference weight, the similarity difference weight and the teacher model prior weight includes: Assigning a first weight to the ranking difference weight; assigning a second weight to the similarity difference weight; and assigning a third weight to the teacher model prior weight; The difference loss weight is determined according to the first weight, the second weight, the third weight, the sorting difference weight, the similarity difference weight and the teacher model prior weight; wherein, as the current iteration number of the iterative training continues to increase, the larger the first weight is, the smaller the second weight and the third weight are.

6. The model distillation method according to claim 1, characterized in that: According to the teacher similarity matrix, a comparative learning sample pair is constructed, including: The samples corresponding to the similarity greater than the first threshold in the teacher similarity matrix are taken as positive sample pairs, and the samples corresponding to the similarity less than the second threshold are taken as negative sample pairs; wherein the first threshold is greater than the second threshold, and as the current iteration number of the iterative training increases, the greater the first threshold, the smaller the second threshold; The positive sample and the negative sample constitute the contrastive learning sample pair.

7. The model distillation method according to claim 1, characterized in that: According to the sorting difference matrix, constructing sorting difference sample pairs, including: Determine the target object that meets the preset conditions in the sorting difference matrix; wherein the preset conditions include any one of the similarity sorting difference being greater than a preset value, the first preset quantile of the sorting result from large to small of the similarity sorting difference, and the first preset number of the sorting result; The samples corresponding to the target object are used as the sorting difference sample pairs.

8. A model distillation apparatus, characterized in that: The device comprises: A matrix acquisition module, used to acquire a teacher similarity matrix of a teacher model, a student similarity matrix of a student model, and a ranking difference matrix representing the ranking difference of similarity between the teacher model and the student model; the calculation accuracy of the teacher model is higher than the calculation accuracy of the student model; A sample pair construction module, used to construct comparative learning sample pairs according to the teacher similarity matrix; and to construct sorting difference sample pairs according to the sorting difference matrix; A distillation loss function construction module, used to construct a distillation loss function based on the contrastive learning sample pair and the sorting difference sample pair; The student model training module is used to iteratively train the student model through the distillation loss function to obtain a target student model.

9. A model distillation device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the model distillation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the model distillation method according to any one of claims 1 to 7 are implemented.

Images