A multi-order knowledge distillation method applied to a recommendation system

Abstract

The application discloses a multi-order knowledge distillation method applied to a recommendation system, and comprises the following steps: pre-training a large-scale teacher model, a medium-scale buffer model and a small-scale student model; jointly training the buffer model by using a binary training set and supervised knowledge output by the teacher model, wherein the supervised knowledge is filtered through a filter with a tanh saturation activation function; jointly training the student model by using the binary training set and supervised knowledge output by the buffer model, wherein the supervised knowledge is filtered through a filter with a sigmoid saturation activation function; and jointly training the teacher model by using a binary training set and high-confidence knowledge output by the student model, wherein the high-confidence knowledge is the sampling probability of the top K items in a ranking list filtered through a filter with an exp function. The application effectively narrows the huge ranking performance gap between the large-scale teacher model and the small-scale student model, makes the performance of the student model closer to that of the teacher model, and greatly shortens the online inference time of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent deep learning recommendation, and particularly relates to a multi-order knowledge distillation method applied to a recommendation system. BACKGROUND

[0002] A recommendation system can provide a personalized item ranking list for each user. However, due to the rapid development of deep learning, the recommendation system becomes more and more complex, and the parameter quantity also increases sharply. Generally speaking, a recommendation model with a large parameter quantity can achieve better performance, but at the same time, there are some problems: such a model has a large inference delay, and requires huge memory and computing cost. Therefore, it is not suitable to use a large recommendation model for online deployment which requires fast response.

[0003] In recent years, knowledge distillation has been widely applied in the field of recommendation system to compress a large recommendation model. The knowledge learned by a pre-trained large teacher model is transferred to a small student model as additional supervision information, so that the performance of the student model is as close as possible to that of the teacher model. A large number of studies have verified that the student model trained with the knowledge distillation technology is superior to the original student model without using any distillation technology in terms of recommendation performance and inference speed.

[0004] Although the application of knowledge distillation in the field of recommendation system effectively solves the problem of online inference delay and quickly provides an item ranking list for users, there are still many problems in how to generate a personalized ranking list that users are really interested in for different users. Therefore, it is necessary to improve the ranking performance of the student model to approach or even surpass the performance of the teacher model. However, there is a huge performance gap between the existing teacher model and the student model; since the large teacher model has more parameters, it can capture knowledge containing rich information to achieve performance improvement. But the small student model trained with the supervision label of the teacher model still has difficulty in understanding how to correctly apply the soft label of the binary training set. SUMMARY

[0005] The present application aims to provide a multi-order knowledge distillation method applied to a recommendation system, which reduces the huge ranking difference between the teacher model and the student model by introducing a buffer model, and designs a filter to screen the knowledge to be transferred by each recommender, thereby shortening the model training and inference time.

[0006] To achieve the above-mentioned purpose, the present application provides a multi-order knowledge distillation method applied to a recommendation system, which comprises:

[0007] a large-scale teacher model with a large pre-training parameter quantity, a small-scale student model with a small parameter quantity, and a buffer model with a parameter quantity between the teacher model and the student model;

[0008] The buffer model is jointly trained with the binary training set and the supervised knowledge output by the teacher model, which is filtered by a filter with a tanh saturation activation function.

[0009] The student model is jointly trained with the binary training set and the supervised knowledge output by the buffer model, which is filtered by a filter with a sigmoid saturation activation function.

[0010] The teacher model is jointly trained with the binary training set and the high-confidence knowledge output by the student model, which is filtered by a filter with an exp function.

[0011] Further, since updating the list once per round reduces the online inference speed, the recommendation ranking list of each model is updated every p epochs ， Where an epoch means that all data in the training set has been forward propagated and backward propagated in the network once, and the value of p is generally 20.

[0012] Further, the teacher model T, the buffer model B and the student model S are trained using the collaborative filtering technology.

[0013] Further, the training of the buffer model is to add the collaborative filtering loss of the buffer model and the distillation loss together in a certain proportion to make the buffer model learn as much knowledge of the teacher model as possible.

[0014] Further, the filter with a tanh saturation activation function, the sampling probability is proportional to The items with a ranking difference exceeding a threshold value are uniformly sampled, where represents the ranking difference between the teacher model and the buffer model, ξ b (>0) is a hyperparameter for controlling the probability stability.

[0015] Further, the training of the student model is to add the collaborative filtering loss of the student model and the distillation loss together in a certain proportion to make the student model learn as much knowledge of the buffer model and the knowledge of the teacher model learned by the buffer model as possible.

[0016] Further, the filter with a sigmoid saturation activation function, the sampling probability is proportional to where represents the ranking difference between the buffer model and the student model, ξ s (>0) is a hyperparameter for controlling the probability stability.

[0017] Further, the training of the teacher model is to optimize the collaborative filtering loss of the teacher model and the distillation loss in a certain proportion, and the optimized teacher model can further improve the performance of the student model.

[0018] As a further, the filter with the exp function, the sampling probability is proportional to Where represents the ranking difference between the student model and the teacher model, and ξ t (> 0) is a hyperparameter that controls the probability of stability.

[0019] As a further, when training the recommendation model, the parameter settings on the two data sets of CiteULike and Foursquare and the two base models of BPR and CDAE are as follows: the learning rate is 10 -5 , the batch is 1024, the optimizer is Adam, the Hit Ratio and the Normalized Discounted Cumulative Gain are used as evaluation indexes, the parameter quantity of the teacher model is ten times that of the student model, and the distillation temperature is 2.

[0020] The above technical scheme adopted by the present application has the advantages compared with the prior art: the present application effectively reduces the huge ranking performance gap between the large teacher model and the small student model, so that the performance of the student model is closer to that of the teacher model. At the same time, the filter selects the top K items with the largest ranking difference score for each model, which greatly shortens the online inference time of the model. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 It is a multi-order knowledge distillation method flow chart applied to a recommendation system;

[0022] Figure 2 It is a target model joint training process chart in the embodiment;

[0023] Figure 3 It is a filter schematic diagram in the embodiment;

[0024] Figure 4 It is a recommendation model updating ranking list schematic diagram;

[0025] Figure 5 It is a multi-order knowledge distillation use effect schematic diagram. DETAILED DESCRIPTION

[0026] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is described and explained below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. Based on the embodiments provided herein, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0027] As shown in Figure 1 The present embodiment provides a multi-order knowledge distillation method applied to a recommendation system, which includes a teacher model, a buffer model, a student model and K high-score items filtered by a filter. The medium-sized buffer model is used to learn the complex soft labels generated by the large teacher model, which can learn the soft labels of the teacher model better than the small student model. In addition, the student model can obtain more useful knowledge from the buffer model. It can be applied to any large recommendation model to compress the large model into a small model for online inference. The recommendation system can be set in a recommendation application that needs to make real-time recommendations to users, which can be shopping websites, comprehensive content publishing websites and related APPs, etc.

[0028] Since it is time-consuming for the student model to learn all the knowledge of the teacher model. Therefore, a ranking difference sampling strategy of the teacher model, the buffer model and the student model is proposed to select key output information. Based on the strategy, each model learns the items that are ranked lower by itself but higher by its source model, and enables the student model to obtain extensive knowledge of different items learned by the teacher model.

[0029] The present embodiment proposes a new knowledge distillation framework on the recommendation model, named MSD, which is a plug-and-play part that narrows the huge ranking performance gap between the source model and the target model. MSD further improves the speed of online inference, enhances the performance of each recommendation model, and reduces the computational cost. The specific implementation process is as follows:

[0030] S1. Pre-train a large teacher model T, a medium buffer model B and a small student model S respectively using collaborative filtering technology;

[0031] S2. Jointly train the buffer model with binary training set and supervised knowledge output by the teacher model, which is filtered by a filter with tanh saturation activation function;

[0032] Specifically, as shown in Figure 2 The present embodiment regards the buffer model as another small supervised model to narrow the huge ranking difference between the teacher model and the student model. The buffer model is trained simultaneously with the binary U-I interaction matrix and the knowledge of the teacher model. The training method of the buffer model is as follows:

[0033]

[0034] where θ * denotes parameters; denotes the collaborative filtering loss of any recommender. is the MSD loss proposed in this application. The hyper-parameter λ T→B is the weight used to control the MSD loss.

[0035] S3. Jointly train the student model with the binary training set and the supervised knowledge output by the buffer model, which is filtered by a filter with sigmoid saturation activation function;

[0036] Specifically, as shown in Figure 2 the student model can use the buffer model as a bridge to better simulate the output of the teacher model. The training method of the student model is as follows:

[0037]

[0038] where θ * denotes parameters. denotes the collaborative filtering loss of any recommender. is the MSD loss proposed in this application. The hyper-parameter λ B→S is the weight used to control the MSD loss.

[0039] S4. Jointly train the teacher model with the binary training set and the high-confidence knowledge output by the student model, which is the sampling probability of the top K items in the recommended ranking list filtered by a filter with exp function.

[0040] Specifically, the teacher model can further improve the ranking ability by selecting the knowledge with higher confidence of the student model. The training method of the teacher model is as follows:

[0041]

[0042] where θ * denotes parameters. denotes the collaborative filtering loss of any recommender. is the MSD loss proposed in this application. The hyper-parameter λ S→T is the weight used to control the MSD loss.

[0043] S5. Obtain the distillation loss of each recommendation model in the following way:

[0044]

[0045]

[0046]

[0047] where, denotes the classic binary cross-entropy loss, is the set of un-interacted items selected by the user through the filter. The recommendation model utilizes the scores to rank the unobserved items from high to low. where, σ(·) is the activation function, z ui is the logit, and T is the temperature of the MSD. As the temperature increases, the probability distribution will generally become flatter.

[0048] The user's latent preference for unobserved items is transferred by In the U-I matrix, the user's unobserved items are only marked as 0, and the items that the user has interacted with are marked as 1. Through the MSD, the student model can further understand the scores of these items predicted by the large teacher model to achieve performance improvement.

[0049] As Figure 3 shown, the present example proposes a filter to determine which knowledge should be transferred to the next recommender, and a sampling strategy is designed for the filter, and different functions are designed for each recommender to better determine which knowledge should be transferred to another recommender.

[0050] The present example designs a probability function to determine which items ranked by the teacher model should be sampled out as knowledge to be transferred to the buffer model, and the function is defined as follows:

[0051]

[0052] where, denotes the ranking difference between the teacher model and the buffer model, ξ b (>0) is a hyperparameter that controls the probability stability, and a∝b means that a is proportional to b. tanh(·) denotes a saturation function, and items with a ranking difference exceeding the threshold are almost uniformly sampled.

[0053] In order to narrow the huge ranking ability difference between the teacher model and the student model, the present example designs a probability function to determine which items ranked by the buffer model should be sampled out as knowledge to be transferred to the student model, and the function is defined as follows:

[0054]

[0055] where, denotes the ranking difference between the buffer model and the student model, ξ s (>0) is a hyperparameter that controls the probability stability; sigmoid(·) denotes a saturation activation function.

[0056] The items ranked high by the student model can be ranked low in the teacher model, so the present example designs a probability function to determine which items ranked by the student model should be sampled out as knowledge transfer to the teacher model, which is defined as follows:

[0057]

[0058] wherein, represents the difference in ranking between the student model and the teacher model, and t is a hyperparameter that controls the probability stationarity. exp(·) represents the exponential function.

[0059] As shown in Figure 4 , the recommended model updates the ranking list, and each iteration of the recommended list requires a large amount of computation time, which is not suitable for online inference tasks. The present example designs to update the recommended list of each model every p epochs, and generally the value of p is 20.

[0060] As shown in Figure 5 , the multi-order knowledge distillation method of the present application can be applied to any commonly used model of a recommendation system, such as LightGCN, BPR, CDAE, NeuMF, etc.

[0061] The present example designs a buffer model as a bridge for knowledge transfer between the teacher model and the student model, to improve the ranking ability of the learning model. At the same time, a filter is designed to select the top K items ranked high by the source recommendation model, to improve the online inference speed of the model. The present example not only improves the ranking ability of the student model, so that it can be applied to real-time updated recommendation tasks, but also improves the online inference speed, to bring better interactive experience to users.

[0062] The foregoing description of specific example embodiments of the application is intended for the purpose of illustration and example only. These descriptions are not intended to limit the application to the precise form disclosed, and obviously many changes and modifications can be suggested to one skilled in the art without departing from the spirit or scope of the application. Embodiments were chosen and described in order to explain the principles of the application and its practical application, to enable others skilled in the art to understand the application for various embodiments and with various modifications as are suited to the particular use contemplated, as well as to enable others skilled in the art to implement and utilize the application. The scope of the application is to be defined by the claims and their equivalents.

Claims

1. A multi-stage knowledge distillation method applied to a recommendation system, characterized in that, Comprise: a large-scale teacher model with a large amount of pre-training parameters, a small-scale student model with a small amount of parameters, and a buffer model with a parameter amount between the teacher model and the student model; jointly training the buffer model with the binary training set and the supervision knowledge output by the teacher model, which is filtered by a filter with a tanh saturation activation function; jointly training the student model with the binary training set and the supervision knowledge output by the buffer model, which is filtered by a filter with a sigmoid saturation activation function; jointly training the teacher model with the binary training set and the high-confidence knowledge output by the student model, which is the sampling probability of the top K items in the recommended ranking list filtered by a filter with an exp function; The training of the buffer model is to combine the collaborative filtering loss of the buffer model and the distillation loss in a certain proportion for training.

2. The multi-stage knowledge distillation method for a recommendation system according to claim 1, wherein, updating the recommended ranking list of each recommendation model every p epochs, wherein an epoch represents that all data in the training set has been forward propagated and backward propagated in the network once. 3.The multi-stage knowledge distillation method applied to a recommendation system of claim 1, wherein, Training a teacher model using collaborative filtering techniques T , a buffer model B and a student model S . 4.The multi-stage knowledge distillation method applied to a recommendation system of claim 1, wherein, Filters with tanh saturation activation function, sampling probability proportional to , items with ranking difference over a threshold are sampled uniformly, where denotes the ranking difference between the teacher and buffer models, is a hyperparameter controlling the probability stationarity. 5.The multi-stage knowledge distillation method applied to a recommendation system of claim 1, wherein, The training of the student model is to add up the collaborative filtering loss of the student model and the distillation loss in a certain proportion.

6. The multi-stage knowledge distillation method for a recommendation system according to claim 5, wherein, The sampling probability with sigmoid saturation activation function is proportional to where represents the ranking difference between the buffer model and the student model, is a hyperparameter that controls the probability stationarity. 7.The multi-stage knowledge distillation method applied to a recommendation system of claim 1, wherein, The training of the teacher model is to optimize the collaborative filtering loss of the teacher model and the distillation loss are added up in a certain proportion.

8. The multi-stage knowledge distillation method for a recommendation system according to claim 7, wherein, Filter with exp function, sampling probability is proportional to where denotes the ranking difference between the student and teacher models, is a hyperparameter that controls the probability stationarity. 9.The multi-stage knowledge distillation method applied to a recommendation system of claim 1, wherein, When training the recommendation model, the parameter settings on the two datasets of CiteULike and Foursquare and the two base models of BPR and CDAE are as follows: the learning rate is , the batch is 1024, the optimizer is Adam, Hit Ratio and Normalized Discounted Cumulative Gain are used as evaluation indexes, the parameter quantity of the teacher model is ten times that of the student model, and the distillation temperature is 2.

Images